Text detection model training method, text detection method and device

Abstract

The present disclosure relates to a training method, a text detection method and a device for a text detection model, wherein the training method for the text detection model comprises: obtaining a target sample image set, wherein the target sample image set comprises a target sample image with a text box label; rotating the target sample image to obtain a rotated target sample image; determining the target position information of the text box in the rotated target sample image; training a neural network for text detection based on the rotated target sample image and the target position information to obtain a text prediction result corresponding to the target sample image; determining loss data based on the text prediction result and the text box label; training the above neural network based on the loss data to obtain a text detection model. The technical solution provided by the embodiments of the present disclosure can not only maintain a high accuracy for text recognition in common natural scenes, but also ensure a high accuracy and recall rate for vertical text detection.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a text detection model training method, a text detection method and a text detection device. Background Art

[0002] Text detection is a very important visual task as a prerequisite for text recognition. With the development of deep learning technology, using neural networks for text detection has become a popular method.

[0003] In the related art, the same network framework is usually adopted to detect texts in different languages ​​by transforming the data set. However, there is a special case in Chinese character detection, namely vertical text. Due to the fact that there is less annotated data for vertical text and it is difficult to confirm the vertical relationship, the network framework in the related art is used to detect it, which cannot ensure the high accuracy and recall rate of vertical text detection, so that the network framework in the related art cannot well support the detection of vertical text. Summary of the invention

[0004] The present disclosure provides a text detection model training method, a text detection method and a device, so as to at least solve the problem that the network framework in the related art cannot ensure high accuracy and recall rate of vertical text detection. The technical solution of the present disclosure is as follows:

[0005] According to a first aspect of an embodiment of the present disclosure, a method for training a text detection model is provided, comprising:

[0006] Acquire a target sample image set, wherein the target sample image set includes target sample images with text box labels;

[0007] Rotating the target sample image to obtain a rotated target sample image;

[0008] Determining target position information of a text box in the rotated target sample image;

[0009] Based on the rotated target sample image and the target position information, performing text detection training on a neural network to obtain a text prediction result corresponding to the target sample image;

[0010] Determining loss data according to the text prediction result and the text box label;

[0011] The neural network is trained based on the loss data to obtain a text detection model.

[0012] In an exemplary embodiment, before acquiring the target sample image set, the text detection model training method further includes:

[0013] Acquire a candidate sample image set, wherein the candidate sample image set includes candidate sample images;

[0014] Performing text detection on the candidate sample image to obtain a candidate sample image annotated with the text box label;

[0015] According to the text box label, obtaining a target candidate sample image containing a vertical text box from the candidate sample image set;

[0016] Oversampling the target candidate sample image to obtain an oversampled target candidate sample image;

[0017] In response to a vertical text editing instruction triggered based on the oversampled target candidate sample image, editing a preset number of vertical texts on the oversampled target candidate sample image to obtain an edited target candidate sample image;

[0018] The target sample image set is determined based on other candidate sample images in the candidate sample image set except the target candidate sample image, and the edited target candidate sample image, wherein the target sample image includes the other candidate sample images and the edited target candidate sample image.

[0019] In an exemplary embodiment, determining the target position information of the text box in the rotated target sample image includes:

[0020] Based on the text box label, determining the original position information of the text box;

[0021] Determining offset information of the original position information during the rotation process;

[0022] Performing a linear transformation on the original position information to obtain transformed original position information;

[0023] The target position information is determined based on the transformed original position information and the offset information.

[0024] In an exemplary embodiment, there are multiple target sample images, and rotating the target sample images to obtain the rotated target sample images includes:

[0025] Determine the rotation center corresponding to each of the plurality of target sample images;

[0026] Based on the rotation centers corresponding to the multiple target sample images, the multiple target sample images are rotated along the corresponding rotation directions by corresponding angles to obtain the multiple rotated target sample images.

[0027] According to a second aspect of an embodiment of the present disclosure, a text detection method is provided, including:

[0028] Acquire the image to be detected;

[0029] Performing text detection processing on the image to be detected based on the text detection model to obtain a text detection result corresponding to the image to be detected;

[0030] The text detection model is trained using the text detection model training method described in any of the above embodiments.

[0031] In an exemplary embodiment, the text detection result includes multiple text detection boxes and the probability that each of the multiple text detection boxes contains text. After performing text detection processing on the image to be detected based on the text detection model to obtain the text detection result corresponding to the image to be detected, the text detection method further includes:

[0032] From the multiple text detection frames, obtaining a preset number of candidate text detection frames whose probability of containing text is greater than a preset probability threshold;

[0033] Determine first size information corresponding to each of the preset number of candidate text detection frames;

[0034] Sort the preset number of candidate text detection frames in descending order according to the corresponding first size information to obtain a first sequence;

[0035] Taking the candidate text detection frame ranked first in the first sequence as the current candidate text detection frame, and migrating the current candidate text detection frame from the first sequence to the candidate set;

[0036] Determine, from the first sequence, other candidate text detection frames whose intersection-over-union ratio with the current candidate text detection frame is greater than a preset intersection-over-union ratio threshold;

[0037] Filtering the other candidate text detection frames;

[0038] Repeat the steps of using the candidate text detection frame that is ranked first in the first sequence as the current candidate text detection frame, and migrating the current candidate text detection frame from the first sequence to the candidate set, to filtering the other candidate text detection frames, until the first sequence is empty, and when the first sequence is empty, using the candidate text detection frame in the candidate set as the target text detection frame corresponding to the image to be detected.

[0039] In an exemplary embodiment, there are multiple target text detection frames. When the first sequence is empty, after taking the candidate text detection frames in the candidate set as the target text detection frames corresponding to the image to be detected, the text detection method further includes:

[0040] Determine second size information corresponding to each of the plurality of target text detection frames;

[0041] Using a preset number of target text detection frames whose second size information meets a preset condition as a preset number of vertical text detection frames;

[0042] Determine the horizontal coordinate information corresponding to each of the preset number of vertical text detection frames;

[0043] Sort the preset number of vertical text detection frames in ascending order according to their corresponding horizontal coordinate information to obtain a second sequence;

[0044] Each vertical text detection frame in the second sequence is traversed in sequence. When traversing each vertical text detection frame, the following operations are performed:

[0045] Determine other vertical text detection frames corresponding to the current vertical text detection frame currently traversed, wherein the other vertical text detection frames corresponding to the current vertical text detection frame are vertical text detection frames in the second sequence that satisfy a preset size relationship and a preset position relationship with the current vertical text detection frame;

[0046] A set of vertical text detection frames corresponding to the image to be detected is determined according to the current vertical text detection frame and the corresponding other vertical text detection frames.

[0047] In an exemplary embodiment, the vertical text detection frame set is a plurality of sets, then after determining a set of vertical text detection frame sets corresponding to the image to be detected according to the current vertical text detection frame and the corresponding target vertical text detection frame, the text detection method further includes:

[0048] Connecting the vertical text detection frames included in each of the multiple sets of vertical text detection frame sets to obtain target vertical text detection frames corresponding to each of the multiple sets of vertical text detection frame sets;

[0049] The target vertical text detection frames corresponding to the multiple groups of vertical text detection frame sets are used as the vertical text detection frames corresponding to the image to be detected.

[0050] According to a third aspect of an embodiment of the present disclosure, a training device for a text detection model is provided, comprising:

[0051] A target sample image set acquisition module is configured to execute acquisition of a target sample image set, wherein the target sample image set includes target sample images with text box labels;

[0052] A rotation module is configured to rotate the target sample image to obtain a rotated target sample image;

[0053] A target position information determination module, configured to determine the target position information of the text box in the rotated target sample image;

[0054] A training module is configured to perform text detection training on a neural network based on the rotated target sample image and the target position information to obtain a text prediction result corresponding to the target sample image;

[0055] A loss data determination module, configured to determine loss data according to the text prediction result and the text box label;

[0056] The text detection model determination module is configured to execute training of the neural network based on the loss data to obtain a text detection model.

[0057] In an exemplary embodiment, the training device of the text detection model further includes:

[0058] A candidate sample image set acquisition module is configured to execute acquisition of a candidate sample image set, wherein the candidate sample image set includes candidate sample images;

[0059] A text detection module is configured to perform text detection on the candidate sample image to obtain a candidate sample image annotated with the text box label;

[0060] A target candidate sample image acquisition module is configured to acquire a target candidate sample image containing a vertical text box from the candidate sample image set according to the text box label;

[0061] An oversampling module is configured to perform oversampling on the target candidate sample image to obtain an oversampled target candidate sample image;

[0062] an editing module, configured to execute a vertical text editing instruction triggered based on the oversampled target candidate sample image, edit a preset number of vertical texts on the oversampled target candidate sample image, and obtain an edited target candidate sample image;

[0063] The target sample image set determination module is configured to determine the target sample image set based on other candidate sample images in the candidate sample image set except the target candidate sample image, and the edited target candidate sample image, wherein the target sample image includes the other candidate sample images and the edited target candidate sample image.

[0064] In an exemplary embodiment, the target location information determination module includes:

[0065] an original position information determining unit, configured to determine the original position information of the text box based on the text box label;

[0066] an offset information determining unit, configured to determine the offset information of the original position information during the rotation process;

[0067] a transformation unit, configured to perform a linear transformation on the original position information to obtain transformed original position information;

[0068] The target position information determining unit is configured to determine the target position information based on the transformed original position information and the offset information.

[0069] In an exemplary embodiment, there are multiple target sample images, and the rotation module includes:

[0070] A rotation center determination unit, configured to determine the rotation center corresponding to each of the plurality of target sample images;

[0071] The target sample image determination unit is configured to rotate the multiple target sample images by corresponding angles along the respective corresponding rotation directions based on the respective corresponding rotation centers of the multiple target sample images to obtain the multiple rotated target sample images.

[0072] According to a fourth aspect of an embodiment of the present disclosure, a text detection device is provided, including:

[0073] The module for acquiring the image to be detected is configured to execute the acquisition of the image to be detected;

[0074] The text detection result determination module is configured to perform text detection processing on the image to be detected based on a text detection model to obtain a text detection result corresponding to the image to be detected; wherein the text detection model is trained using the training method of the text detection model described in any of the above embodiments.

[0075] In an exemplary embodiment, the text detection result includes a plurality of text detection frames and the probability that each of the plurality of text detection frames contains text, and the text detection device further includes:

[0076] A candidate text detection frame acquisition module is configured to acquire a preset number of candidate text detection frames whose probability of containing text is greater than a preset probability threshold from the multiple text detection frames;

[0077] A first size information determination module is configured to determine first size information corresponding to each of the preset number of candidate text detection frames;

[0078] A first sequence acquisition module is configured to sort the preset number of candidate text detection frames in descending order according to corresponding first size information to obtain a first sequence;

[0079] A migration module is configured to execute the process of taking the candidate text detection frame that is ranked first in the first sequence as the current candidate text detection frame, and migrating the current candidate text detection frame from the first sequence to the candidate set;

[0080] Another candidate text detection frame determination module is configured to determine, from the first sequence, another candidate text detection frame whose intersection-over-union ratio with the current candidate text detection frame is greater than a preset intersection-over-union ratio threshold;

[0081] A filtering module, configured to filter the other candidate text detection frames;

[0082] The target text detection frame determination module is configured to execute the steps of repeatedly using the candidate text detection frame sorted first in the first sequence as the current candidate text detection frame, migrating the current candidate text detection frame from the first sequence to the candidate set, and filtering the other candidate text detection frames until the first sequence is empty, and when the first sequence is empty, using the candidate text detection frame in the candidate set as the target text detection frame corresponding to the image to be detected.

[0083] In an exemplary embodiment, there are multiple target text detection frames, and the text detection device further includes:

[0084] A second size information determination module is configured to determine the second size information corresponding to each of the plurality of target text detection frames;

[0085] A vertical text detection frame determination module is configured to execute a preset number of target text detection frames whose second size information meets a preset condition as a preset number of vertical text detection frames;

[0086] A horizontal coordinate information determination module is configured to determine the horizontal coordinate information corresponding to each of the preset number of vertical text detection frames;

[0087] A second sequence determination module is configured to sort the preset number of vertical text detection frames in ascending order according to their corresponding horizontal coordinate information to obtain a second sequence;

[0088] The traversal module is configured to sequentially traverse each vertical text detection frame in the second sequence, and when traversing each vertical text detection frame, perform the following operations:

[0089] The other vertical text detection frame determination module is configured to determine other vertical text detection frames corresponding to the current vertical text detection frame currently traversed, wherein the other vertical text detection frames corresponding to the current vertical text detection frame are vertical text detection frames in the second sequence that satisfy a preset size relationship and a preset position relationship with the current vertical text detection frame;

[0090] The detection frame set determination module is configured to determine a set of vertical text detection frame sets corresponding to the image to be detected based on the current vertical text detection frame and the corresponding other vertical text detection frames.

[0091] In an exemplary embodiment, the vertical text detection frame set is a plurality of groups, and the text detection device further includes:

[0092] a connection module configured to perform connection processing on the vertical text detection frames respectively included in the plurality of sets of vertical text detection frame sets to obtain target vertical text detection frames respectively corresponding to the plurality of sets of vertical text detection frame sets;

[0093] The determination module is configured to execute the target vertical text detection frame corresponding to each of the plurality of groups of vertical text detection frame sets as the vertical text detection frame corresponding to the image to be detected.

[0094] According to a fifth aspect of an embodiment of the present disclosure, there is provided an electronic device, including:

[0095] processor;

[0096] a memory for storing instructions executable by the processor;

[0097] The processor is configured to execute the instructions to implement the text detection model training method as described in any of the above embodiments or the text detection method as described in any of the above embodiments.

[0098] According to the sixth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the training method of the text detection model as described in any of the above embodiments or the text detection method as described in any of the above embodiments.

[0099] According to a seventh aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program, which, when executed by a processor, implements the training method of the text detection model described in any of the above embodiments or the text detection method described in any of the above embodiments.

[0100] The technical solution provided by the embodiments of the present disclosure brings at least the following beneficial effects:

[0101] In the training and learning process of the text detection model, the present invention rotates the target sample image, and calculates the target position information of the text box in the target sample image under the rotation transformation in real time, and then trains the text detection model according to the rotated target sample image and the target position information, so that the trained text detection model can detect text boxes at any angle, thereby optimizing the detection effect of vertical text while achieving general text detection tasks. Subsequently, when the text detection model is used to perform text detection on the image to be detected, not only can a high accuracy be maintained for text recognition in common natural scenes, but also a high accuracy and recall rate for vertical text detection can be ensured.

[0102] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0103] The drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute improper limitations on the present disclosure.

[0104] Figure 1 The figure is a schematic diagram of an application environment of a text detection system according to an exemplary embodiment.

[0105] Figure 2 It is a flowchart of a method for training a text detection model according to an exemplary embodiment.

[0106] Figure 3 The figure is a schematic diagram of a process of determining a target sample image set according to an exemplary embodiment.

[0107] Figure 4 It is a schematic diagram of a process of determining target position information of a text box in the rotated target sample image according to an exemplary embodiment.

[0108] Figure 5 It is a flowchart of a text detection method according to an exemplary embodiment.

[0109] Figure 6The present invention is a schematic diagram showing a flow chart of determining a target text detection frame corresponding to an image to be detected according to an exemplary embodiment.

[0110] Figure 7 The present invention is a schematic diagram showing a flow chart of determining a set of vertical text detection frames corresponding to an image to be detected according to an exemplary embodiment.

[0111] Figure 8 The present invention is a schematic diagram showing a flow chart of determining a vertical text detection frame corresponding to an image to be detected according to an exemplary embodiment.

[0112] Fig. 9 The diagram is a schematic diagram showing a vertical text detection frame in a vertical text detection frame set before and after being connected according to an exemplary embodiment. Fig. 9 (a) is the schematic diagram before connection. Fig. 9 (b) is the schematic diagram after connection.

[0113] Fig.10 It is a block diagram of a training device for a text detection model according to an exemplary embodiment.

[0114] Fig.11 It is a block diagram of a text detection device according to an exemplary embodiment.

[0115] Fig.12 It is a block diagram of an electronic device for text detection according to an exemplary embodiment. DETAILED DESCRIPTION

[0116] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings.

[0117] It should be noted that the terms "first", "second", etc. in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0118] See also Figure 1 , Figure 1 A schematic diagram of an application environment of a text detection system according to an exemplary embodiment. Figure 1As shown, the application environment may include a training device 001 , a terminal 002 and a server 003 .

[0119] The training device 001 is used to perform text detection training and send the trained text detection model to the server 003. Optionally, the training device 001 may include a terminal device such as a smart phone, a desktop computer, a tablet computer, a laptop computer, a digital assistant, an augmented reality (AR) / virtual reality (VR) device, a smart wearable device, etc., and may also include an independently running server, or a distributed server, or a server cluster consisting of multiple servers.

[0120] The terminal 002 can be used to collect the image to be detected and send the image to be detected to the server 003. Optionally, the terminal 002 may include electronic devices such as smart phones, desktop computers, tablet computers, laptop computers, smart speakers, digital assistants, augmented reality (AR) / virtual reality (VR) devices, smart wearable devices, etc. It can also be software running on the above electronic devices, such as applications, applets, etc. Optionally, the operating system running on the electronic device may include but is not limited to Android system, IOS system, Linux, Windows, etc.

[0121] Server 003 can be used to perform text detection on the image to be detected based on the detection model obtained by training, and determine the text detection result. Optionally, server 003 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0122] In addition, it should be noted that Figure 1 The diagram shown is only a schematic diagram of an application environment of the text detection system provided by the present disclosure. In practical applications, other application environments may also be included. For example, the server 003 may also be used to train the text detection model. Accordingly, the text detection system may include the terminal 002 and the server 003.

[0123] The following introduces a training method for a text detection model disclosed in the present invention. Figure 2 is a flowchart of a method for training a text detection model according to an exemplary embodiment. Figure 2 As shown, the following steps may be included.

[0124] In step S11, a target sample image set is obtained, where the target sample image set includes target sample images with text box labels.

[0125] In the embodiment of the present disclosure, when training a text detection model, a target sample image set may be obtained, and the target sample image set may include at least one target sample image, each of which is annotated with a corresponding text box label.

[0126] Exemplarily, the text box label may include: whether a text box is included, the probability of the text box containing text, text box position information, text box type information, etc.

[0127] Exemplarily, the text box position information may be coordinate information of the text box in the target sample image.

[0128] Exemplarily, the text box type information may include a horizontal text box and a vertical text box. Optionally, the vertical text box may be a text box for vertical text or a minimum circumscribed rectangular box for oblique text.

[0129] In an optional embodiment, if Figure 3 As shown, Figure 3 is a schematic diagram of a process for determining a target sample image set according to an exemplary embodiment. Accordingly, before obtaining the target sample image set, the method may further include:

[0130] In step S01, a candidate sample image set is obtained, where the candidate sample image set includes candidate sample images.

[0131] In step S02, text detection is performed on the candidate sample image to obtain a candidate sample image marked with the text box label.

[0132] In step S03, according to the text box label, a target candidate sample image containing a vertical text box is obtained from the candidate sample image set.

[0133] In step S04, the target candidate sample image is oversampled to obtain an oversampled target candidate sample image.

[0134] In step S05, in response to a vertical text editing instruction triggered based on the oversampled target candidate sample image, a preset number of vertical texts are edited on the oversampled target candidate sample image to obtain an edited target candidate sample image.

[0135] In step S06, based on other candidate sample images in the candidate sample image set except the target candidate sample image, and the edited target candidate sample image, the target sample image set is determined, and the target sample image includes the other candidate sample images and the edited target candidate sample image.

[0136] Exemplarily, in step S01, a candidate sample image set consisting of at least one candidate sample image may be acquired in a preset manner.

[0137] Exemplarily, in step S02, text detection can be automatically performed on each candidate sample image by a preset text detection method or a preset text detection model, and the text detection result is returned in the form of a text box. According to the text detection result returned in the form of a text box, a text box is annotated for each candidate sample image to obtain a candidate sample image annotated with a text box label. Optionally, the text box label may include: whether a text box is contained, the probability of containing text in the text box, text box position information, text box type information, etc.

[0138] Optionally, after step S02, for candidate sample images whose text boxes are not fully annotated, the unannotated text boxes in the candidate sample images may be annotated in response to a text box supplementation instruction triggered based on the candidate sample images. For candidate sample images whose text box annotations are deviated, the text boxes in the candidate sample images whose annotations are deviated may be adjusted in response to a text box adjustment instruction triggered based on the candidate sample images.

[0139] Through the above-mentioned step S02, text detection can be automatically performed on the candidate sample image, which greatly saves the cost of text box annotation in the candidate sample image and improves the efficiency and accuracy of text box annotation; in addition, through the above-mentioned text box supplementation instructions and the above-mentioned text box adjustment instructions, the annotation result in step S02 can be supplemented or the deviation can be adjusted, thereby further improving the accuracy of text box annotation in the candidate sample image, and thus making the trained text detection model have higher text detection accuracy.

[0140] In practical applications, the candidate sample images obtained by the annotation in step S02 may contain less vertical text. Using the less vertical text to train the text detection model may make the trained text detection model unable to ensure the high accuracy and recall rate of vertical text detection. In order to enable the trained text detection model to have high accuracy and recall rate for vertical text detection while achieving general text detection tasks, illustratively, in step S03 and step S04, the candidate sample images containing vertical text can be expanded to enhance the candidate sample image set offline. Specifically: according to the text box type information in the pre-annotated text box label, the target candidate sample image containing the vertical text box can be obtained, and the target candidate sample image can be oversampled according to a preset multiple to obtain the oversampled target candidate sample image. Among them, oversampling is a sampling method for increasing samples, that is, increasing the number of samples by a preset number of times.

[0141] Exemplarily, in order to further enhance the candidate sample image set offline, thereby further ensuring that the trained text detection model can achieve a general text detection task while also having a high accuracy and recall rate for detecting vertical text, in step S05, vertical text can also be randomly added to the oversampled target candidate sample image.

[0142] Exemplarily, the putText method in opencv can be used to respond to the vertical text marking instruction triggered in the area without text in the oversampled target candidate sample image, edit a preset number of vertical random texts in the area without text, and obtain the edited target candidate sample image. Among them, opencv is a cross-platform computer vision and machine learning software library. putText is a function that copies text from a storage area to the screen, which is used to display text at a certain position in an image.

[0143] Optionally, the parameters in putText can be as follows:

[0144] Parameter 1: the image of the text to be displayed (i.e. the target candidate sample image after oversampling).

[0145] Parameter 2: the text to be displayed (i.e. random vertical text).

[0146] Parameter 3: The coordinates of the lower left corner of the text in the image.

[0147] Parameter 4: font type.

[0148] Parameter 5: font size.

[0149] Parameter 6: font color.

[0150] Parameter 7: font thickness.

[0151] Parameter 8: Line type.

[0152] The disclosed embodiment uses the above two offline enhancement strategies (expanding candidate sample images containing vertical text and adding random vertical text to the expanded target candidate sample images) to enhance the less common vertical text in natural scenes, and finally completes the collection of the target sample image set. Due to the expansion of candidate sample images containing vertical text and the addition of random vertical text to the expanded target candidate sample images, the diversity of candidate sample images is enriched, which indirectly increases the amount of training data for the text detection model, thereby ensuring that the trained text detection model can achieve general text detection tasks while also having a high accuracy and recall rate for vertical text detection.

[0153] In step S12, the target sample image is rotated to obtain a rotated target sample image.

[0154] For example, in order to enable the trained text detection model to detect text boxes at any angle and further improve the accuracy and recall rate of vertical text detection, random online data enhancement operations can be performed on the target sample images during the data reading phase of model training.

[0155] Optionally, the online data enhancement operation includes online rotation, color channel transformation, color enhancement, occlusion and other operations.

[0156] For the online rotation operation, in a specific embodiment, when there are multiple target sample images, rotating the target sample images to obtain the rotated target sample images may include the following steps:

[0157] Determine the rotation center corresponding to each of the plurality of target sample images.

[0158] Based on the respective corresponding rotation centers of the plurality of target sample images, the plurality of target sample images are rotated along the respective corresponding rotation directions by corresponding angles to obtain the plurality of rotated target sample images.

[0159] The online rotation operation in the disclosed embodiment is to enable the trained text detection model to detect text boxes at any angle. During the rotation process, for any target sample image, it can be randomly rotated at any angle between 0 and 360 degrees along the corresponding rotation center and rotation direction.

[0160] Optionally, in order to further improve the diversity and richness of the target sample images, and thus further improve the text detection accuracy of the text detection model, the rotation angles of the multiple target sample images may be uniformly sampled between 0° and 360°.

[0161] In step S13, the target position information of the text box in the rotated target sample image is determined.

[0162] In the disclosed embodiment, after the target sample image is rotated, the target position information of the text box included in the rotated target sample image may be calculated.

[0163] Optionally, the target position information may be coordinate information of the text box in the target sample image.

[0164] In a specific embodiment, affine transformation can be used to calculate in real time the coordinate transformation of the text box under the rotation transformation. Figure 4 FIG. 1 is a flow chart showing a method for determining target position information of a text box in the rotated target sample image according to an exemplary embodiment. Figure 4 As shown, the following steps may be included:

[0165] In step S131, the original position information of the text box is determined based on the text box label.

[0166] In step S132, the offset information of the original position information during the rotation process is determined.

[0167] In step S133, a linear transformation is performed on the original position information to obtain transformed original position information.

[0168] In step S134, the target position information is determined based on the transformed original position information and the offset information.

[0169] The affine transformation on an image is essentially a pixel point in an image that is moved to another location through a certain linear transformation. Mathematically speaking, it is a process of performing a linear transformation on a vector space and adding a translation vector to transform it into another vector space.

[0170] Assume that a vector space m: m = (x, y) is transformed to another vector space n: n = (x', y'). The linear transformation of the vector space from m to n is n = A*m+b. Then,

[0171] x′=A 00 *x+A 01 *y+b0

[0172] y′=A 10 *x+A11 *y+b1,

[0173] Among them, b0 and b1 are offset information (ie, offset).

[0174] In the disclosed embodiment, since the text box label includes the text box position information, in step S131, the original position information of the text box before the rotation transformation can be obtained from the text box label. And in step S132, the offset information during the rotation process is determined (for example, b0, b1 in the above formula). Then in step S133, the original position information is linearly transformed using a preset linear transformation method to obtain the transformed original position information. Finally, in step S134, the transformed original position information and the offset information are added to obtain the above target position information.

[0175] In the disclosed embodiment, the position information of the text box in the target sample image under the rotation transformation can be calculated in real time and accurately through affine transformation, thereby improving the text detection accuracy of the trained text detection model. It has been experimentally verified that the mean average precision (MAP) of the text detection model trained using the above-mentioned online enhancement operation (i.e., the above-mentioned rotation and affine transformation method) is improved by about 2 percentage points (PP) compared with the MAP of the text detection model trained without the above-mentioned online enhancement operation.

[0176] In step S14, based on the rotated target sample image and the target position information, a neural network is trained for text detection to obtain a text prediction result corresponding to the target sample image.

[0177] In step S15, the loss data is determined based on the text prediction result and the text box label.

[0178] In step S16, the neural network is trained based on the loss data to obtain a text detection model.

[0179] In the disclosed embodiment, the rotated target sample image and the target position information may be input into a neural network for text detection training to obtain a text prediction result corresponding to the target sample image.

[0180] In step S15, the loss data can be determined according to the text prediction result and the text box label. In step S16, if the loss data does not meet the preset conditions, the text detection training steps are repeated. If the loss data meets the preset conditions, the current neural network model is used as the text detection model.

[0181] Exemplarily, the neural network may include a target detection algorithm model, and the target detection algorithm model may include Faster RCNN, CenterNet, but is not limited thereto.

[0182] For example, the input of model training can be a 512*512 RGB image. For target sample images whose aspect ratio is not 1:1, before resizing the image, padding can be used to expand it into a square image. RGB is an industrial standard that obtains a variety of colors by changing the three color channels of red (R), green (G), and blue (B) and superimposing them on each other.

[0183] For example, since vertical text and horizontal text can be judged by aspect ratio, in order to reduce the classification loss of the model, more expressive power of the model is used to learn the position of the text, rather than focusing on the direction of the text in the current detection box, especially in the early stage of model training, when the text box is incomplete (such as vertical text only frames a certain character in the middle), the direction of the text is difficult to distinguish. Therefore, a single-target detection task can be used to enable the model to converge faster.

[0184] Exemplarily, when using a single-target detection task, the text prediction result obtained in the above step S14 may include: the probability that the position information of the text box reaches the position information contained in the above text box label, the probability that the text contained in the text box reaches the target level, and the target level is the probability of the text contained in the above text box label, and so on.

[0185] For example, in some scenarios, a multi-target detection task can also be used so that the model can be used to learn the location of the text and focus on the direction of the text in the current detection box. Accordingly, the text prediction result obtained in the above step S14 can also include the probability that the type information of the text box reaches the text box type included in the above text box label.

[0186] For example, the neural network is CenterNet, and the above model training process is explained: "DLA_34" is used as the backbone network, and the model is trained by a single machine eight-card distributed training method. The single machine Batch_Size can be 32, and the total Batch Size can be 256; the initial learning rate can be 1.25e -4, and the learning rate can be reduced to 1 / 10 of the previous one starting from the 10th and 20th epochs respectively; Adam is selected as the optimizer. When the model is trained to the 26th epoch, the loss data (Loss) tends to be stable. At this time, the model training can be terminated and the current model is used as the above text detection model.

[0187] DLA is the abbreviation of Deep Layer Aggregation, and its Chinese name is Deep Aggregation. Batch refers to a part of the data sent to the neural network for training each time, and Batch Size refers to the number of training samples in each batch. Epoch refers to a complete training of the neural model using the target sample images in the target sample image set. Adam is an optimization algorithm.

[0188] In summary, during the training process of the above-mentioned text detection model, the candidate sample images were enhanced offline in two aspects (expansion of the candidate sample images containing vertical text and addition of random vertical text to the expanded candidate sample images), and the vertical text that is less common in natural scenes was enhanced in a targeted manner, which increased the diversity and richness of the candidate sample images, and indirectly increased the amount of training data for the text detection model, thereby ensuring that the trained text detection model can achieve general text detection tasks while also having a high accuracy and recall rate for vertical text detection. The above-mentioned online enhancement operation (i.e., the above-mentioned rotation and affine transformation method) enables the trained text detection model to further detect text boxes at any angle, thereby further improving the accuracy and recall rate of the trained text detection model for vertical text detection.

[0189] A text detection method of the present disclosure is introduced below. Figure 5 is a flowchart of a text detection method according to an exemplary embodiment. Figure 5 As shown, the text detection method is used for Figure 1 The system including the training device 001, the terminal 002 and the server 003 shown in the figure is used for illustration, and may include the following steps.

[0190] In step S21, an image to be detected is obtained.

[0191] In step S22, text detection processing is performed on the image to be detected based on the text detection model to obtain a text detection result corresponding to the image to be detected. The text detection model is trained using the training method of the text detection model described in any of the above embodiments.

[0192] Optionally, the text detection result may include multiple text detection boxes, the probability that each of the multiple text detection boxes contains text, position information corresponding to each of the multiple text detection boxes, etc.

[0193] Exemplarily, when the text detection model is used to learn a multi-objective task, the text detection result may also include type information of the text box.

[0194] In the training process of the above text detection model, the candidate sample images are enhanced offline in two aspects to obtain the enhanced target sample image set, and then the target sample images are enhanced online, ensuring that the trained text detection model can achieve general text detection tasks while optimizing the detection effect of vertical text. Therefore, using this text detection model to perform text detection on the image to be detected can not only maintain a high accuracy for text recognition in common natural scenes, but also ensure a high accuracy and recall rate for vertical text detection.

[0195] In practical applications, when using the above text detection model to perform text detection on an image to be detected, there may still be a situation where some text in the image to be detected (too large spacing, too large angle) cannot be detected. In order to improve the comprehensiveness and accuracy of text detection, the embodiments of the present disclosure can also post-process the multiple text detection boxes included in the above text detection results.

[0196] A post-processing operation of the embodiment of the present disclosure may be: determining a target text detection frame corresponding to the image to be detected from multiple text detection frames in a preset manner.

[0197] Figure 6 FIG. 1 is a flow chart showing a method of determining a target text detection frame corresponding to an image to be detected according to an exemplary embodiment. Figure 6 As shown, after performing text detection processing on the above-mentioned image to be detected based on the text detection model to obtain the text detection result corresponding to the above-mentioned image to be detected, the following steps may also be included:

[0198] In step S31, a preset number of candidate text detection frames whose probability of containing text is greater than a preset probability threshold are obtained from the above-mentioned multiple text detection frames.

[0199] In step S32, first size information corresponding to each of the preset number of candidate text detection frames is determined.

[0200] In step S33, the preset number of candidate text detection frames are sorted in descending order according to the corresponding first size information to obtain a first sequence.

[0201] In step S34, the candidate text detection frame ranked first in the first sequence is used as the current candidate text detection frame, and the current candidate text detection frame is migrated from the first sequence to the candidate set.

[0202] In step S35, other candidate text detection frames whose IoU ratio with the current candidate text detection frame is greater than a preset IoU ratio threshold are determined from the first sequence.

[0203] In step S36, the other candidate text detection boxes are filtered.

[0204] In step S37, the step of repeatedly using the candidate text detection frame ranked first in the above-mentioned first sequence as the current candidate text detection frame, and migrating the above-mentioned current candidate text detection frame from the above-mentioned first sequence to the candidate set, to the step of filtering the above-mentioned other candidate text detection frames, until the above-mentioned first sequence is empty, and when the above-mentioned first sequence is empty, the candidate text detection frame in the above-mentioned candidate set is used as the target text detection frame corresponding to the above-mentioned image to be detected.

[0205] In summary, the disclosed embodiment can determine the above-mentioned target text detection frame from multiple text detection frames based on the probability that each of the multiple text detection frames contains text, the first size information, and the intersection and union ratio between each other. Specifically, a preset number of candidate text detection frames whose probability of containing text is greater than a preset probability threshold can be obtained from the multiple text detection frames included in the text detection results. Then, the preset number of candidate text detection frames are sorted in descending order according to their respective corresponding first size information to obtain a first sequence. Optionally, the first size information can be area information (the product of width information and height information), size information (width information, height information, the ratio between width information and height information, etc.), etc.

[0206] Assume that the first sequence is: candidate text detection frame 1—select candidate text detection frame 2—select candidate text detection frame 3—select candidate text detection frame 4—select candidate text detection frame 5. Then in the above step S34, the candidate text detection frame 1 can be used as the current candidate text detection frame, and the candidate text detection frame 1 can be migrated from the above first sequence to the above candidate set. In the above step S35, the intersection-over-Union (IOU) of the selected candidate text detection frame 2, the selected candidate text detection frame 3, the selected candidate text detection frame 4 and the selected candidate text detection frame 5 with the candidate text detection frame 1 is calculated respectively, so as to determine other candidate text detection frames (assuming that the selected candidate text detection frame 4) whose IOU is greater than the preset intersection-over-Union threshold (for example, 0.75), then in the above step S36, the other candidate text detection frames (assuming that the selected candidate text detection frame 4) are deleted. Then in step S37, repeat the above steps S34-S36. Since candidate text detection frame 1 has been migrated to the above candidate set, the candidate text detection frame ranked first at this time is candidate text detection frame 2. Then candidate text detection frame 2 is selected as the current candidate text detection frame, and candidate text detection frame 2 is migrated from the above first sequence to the above candidate set. The IOU of candidate text detection frame 3 and candidate text detection frame 5 with candidate text detection frame 2 are calculated respectively, so as to determine other candidate text detection frames (assuming candidate text detection frame 3) whose IOU is greater than the preset intersection-union ratio threshold, and then delete other candidate text detection frames (assuming candidate text detection frame 3), and so on, until the first sequence is empty, and finally the candidate text detection frame in the above candidate set is used as the target text detection frame corresponding to the image to be detected.

[0207] Among them, IOU is the ratio of the area of ​​the intersection of two bounding boxes to the area of ​​the union of the two bounding boxes, and the calculation formula is as follows:

[0208]

[0209] Since the multiple text detection frames included in the text detection results may have a large overlap, thereby affecting the accuracy of the text detection results, the above steps S31-S37 can effectively remove the overlapping parts between the multiple text detection frames, so as to find the best target detection frame, and also have a good detection rate for texts with large spacing and angles, thereby improving the comprehensiveness and accuracy of the text detection results.

[0210] After determining the above-mentioned target text detection frame, the embodiment of the present disclosure may further perform post-processing operations on the vertical text detection frame of vertical text detection, thereby determining a set of vertical text detection frames corresponding to the image to be detected from the target text detection frame.

[0211] Figure 7 FIG. 1 is a flow chart showing a method of determining a set of vertical text detection frames corresponding to an image to be detected according to an exemplary embodiment. Figure 7 As shown, if there are multiple target text detection frames, when the first sequence is empty, after taking the candidate text detection frames in the candidate set as the target text detection frames corresponding to the image to be detected, the following steps may also be included:

[0212] In step S41, second size information corresponding to each of the plurality of target text detection frames is determined.

[0213] In step S42, a preset number of target text detection frames whose second size information meets a preset condition are used as a preset number of vertical text detection frames.

[0214] In step S43, the horizontal coordinate information corresponding to each of the preset number of vertical text detection frames is determined.

[0215] In step S44, the preset number of vertical text detection frames are sorted in ascending order according to their corresponding horizontal coordinate information to obtain a second sequence.

[0216] In step S45, each vertical text detection frame in the second sequence is traversed in sequence, and the following operations are performed when each vertical text detection frame is traversed.

[0217] In step S46, other vertical text detection frames corresponding to the current vertical text detection frame currently traversed are determined, and the other vertical text detection frames corresponding to the above-mentioned current vertical text detection frame are vertical text detection frames in the above-mentioned second sequence that satisfy a preset size relationship and a preset position relationship with the above-mentioned current vertical text detection frame.

[0218] In step S47, a set of vertical text detection frames corresponding to the image to be detected is determined according to the current vertical text detection frame and the other corresponding vertical text detection frames.

[0219] The multiple target text detection frames determined through the above steps S31 to S37 may include horizontal text detection frames and vertical text detection frames. The embodiment of the present disclosure can determine a vertical text detection frame set from the multiple target text detection frames based on the second size information, horizontal coordinate information corresponding to each of the multiple target text detection frames, and the size relationship and position relationship between the target text detection frames.

[0220] Exemplarily, the second size information in the above step S41 may be a ratio of width information to height information, or may be a ratio of height information to width information.

[0221] Exemplarily, when the second size information is the ratio of width information to height information, the preset condition in the above step S42 may be "the ratio of width information to height information is less than a preset aspect ratio threshold (e.g., 1.5)". When the second size information is the ratio of height information to width information, the preset condition in the above step S42 may be "the ratio of height information to width information is greater than a preset aspect ratio threshold".

[0222] Exemplarily, the horizontal coordinate information in the above step S43 may be the horizontal coordinate information of the lower left corner or the upper left corner of the vertical text detection frame. It should be noted that the horizontal coordinate information of the upper left corner and the lower left corner of the text detection frame are equal.

[0223] Exemplarily, the "preset size relationship" in the above step S46 may represent the relationship between the size information of the current vertical text detection frame and the size information of the corresponding other vertical text detection frames. Optionally, the "preset size relationship" may be the ratio of the width information between the two (the current vertical text detection frame and the corresponding other vertical text detection frames). Specifically, the "preset size relationship" may be the ratio between the larger width and the smaller width of the two.

[0224] Exemplarily, when the "preset size relationship" is the ratio between the larger width and the smaller width between the two, the "preset size relationship" may be greater than the preset width threshold. Assuming that the width of the current vertical text detection box is w1, and the width of the corresponding other vertical text detection box is w2, and w1 is greater than w2, then the ratio between w1 and w2 is greater than the preset width threshold.

[0225] Exemplarily, the "preset position relationship" in the above step S46 can characterize the relationship between the position information of the current vertical text detection frame and the position information of the corresponding other vertical text detection frames. Optionally, the "preset position relationship" can be the height information of the two (the current vertical text detection frame and the corresponding other vertical text detection frames) in the image to be detected. Specifically, the "preset position relationship" can be the difference in height information of the two in the image to be detected, which is less than the height information of any one of the two detection frames in the image to be detected. Assuming that the height information of the current vertical text detection frame in the image to be detected is h1, and the height information of the corresponding other vertical text detection frames in the image to be detected is h2, then the difference between h1 and h2 is less than h1 or h2.

[0226] Assume that the second sequence obtained in the above step S44 is: vertical text detection frame 1 - vertical text detection frame 2 - vertical text detection frame 3 - vertical text detection frame 4 - vertical text detection frame 5.

[0227] In the above steps S45 to S47, the above second sequence can be traversed in sequence, that is, first traverse the vertical text detection frame 1, and determine the vertical text detection frame that satisfies the above preset size relationship and the above preset position relationship with the vertical text detection frame 1 from the vertical text detection frame 2, the vertical text detection frame 3, the vertical text detection frame 4, and the vertical text detection frame 5. Assuming that the vertical text detection frame 3 satisfies the condition, the vertical text detection frame 1 and the vertical text detection frame 3 are combined into a set of vertical text detection frame sets (that is, a set including the vertical text detection frame 1 and the vertical text detection frame 3); then traverse the vertical text detection frame 1, and then determine the vertical text detection frame 2, the vertical text detection frame 3, the vertical text detection frame 4, and the vertical text detection frame 5. Text detection frame 2, determine the vertical text detection frame that satisfies the above-mentioned preset size relationship and the above-mentioned preset position relationship with the vertical text detection frame 2 from vertical text detection frame 1, vertical text detection frame 3, vertical text detection frame 4, and vertical text detection frame 5. Assuming that vertical text detection frame 4 satisfies the condition, vertical text detection frame 2 and vertical text detection frame 4 are combined into a group of vertical text detection frame sets (that is, a set including vertical text detection frame 2 and vertical text detection frame 4), and so on, until each vertical text detection frame in the second sequence is traversed, thereby obtaining multiple groups of vertical text detection frame sets.

[0228] It should be noted that if during the traversal process, a current vertical text detection frame does not detect other vertical text detection frames that satisfy the above-mentioned preset size relationship and preset position relationship, the current vertical text detection frame can be placed separately in a group of vertical text detection frame sets.

[0229] It should be noted that among the multiple sets of vertical text detection frame sets obtained by the above traversal method, there may be overlapping vertical text detection frame sets. In order to improve the accuracy of the final vertical text determination, the overlapping vertical text detection frame sets can be deduplicated.

[0230] Exemplarily, the above process of determining the set of vertical text detection frames corresponding to the image to be detected can also be performed directly after the above step S22, that is, the operations of the above steps S41 to S47 are directly performed on the multiple text detection frames contained in the text detection results obtained in the above step S22 (that is, the target text detection frame in steps S41 to S47 is directly replaced with the multiple text detection frames contained in the text detection results), so as to directly determine the set of vertical text detection frames corresponding to the image to be detected from the multiple text detection frames.

[0231] Since the size information (for example, aspect ratio information) of horizontal text is different from that of vertical text, the disclosed embodiment can quickly and accurately determine a preset number of vertical text detection frames from multiple target text detection frames through the above steps S41-S42, thereby improving the timeliness and accuracy of vertical text detection. In addition, since multiple vertical text detection frames may be determined for a complete vertical text during the detection process, in order to accurately determine multiple vertical text detection frames belonging to a complete vertical text and further improve the accuracy of vertical text detection, the disclosed embodiment combines several vertical text detection frames of preset size relationship and preset position relationship into a group of vertical text detection frame sets through the above steps S43-S47, and each vertical text detection frame included in each group of vertical text detection frame sets corresponds to a complete vertical text.

[0232] After determining the vertical text detection frame set, the embodiment of the present disclosure may further perform post-processing operations on the vertical text detection frame set, thereby determining the vertical text detection frame that ultimately corresponds to the image to be detected.

[0233] Figure 8 FIG. 1 is a flow chart showing a method of determining a vertical text detection frame corresponding to an image to be detected according to an exemplary embodiment. Figure 8 As shown, the above vertical text detection frame set is a plurality of groups. After determining a group of vertical text detection frame sets corresponding to the above image to be detected according to the above current vertical text detection frame and the above corresponding target vertical text detection frame, the following steps may also be included:

[0234] In step S51, the vertical text detection frames included in each of the plurality of vertical text detection frame sets are connected to obtain target vertical text detection frames corresponding to each of the plurality of vertical text detection frame sets.

[0235] In step S52, the target vertical text detection frames corresponding to the plurality of sets of vertical text detection frames are used as the vertical text detection frames corresponding to the image to be detected.

[0236] For example, the connectivity in step S51 may be: taking the maximum connectivity graph or the minimum bounding rectangle of the vertical text detection frames included in each set of vertical text detection frames. For the text in the vertical text detection frames, the connectivity may be: fusing the text in each vertical text detection frame in the same vertical text detection frame set.

[0237] It should be noted that if a certain set of vertical text detection frame sets includes only one vertical text detection frame, the vertical text detection frame can be directly used as the target vertical text detection frame corresponding to the set of vertical text detection frame sets.

[0238] Exemplarily, in the above step S52, the target vertical text detection frame obtained by the connection can be used as the vertical text detection frame that finally corresponds to the above image to be detected.

[0239] Fig. 9 The diagram is a schematic diagram showing a vertical text detection frame in a vertical text detection frame set before and after being connected according to an exemplary embodiment. Fig. 9 (a) is the schematic diagram before connection. Fig. 9 (b) is a schematic diagram after connection. Fig. 9 As shown in (a), before the connection, the vertical text detection frame A, the vertical text detection frame B and the vertical text detection frame C determined by the above steps S41 to S47 form a set of vertical text detection frames. After the vertical text detection frame A, the vertical text detection frame B and the vertical text detection frame C are connected in the above step S51, the vertical text detection frame A, the vertical text detection frame B and the vertical text detection frame C are obtained. Fig. 9 (b) is a schematic diagram of the connected target vertical text detection box.

[0240] The embodiment of the present disclosure performs a connection operation on multiple vertical text detection frames of a complete vertical text through the above step S51, thereby further improving the accuracy of vertical text detection.

[0241] Verified by experimental results, compared with the model that does not perform post-processing of the detection frame, the embodiment of the present disclosure performs the above-mentioned post-processing on the multiple text detection frames included in the text detection results, so that the MAP corresponding to the text detection results is improved by 4.9 percentage points. In addition, the text detection model provided by the embodiment of the present disclosure is a general text detection algorithm that supports vertical text detection. While maintaining high-precision detection of text in common natural scenes, it also has a high recall rate and accuracy rate for vertical text detection. By detecting vertical text, more accurate candidate frames can be provided for downstream recognition tasks, thereby achieving recognition accuracy and better serving other business scenarios.

[0242] Fig.10 is a block diagram of a training device for a text detection model according to an exemplary embodiment. Fig.10 The device may include a target sample image set acquisition module 61, a rotation module 62, a target position information determination module 63, a training module 64, a loss data determination module 65 and a text detection model determination module 66.

[0243] The target sample image set acquisition module 61 is configured to execute acquisition of a target sample image set, wherein the target sample image set includes target sample images with text box labels.

[0244] The rotation module 62 is configured to rotate the target sample image to obtain a rotated target sample image.

[0245] The target position information determining module 63 is configured to determine the target position information of the text box in the rotated target sample image.

[0246] The training module 64 is configured to perform text detection training on the neural network based on the rotated target sample image and the target position information to obtain a text prediction result corresponding to the target sample image.

[0247] The loss data determination module 65 is configured to determine the loss data according to the text prediction result and the text box label.

[0248] The text detection model determination module 66 is configured to execute training of the neural network based on the loss data to obtain a text detection model.

[0249] In an exemplary embodiment, the training device of the above text detection model may further include:

[0250] The candidate sample image set acquisition module is configured to execute acquisition of a candidate sample image set, wherein the candidate sample image set includes candidate sample images.

[0251] The text detection module is configured to perform text detection on the candidate sample image to obtain a candidate sample image annotated with the text box label.

[0252] The target candidate sample image acquisition module is configured to execute, according to the above text box label, acquiring a target candidate sample image containing a vertical text box from the above candidate sample image set.

[0253] The oversampling module is configured to perform oversampling on the target candidate sample image to obtain an oversampled target candidate sample image.

[0254] The editing module is configured to execute a vertical text editing instruction in response to being triggered based on the above-mentioned oversampled target candidate sample image, edit a preset number of vertical texts on the above-mentioned oversampled target candidate sample image, and obtain an edited target candidate sample image.

[0255] The target sample image set determination module is configured to determine the target sample image set based on other candidate sample images in the candidate sample image set except the target candidate sample image, and the edited target candidate sample image, wherein the target sample image includes the other candidate sample images and the edited target candidate sample image.

[0256] In an exemplary embodiment, the target location information determination module 63 may include:

[0257] The original position information determining unit is configured to determine the original position information of the text box based on the text box label.

[0258] The offset information determining unit is configured to determine the offset information of the original position information during the rotation process.

[0259] The transformation unit is configured to perform a linear transformation on the original position information to obtain the transformed original position information.

[0260] The target position information determining unit is configured to determine the target position information based on the transformed original position information and the offset information.

[0261] In an exemplary embodiment, there are multiple target sample images, and the rotation module 62 may include:

[0262] The rotation center determining unit is configured to determine the rotation centers corresponding to each of the plurality of target sample images.

[0263] The target sample image determination unit is configured to rotate the above-mentioned multiple target sample images by corresponding angles along their respective corresponding rotation directions based on the rotation centers respectively corresponding to the above-mentioned multiple target sample images, so as to obtain multiple rotated target sample images.

[0264] Fig.11 is a block diagram of a text detection device according to an exemplary embodiment. Fig.11 The device may include a to-be-detected image acquisition module 71 and a text detection result determination module 72.

[0265] The to-be-detected image acquisition module 71 is configured to acquire the to-be-detected image.

[0266] The text detection result determination module 72 is configured to perform text detection processing on the above-mentioned image to be detected based on a text detection model to obtain a text detection result corresponding to the above-mentioned image to be detected; wherein the above-mentioned text detection model is trained using the training method of the text detection model described in any of the above-mentioned embodiments.

[0267] In an exemplary embodiment, the text detection result includes a plurality of text detection frames and the probability that each of the plurality of text detection frames contains text, and the text detection device may further include:

[0268] The candidate text detection frame acquisition module is configured to execute, from the above-mentioned multiple text detection frames, obtaining a preset number of candidate text detection frames whose probability of containing text is greater than a preset probability threshold.

[0269] The first size information determination module is configured to determine the first size information corresponding to each of the preset number of candidate text detection frames.

[0270] The first sequence acquisition module is configured to sort the preset number of candidate text detection frames in descending order according to corresponding first size information to obtain a first sequence.

[0271] The migration module is configured to execute the execution of taking the candidate text detection frame ranked first in the above-mentioned first sequence as the current candidate text detection frame, and migrating the above-mentioned current candidate text detection frame from the above-mentioned first sequence to the candidate set.

[0272] The module for determining other candidate text detection frames is configured to determine, from the first sequence, other candidate text detection frames whose intersection-over-union ratio with the current candidate text detection frame is greater than a preset intersection-over-union ratio threshold.

[0273] The filtering module is configured to filter the other candidate text detection boxes.

[0274] The target text detection frame determination module is configured to repeatedly perform the steps of taking the candidate text detection frame ranked first in the above-mentioned first sequence as the current candidate text detection frame, and migrating the above-mentioned current candidate text detection frame from the above-mentioned first sequence to the candidate set, to filtering the above-mentioned other candidate text detection frames, until the above-mentioned first sequence is empty, and when the above-mentioned first sequence is empty, taking the candidate text detection frame in the above-mentioned candidate set as the target text detection frame corresponding to the above-mentioned image to be detected.

[0275] In an exemplary embodiment, if there are multiple target text detection frames, the text detection device may further include:

[0276] The second size information determination module is configured to determine the second size information corresponding to each of the plurality of target text detection frames.

[0277] The vertical text detection frame determination module is configured to execute a preset number of target text detection frames whose second size information meets a preset condition as a preset number of vertical text detection frames.

[0278] The horizontal coordinate information determination module is configured to determine the horizontal coordinate information corresponding to each of the preset number of vertical text detection boxes.

[0279] The second sequence determination module is configured to sort the preset number of vertical text detection boxes in ascending order according to their corresponding horizontal coordinate information to obtain a second sequence.

[0280] The traversal module is configured to sequentially traverse each vertical text detection frame in the second sequence, and perform the following operations when traversing each vertical text detection frame.

[0281] The other vertical text detection frame determination module is configured to determine other vertical text detection frames corresponding to the current vertical text detection frame currently traversed, and the other vertical text detection frames corresponding to the above-mentioned current vertical text detection frame are vertical text detection frames in the above-mentioned second sequence that satisfy a preset size relationship and a preset position relationship with the above-mentioned current vertical text detection frame.

[0282] The detection frame set determination module is configured to determine a set of vertical text detection frames corresponding to the above-mentioned image to be detected based on the above-mentioned current vertical text detection frame and the above-mentioned corresponding other vertical text detection frames.

[0283] In an exemplary embodiment, the vertical text detection frame set is a plurality of groups, and the text detection device may further include:

[0284] The connection module is configured to perform connection processing on the vertical text detection frames contained in each of the multiple sets of vertical text detection frame sets to obtain the target vertical text detection frames corresponding to each of the multiple sets of vertical text detection frame sets.

[0285] The determination module is configured to execute the target vertical text detection frame corresponding to each of the above-mentioned multiple groups of vertical text detection frame sets as the vertical text detection frame corresponding to the above-mentioned image to be detected.

[0286] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0287] In an exemplary embodiment, an electronic device is also provided, comprising a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the steps of any text detection model training method or any text detection method in the above-mentioned embodiments when executing the instructions stored in the memory.

[0288] The electronic device may be a terminal, a server or a similar computing device. For example, the electronic device is a server. Fig.12It is a block diagram of an electronic device for text detection according to an exemplary embodiment. The electronic device 80 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 81 (the central processing unit 81 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA), a memory 83 for storing data, and one or more storage media 82 (such as one or more mass storage devices) for storing application programs 823 or data 822. Among them, the memory 83 and the storage medium 82 can be short-term storage or permanent storage. The program stored in the storage medium 82 may include one or more modules, each of which may include a series of instruction operations in the electronic device. Furthermore, the central processing unit 81 can be configured to communicate with the storage medium 82 to execute a series of instruction operations in the storage medium 82 on the electronic device 80. The electronic device 80 may also include one or more power supplies 86, one or more wired or wireless network interfaces 85, one or more input and output interfaces 84, and / or one or more operating systems 821, such as Windows ServerTM, Mac OS XTM, UnixTM, LinuxTM, FreeBSDTM, etc.

[0289] The input / output interface 84 may be used to receive or send data via a network. The specific example of the network may include a wireless network provided by a communication provider of the electronic device 80. In one example, the input / output interface 84 includes a network adapter (Network Interface Controller, NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In an exemplary embodiment, the input / output interface 84 may be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0290] It can be understood by those skilled in the art that Fig.12 The structure shown is only for illustration and does not limit the structure of the above electronic device. Fig.12 More or fewer components as shown, or with Fig.12 Different configurations are shown.

[0291] In an exemplary embodiment, a computer-readable storage medium is also provided. When the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device can execute the steps of any text detection model training method or any text detection method in the above-mentioned embodiments.

[0292] In an exemplary embodiment, a computer program product is also provided, including a computer program, which, when executed by a processor, implements the text detection model training method or text detection method provided in any of the above-mentioned embodiments.

[0293] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present disclosure can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0294] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0295] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A training method for a text detection model, characterized in that: include: Acquire a target sample image set, wherein the target sample image set includes target sample images with text box labels; Rotating the target sample image to obtain a rotated target sample image; Determining target position information of a text box in the rotated target sample image; Based on the rotated target sample image and the target position information, performing text detection training on a neural network to obtain a text prediction result corresponding to the target sample image; Determining loss data according to the text prediction result and the text box label; The neural network is trained based on the loss data to obtain a text detection model; the text detection model is used to perform text detection processing on the image to be detected to obtain a text detection result; the text detection result includes multiple text detection boxes; Determining a preset number of vertical text detection frames from the plurality of text detection frames; Combining a plurality of the vertical text detection frames having a preset size relationship and a preset position relationship into a set of vertical text detection frame sets, to obtain a plurality of sets of vertical text detection frame sets; Connecting the vertical text detection frames contained in each of the plurality of sets of vertical text detection frames to obtain a vertical text detection frame corresponding to the image to be detected.

2. The training method of the text detection model according to claim 1, characterized in that: Before acquiring the target sample image set, the text detection model training method further includes: Acquire a candidate sample image set, wherein the candidate sample image set includes candidate sample images; Performing text detection on the candidate sample image to obtain a candidate sample image annotated with the text box label; According to the text box label, obtaining a target candidate sample image containing a vertical text box from the candidate sample image set; Oversampling the target candidate sample image to obtain an oversampled target candidate sample image; In response to a vertical text editing instruction triggered based on the oversampled target candidate sample image, editing a preset number of vertical texts on the oversampled target candidate sample image to obtain an edited target candidate sample image; The target sample image set is determined based on other candidate sample images in the candidate sample image set except the target candidate sample image, and the edited target candidate sample image, wherein the target sample image includes the other candidate sample images and the edited target candidate sample image.

3. The training method of the text detection model according to claim 1 or 2, characterized in that: The determining target position information of the text box in the rotated target sample image includes: Based on the text box label, determining the original position information of the text box; Determining offset information of the original position information during the rotation process; Performing a linear transformation on the original position information to obtain transformed original position information; The target position information is determined based on the transformed original position information and the offset information.

4. The training method of the text detection model according to claim 1 or 2, characterized in that: If there are multiple target sample images, rotating the target sample images to obtain rotated target sample images includes: Determine the rotation center corresponding to each of the plurality of target sample images; Based on the rotation centers corresponding to the multiple target sample images, the multiple target sample images are rotated along the corresponding rotation directions by corresponding angles to obtain the multiple rotated target sample images.

5. A text detection method, characterized in that: include: Acquire the image to be detected; Performing text detection processing on the image to be detected based on the text detection model to obtain a text detection result corresponding to the image to be detected; The text detection model is trained using the text detection model training method described in any one of claims 1 to 4.

6. The text detection method according to claim 5, characterized in that: The text detection result includes multiple text detection frames and the probability that each of the multiple text detection frames contains text. After performing text detection processing on the image to be detected based on the text detection model to obtain the text detection result corresponding to the image to be detected, the text detection method further includes: From the multiple text detection frames, obtaining a preset number of candidate text detection frames whose probability of containing text is greater than a preset probability threshold; Determine first size information corresponding to each of the preset number of candidate text detection frames; Sort the preset number of candidate text detection frames in descending order according to the corresponding first size information to obtain a first sequence; Taking the candidate text detection frame ranked first in the first sequence as the current candidate text detection frame, and migrating the current candidate text detection frame from the first sequence to the candidate set; Determine, from the first sequence, other candidate text detection frames whose intersection-over-union ratio with the current candidate text detection frame is greater than a preset intersection-over-union ratio threshold; Filtering the other candidate text detection frames; Repeat the steps of using the candidate text detection frame that is ranked first in the first sequence as the current candidate text detection frame, and migrating the current candidate text detection frame from the first sequence to the candidate set, to filtering the other candidate text detection frames, until the first sequence is empty, and when the first sequence is empty, using the candidate text detection frame in the candidate set as the target text detection frame corresponding to the image to be detected.

7. The text detection method according to claim 6, characterized in that: If there are multiple target text detection frames, then when the first sequence is empty, after taking the candidate text detection frames in the candidate set as the target text detection frames corresponding to the image to be detected, the text detection method further includes: Determine second size information corresponding to each of the plurality of target text detection frames; Using a preset number of target text detection frames whose second size information meets a preset condition as a preset number of vertical text detection frames; Determine the horizontal coordinate information corresponding to each of the preset number of vertical text detection frames; Sort the preset number of vertical text detection frames in ascending order according to their corresponding horizontal coordinate information to obtain a second sequence; Each vertical text detection frame in the second sequence is traversed in sequence. When traversing each vertical text detection frame, the following operations are performed: Determine other vertical text detection frames corresponding to the current vertical text detection frame currently traversed, wherein the other vertical text detection frames corresponding to the current vertical text detection frame are vertical text detection frames in the second sequence that satisfy a preset size relationship and a preset position relationship with the current vertical text detection frame; A set of vertical text detection frames corresponding to the image to be detected is determined according to the current vertical text detection frame and the corresponding other vertical text detection frames.

8. The text detection method according to claim 7, characterized in that: The vertical text detection frame set is a plurality of sets, then after determining a set of vertical text detection frame sets corresponding to the image to be detected according to the current vertical text detection frame and the corresponding target vertical text detection frame, the text detection method further includes: Connecting the vertical text detection frames included in each of the multiple sets of vertical text detection frame sets to obtain target vertical text detection frames corresponding to each of the multiple sets of vertical text detection frame sets; The target vertical text detection frames corresponding to the multiple groups of vertical text detection frame sets are used as the vertical text detection frames corresponding to the image to be detected.

9. A training device for a text detection model, characterized in that: include: A target sample image set acquisition module is configured to execute acquisition of a target sample image set, wherein the target sample image set includes target sample images with text box labels; A rotation module is configured to rotate the target sample image to obtain a rotated target sample image; A target position information determination module, configured to determine the target position information of the text box in the rotated target sample image; A training module is configured to perform text detection training on a neural network based on the rotated target sample image and the target position information to obtain a text prediction result corresponding to the target sample image; A loss data determination module, configured to determine loss data according to the text prediction result and the text box label; A text detection model determination module is configured to execute training of the neural network based on the loss data to obtain a text detection model; the text detection model is used to perform text detection processing on the image to be detected to obtain a text detection result; the text detection result includes a plurality of text detection boxes; A vertical text detection frame determination module is configured to determine a preset number of vertical text detection frames from the plurality of text detection frames; a detection frame set determination module, configured to group a plurality of the vertical text detection frames having a preset size relationship and a preset position relationship into a group of vertical text detection frame sets, to obtain a plurality of groups of vertical text detection frame sets; The determination module is configured to perform connectivity processing on the vertical text detection frames respectively included in the plurality of sets of vertical text detection frame sets to obtain the vertical text detection frame corresponding to the image to be detected.

10. The text detection model training device according to claim 9, characterized in that: The training device of the text detection model also includes: A candidate sample image set acquisition module is configured to execute acquisition of a candidate sample image set, wherein the candidate sample image set includes candidate sample images; A text detection module is configured to perform text detection on the candidate sample image to obtain a candidate sample image annotated with the text box label; A target candidate sample image acquisition module is configured to acquire a target candidate sample image containing a vertical text box from the candidate sample image set according to the text box label; An oversampling module is configured to perform oversampling on the target candidate sample image to obtain an oversampled target candidate sample image; an editing module, configured to execute a vertical text editing instruction triggered based on the oversampled target candidate sample image, edit a preset number of vertical texts on the oversampled target candidate sample image, and obtain an edited target candidate sample image; The target sample image set determination module is configured to determine the target sample image set based on other candidate sample images in the candidate sample image set except the target candidate sample image, and the edited target candidate sample image, wherein the target sample image includes the other candidate sample images and the edited target candidate sample image.

11. The text detection model training device according to claim 9 or 10, characterized in that: The target location information determination module includes: an original position information determining unit, configured to determine the original position information of the text box based on the text box label; an offset information determining unit, configured to determine the offset information of the original position information during the rotation process; a transformation unit, configured to perform a linear transformation on the original position information to obtain transformed original position information; The target position information determining unit is configured to determine the target position information based on the transformed original position information and the offset information.

12. The text detection model training device according to claim 9 or 10, characterized in that: If there are multiple target sample images, the rotation module includes: A rotation center determination unit, configured to determine the rotation center corresponding to each of the plurality of target sample images; The target sample image determination unit is configured to rotate the multiple target sample images by corresponding angles along the respective corresponding rotation directions based on the respective corresponding rotation centers of the multiple target sample images to obtain the multiple rotated target sample images.

13. A text detection device, characterized in that: include: The module for acquiring the image to be detected is configured to execute the acquisition of the image to be detected; The text detection result determination module is configured to perform text detection processing on the image to be detected based on a text detection model to obtain a text detection result corresponding to the image to be detected; wherein the text detection model is trained using the training method of the text detection model described in any one of claims 1 to 4.

14. The text detection device according to claim 13, characterized in that: The text detection result includes a plurality of text detection frames and the probability that each of the plurality of text detection frames contains text, and the text detection device further includes: A candidate text detection frame acquisition module is configured to acquire a preset number of candidate text detection frames whose probability of containing text is greater than a preset probability threshold from the multiple text detection frames; A first size information determination module is configured to determine first size information corresponding to each of the preset number of candidate text detection frames; A first sequence acquisition module is configured to sort the preset number of candidate text detection frames in descending order according to corresponding first size information to obtain a first sequence; A migration module is configured to execute the process of taking the candidate text detection frame that is ranked first in the first sequence as the current candidate text detection frame, and migrating the current candidate text detection frame from the first sequence to the candidate set; Another candidate text detection frame determination module is configured to determine, from the first sequence, another candidate text detection frame whose intersection-over-union ratio with the current candidate text detection frame is greater than a preset intersection-over-union ratio threshold; A filtering module, configured to filter the other candidate text detection frames; The target text detection frame determination module is configured to execute the steps of repeatedly using the candidate text detection frame sorted first in the first sequence as the current candidate text detection frame, migrating the current candidate text detection frame from the first sequence to the candidate set, and filtering the other candidate text detection frames until the first sequence is empty, and when the first sequence is empty, using the candidate text detection frame in the candidate set as the target text detection frame corresponding to the image to be detected.

15. The text detection device according to claim 14, characterized in that: If there are multiple target text detection frames, the text detection device further includes: A second size information determination module is configured to determine the second size information corresponding to each of the plurality of target text detection frames; A vertical text detection frame determination module is configured to execute a preset number of target text detection frames whose second size information meets a preset condition as a preset number of vertical text detection frames; A horizontal coordinate information determination module is configured to determine the horizontal coordinate information corresponding to each of the preset number of vertical text detection frames; A second sequence determination module is configured to sort the preset number of vertical text detection frames in ascending order according to their corresponding horizontal coordinate information to obtain a second sequence; The traversal module is configured to sequentially traverse each vertical text detection frame in the second sequence, and perform the following operations when traversing each vertical text detection frame: The other vertical text detection frame determination module is configured to determine other vertical text detection frames corresponding to the current vertical text detection frame currently traversed, wherein the other vertical text detection frames corresponding to the current vertical text detection frame are vertical text detection frames in the second sequence that satisfy a preset size relationship and a preset position relationship with the current vertical text detection frame; The detection frame set determination module is configured to determine a set of vertical text detection frame sets corresponding to the image to be detected based on the current vertical text detection frame and the corresponding other vertical text detection frames.

16. The text detection device according to claim 15, characterized in that: The vertical text detection frame set is a plurality of groups, and the text detection device further includes: a connection module configured to perform connection processing on the vertical text detection frames respectively included in the plurality of sets of vertical text detection frame sets to obtain target vertical text detection frames respectively corresponding to the plurality of sets of vertical text detection frame sets; The determination module is configured to execute the target vertical text detection frame corresponding to each of the plurality of groups of vertical text detection frame sets as the vertical text detection frame corresponding to the image to be detected.

17. An electronic device, characterized in that: include: processor; a memory for storing instructions executable by the processor; The processor is configured to execute the instructions to implement the text detection model training method as described in any one of claims 1 to 4 or the text detection method as described in any one of claims 5 to 8.

18. A computer-readable storage medium, when the instructions in the computer-readable storage medium are executed by a processor of an electronic device, the electronic device executes the training method of the text detection model as described in any one of claims 1 to 4 or the text detection method as described in any one of claims 5 to 8.

19. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the training method of the text detection model described in any one of claims 1 to 4 or the text detection method described in any one of claims 5 to 8 is implemented.

Images