Deep learning-based curved text detection and recognition method and system

Abstract

The invention discloses a curved text detection and recognition method and system based on deep learning, and belongs to the technical field of artificial intelligence and computer vision, and the method comprises the steps: building a text detection model and a text recognition model; respectively training the text detection model and the text recognition model to correspondingly obtain a trained text detection model and a trained text recognition model; performing position marking on the text area based on the trained text detection model to generate a marking result; recognizing and extracting the text content based on the trained text recognition model to generate a recognition result; and generating text content based on the marking result and the recognition result. The invention provides a deep learning model fusing feature enhancement and image correction, which comprises a text detection model and a text recognition model, and significantly improves the curved text positioning and content extraction capabilities in a complex scene.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and computer vision technology, and specifically relates to a method and system for detecting and recognizing curved text based on deep learning. Background Art

[0002] Text is one of the core carriers of information dissemination. Text in natural scenes (such as street signs and product packaging) contains rich semantic information. Efficiently extracting and utilizing this text will greatly enhance our daily lives and industrial production. With the advancement of mobile device imaging capabilities, the number of natural scene images is increasing. However, traditional text detection and recognition techniques are only suitable for neatly printed documents and struggle with complex scenes. To overcome this limitation, research has gradually shifted its focus to detecting and recognizing text in natural scenes.

[0003] Text detection and recognition in natural scenes falls within the field of computer vision and can draw upon methods from image analysis and pattern recognition. Existing methods can be categorized as traditional and deep learning-based. Traditional text detection methods rely on manually designed features for text detection and can be categorized as connected domain-based (e.g., MESR, SWT) and sliding window-based methods. Deep learning-based text detection methods are primarily divided into two types: region-restructuring-based methods and segmentation-based methods. Region-restructuring-based methods (e.g., CTPN, EAST) draw on techniques related to object detection, while segmentation-based methods (e.g., PSENet, SAE) transform the text detection task into a pixel-level classification problem, detecting text by segmenting text regions. Traditional text recognition methods typically utilize manually designed features and related rules for recognition. Deep learning-based text recognition methods can be categorized, based on their decoding methods, as connectionist temporal classification-based methods (e.g., CRNN, DTRN) and attention-based methods (e.g., Transformer).

[0004] However, existing technologies suffer from the following issues: First, traditional methods are only suitable for simple, regular text detection tasks and struggle to effectively handle complex text that is curved, deformed, or undergoes perspective transformations in natural scenes. Second, classic deep learning text detection and recognition models fail to adequately account for the special case of curved text, failing to fully adapt to its shape characteristics and insufficiently extracting features from curved text. Summary of the Invention

[0005] The purpose of the embodiments of the present application is to provide a method for detecting and recognizing curved text based on deep learning, which can solve the problem that the existing technology is difficult to effectively process complex text that is curved, deformed or perspective transformed in natural scenes, and cannot fully fit the shape characteristics of the curved text, and the feature extraction of the curved text is insufficient.

[0006] In order to solve the above technical problems, this application is implemented as follows: In the first aspect, an embodiment of the present application provides a method for detecting and recognizing curved text based on deep learning, the method comprising: respectively building a text detection model and a text recognition model; respectively training the text detection model and the text recognition model to obtain a trained text detection model and a trained text recognition model; marking the position of the text area based on the trained text detection model to generate a marking result; identifying and extracting the text content based on the trained text recognition model to generate a recognition result; and generating text content based on the marking result and the recognition result.

[0007] As an optional implementation of the first aspect of the present application, the text detection model includes a contextual attention mechanism layer, a dynamic snake convolution layer, a feature fusion layer and a global attention mechanism layer, and the text recognition model includes a geometric correction layer, a color correction layer, an image block embedding layer, a partial convolution layer and a global block layer.

[0008] As an optional implementation of the first aspect of the present application, the process of generating a labeling result includes: performing a convolution operation on the input image to downsample it to generate a first feature map, a second feature map, a third feature map, and a fourth feature map, the size of the first feature map is half of the input image, the size of the second feature map is one-quarter of the input image, the size of the third feature map is one-eighth of the input image, and the size of the fourth feature map is one-sixteenth of the input image; performing convolution processing on the fourth feature map according to the dynamic snake convolution layer to generate a snake convolution feature; performing adaptive weighting processing on the snake convolution feature according to the context attention mechanism layer to generate an attention representation of the curved text feature; splicing the attention representation of the curved text feature with the second feature map, the third feature map, and the fourth feature map respectively to obtain a spliced ​​feature; performing feature fusion processing on the spliced ​​feature according to the global attention mechanism layer to generate a multi-scale fusion feature; performing probability map conversion and threshold map conversion on the multi-scale fusion feature respectively to generate a labeling result.

[0009] As an optional implementation of the first aspect of the present application, the specific process of generating an attention representation of a curved text feature includes: inputting the serpentine convolution features into three parallel distributed convolution layers for convolution processing to generate a first serpentine convolution feature, a second serpentine convolution feature, and a third serpentine convolution feature; transforming and transposing the first serpentine convolution feature to generate a query matrix, and transforming the second serpentine convolution feature to generate a key matrix; multiplying the key matrix and the query matrix and normalizing them to generate an attention matrix; and performing bitwise multiplication of the attention matrix and the third serpentine convolution feature to generate an attention representation of the curved text feature.

[0010] As an optional implementation of the first aspect of the present application, when feature fusion processing is performed on the splicing features according to the global attention mechanism layer to generate multi-scale fusion features, the global attention mechanism layer includes a channel attention layer and a spatial attention layer, wherein the specific process of generating multi-scale fusion features includes: according to the channel attention layer, channel weight calibration processing is performed on the splicing features to generate channel enhancement features; according to the spatial attention layer, spatial weight focusing processing is performed on the channel enhancement features to generate multi-scale fusion features.

[0011] As an optional implementation of the first aspect of the present application, the process of generating a recognition result includes: performing geometric correction and color correction processing on the attention representation of the curved text feature according to the image correction layer to generate a corrected image; performing image embedding processing, partial convolution processing, and global block processing on the corrected image according to the first image block embedding layer, the first partial convolution layer, and the first global block layer, and then merging them to generate a first correction result; performing partial convolution processing and global block processing on the first correction result according to the second image block embedding layer, the second partial convolution layer, and the second global block layer, and then merging them to generate a second correction result; performing partial convolution processing and global block processing on the second correction result according to the third partial convolution layer and the third global block layer, and then merging them to generate a third correction result. The third correction result is decoded to generate a recognition result.

[0012] As an optional implementation of the first aspect of the present application, the process of obtaining a trained text detection model includes: obtaining a probability map loss function and a binary map loss function based on binary cross entropy loss; obtaining a threshold map loss function based on the absolute difference between the threshold map prediction value and the threshold map label; constructing a loss function of the text detection model based on the probability map loss function, the binary map loss function and the threshold map loss function; training the text detection model based on the loss function of the text detection model to obtain a trained text detection model.

[0013] In a second aspect, an embodiment of the present application provides a curved text detection and recognition system based on deep learning, the system comprising: A text detection model generation module is used to generate a text detection model based on the contextual attention mechanism layer, dynamic snake convolution layer, feature fusion layer, and global attention mechanism layer; A text recognition model generation module is used to generate a text recognition model based on the geometry correction layer, color correction layer, image block embedding layer, partial convolution layer and global block layer; A training module is used to train the text detection module and the text recognition module respectively, and obtain a trained text detection module and a trained text recognition module respectively; A marking result generation module is used to mark the position of the text area using the trained text detection module to generate a marking result; The recognition result generation module is used to use the trained text recognition module to identify and extract text content to generate recognition results; The text content generation module is used to generate text content based on the marking results and recognition results.

[0014] In a third aspect, an embodiment of the present application provides an electronic device comprising a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of the method described in the first aspect are implemented.

[0015] In a fourth aspect, an embodiment of the present application provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the steps of the method described in the first aspect are implemented.

[0016] Compared to existing technologies, the present invention provides a deep learning-based method for curved text detection and recognition. It addresses the issue of insufficient curved text detection and recognition accuracy in natural scenes by proposing a deep learning model that integrates feature enhancement and image correction, significantly improving the ability to locate and extract content from curved text in complex scenes. This method utilizes a dynamic snake-like convolution module to accurately capture local geometric features of text, combines a contextual attention mechanism to enhance semantic relevance, and introduces a global attention mechanism to achieve cross-dimensional, multi-scale feature fusion. This effectively addresses the issue of traditional models' insufficient attention representation for curved text features. Furthermore, an innovative dual geometry and color correction module transforms curved and blurred text into a regular, clear form through deformation adjustment and image quality optimization, significantly reducing the difficulty of subsequent recognition. During feature extraction, partial convolutions and global blocks are used to collaboratively extract local details and global semantic information. Combined with a progressive multi-scale processing strategy, this method further enhances the model's robustness to curved text. Experiments demonstrate that this method achieves significant performance improvements on multiple curved text datasets while maintaining compatibility with regular text, providing reliable technical support for the efficient extraction of text information in natural scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is a flowchart of a method for detecting and recognizing curved text based on deep learning provided in the first embodiment of the present application; Figure 2 This is a diagram of the overall network structure of the text detection model based on feature enhancement and attention provided by the first embodiment of the present application; Figure 3 Schematic diagram of a dynamic serpentine convolution layer provided in the first embodiment of the present application; Figure 4 This is a diagram of the contextual attention mechanism layer structure provided by the first embodiment of the present application; Figure 5 This is a diagram of the global attention mechanism layer structure provided by the first embodiment of the present application; Figure 6 This is a diagram showing the overall structure of a text recognition model based on image correction provided in the first embodiment of the present application; Figure 7 This is a schematic diagram of image correction provided by the first embodiment of the present application; Figure 8 This is an internal structure diagram of a curved text detection and recognition system based on deep learning provided in the second embodiment of the present application. DETAILED DESCRIPTION

[0018] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0019] The terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and are not used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of this application can be implemented in an order other than those illustrated or described herein. In addition, the term "and / or" in the specification and claims refers to at least one of the connected objects, and the character " / " generally indicates that the objects associated with each other are in an "or" relationship.

[0020] Below, in conjunction with the accompanying drawings, a method for detecting and recognizing curved text based on deep learning provided by an embodiment of the present application is described in detail through specific embodiments and their application scenarios.

[0021] Example 1 See also Figure 1 , which is a flowchart of a curved text detection and recognition method based on deep learning provided by the present invention, and the method includes steps S1 to S5.

[0022] Step S1: Build a text detection model and a text recognition model respectively.

[0023] The text detection model provided by the present invention includes a context attention mechanism layer, a dynamic snake convolution layer, a feature fusion layer and a global attention mechanism layer, which are used to generate labeling results. Figure 2The overall network structure diagram of the text detection model based on feature enhancement and attention provided by the present invention is shown in FIG. Figure 3 Schematic diagram of the dynamic snake-shaped convolutional layer provided by the present invention, Figure 4 This is a structural diagram of the contextual attention mechanism layer provided by the present invention.

[0024] See also Figure 5 , which shows the layer structure of the global attention mechanism provided by this invention. The global attention mechanism takes into account the importance of cross-dimensional interactions. Combining the channel and spatial attention mechanisms, it retains and amplifies feature information in the three dimensions of channel, spatial width, and spatial height, as well as the interactions between them. This allows for the collection of more important dimensional information, reduces information loss, and globally focuses on relevant information about features at different scales.

[0025] The text detection model proposed in this paper supplements the free learning process with geometric constraints, limiting the range and direction of the convolution kernel's movement. This allows for more accurate feature capture of curved text and better focus on the specific local geometric structure of the text. Furthermore, an iterative strategy controls the movement of the convolution kernel, ensuring that the perception area is continuous and does not diverge due to excessive deformation or offset.

[0026] Step S2: Train the text detection model and the text recognition model respectively, and obtain a trained text detection model and a trained text recognition model accordingly.

[0027] Specifically, in the process of training the text detection model, the present invention uses the CTW1500 dataset for training. The CTW1500 dataset is a scene text dataset specifically for curved text detection, which mainly focuses on curved text. There are 1,500 images in total, of which 1,000 are training sets and 500 are test sets. Each image contains at least one line of curved text.

[0028] Loss function of text detection model It can be expressed as the weighted sum of each loss, and the formula is as follows: ; in, represents the probability map loss, represents the ternary graph loss, represents the threshold map loss, , using binary cross entropy loss to get probability map loss and binary image loss , as follows: ; in, represents the predicted value, Represents the label value, represents the sample set, and The ratio of positive samples to negative samples is 1:3.

[0029] The threshold map loss is obtained by calculating the sum of the absolute differences between the threshold map prediction value and the threshold map label , the formula is as follows: ; in, is the set of pixel numbers within the dilation boundary, Represents the label of the threshold map, Represents the predicted values ​​of the threshold map.

[0030] In summary, the present invention uses the loss function of the text detection model Train the text detection model to obtain a trained text detection model.

[0031] Furthermore, in the process of training the text recognition model, the present invention uses the CUTE80 dataset for training. The CUTE80 dataset mainly consists of curved text images, with a total of 80 images collected in natural scenes. 288 images with text instances can be accurately cropped as needed. The text recognition model uses CTC loss to avoid the manual alignment of input and output operations. Specifically, the CTC loss function L2 and conditional probability of The relationship can be expressed as: ; in, is a given input sequence In the case of , the output sequence is The conditional probability of . This conditional probability is obtained through all possible alignment paths. The probability and is calculated as: ; in, Is to align the path Mapping to output sequence (by merging repeated characters and removing whitespace). Each path The probability of is the product of the predicted probabilities at each time step: ; in, Indicates that this is time, Indicates the time when the output ends. Representation Label probability.

[0032] In summary, the present invention uses the CTC loss function L2The text recognition model is trained to obtain a trained text recognition model. The CTC loss function used in the present invention is L2 In fact, it is minimizing the negative log-likelihood, that is, calculating the conditional probability of maximizing the correct label sequence. In this way, the text recognition model can learn the mapping relationship between the input sequence and the output sequence without explicit alignment.

[0033] Step S3: Mark the position of the text area based on the trained text detection model to generate a marking result.

[0034] Specifically, the specific process of generating the labeling result includes: performing convolution operation on the input image to downsample it to generate a first feature map, a second feature map, a third feature map, and a fourth feature map, where the size of the first feature map is half of the input image, the size of the second feature map is one-quarter of the input image, the size of the third feature map is one-eighth of the input image, and the size of the fourth feature map is one-sixteenth of the input image; performing convolution processing on the fourth feature map according to the dynamic snake convolution layer to generate snake convolution features; performing adaptive weighting processing on the snake convolution features according to the context attention mechanism layer to generate an attention representation of the curved text feature; splicing the attention representation of the curved text feature with the second feature map, the third feature map, and the fourth feature map respectively to obtain a spliced ​​feature; performing feature fusion processing on the spliced ​​feature according to the global attention mechanism layer to generate a multi-scale fusion feature; performing probability map conversion and threshold map conversion on the multi-scale fusion feature respectively to generate the labeling result.

[0035] Specifically, the specific process of generating the attention representation of the curved text feature includes: inputting the snake convolution features into three parallel distributed convolution layers for convolution processing to generate the first snake convolution feature, the second snake convolution feature and the third snake convolution feature; transforming and transposing the first snake convolution feature to generate a query matrix, and transforming the second snake convolution feature to generate a key matrix; multiplying the key matrix and the query matrix and normalizing them to generate the attention matrix; and bitwise multiplying the attention matrix and the third snake convolution feature to generate the attention representation of the curved text feature.

[0036] The process of adaptively weighting the snake convolution features according to the contextual attention mechanism layer to generate the attention representation of the curved text features is described in

[15] . Figure 4 , first, given the input features ,use These two convolutions calculate the input features and obtain the matrix by transformation and transposition , and the matrix is ​​obtained by transformation , and ensure that the matrix obtained after transformation The conversion calculation process is as follows: ; ; Then, the matrix and matrix Transformed into ,in , then we need the relevant matrix , used to obtain the relationship between each sub-region. Same matrix Do matrix multiplication, The calculation process is as follows: ; in, , and subsequently transformed into , after normalization, we get the attention matrix . And the feature map Through convolution Convert to Features , the process is as follows: ; Finally, the attention matrix obtained by the above process is Same characteristics ( ∈ ) bitwise multiplication to get the Attention representation of feature maps , the mathematical expression is: ; Furthermore, when the spliced ​​features are fused according to the global attention mechanism layer to generate multi-scale fused features, the global attention mechanism layer includes a channel attention layer and a spatial attention layer. The specific process of generating the multi-scale fused features includes: performing channel weight calibration processing on the spliced ​​features according to the channel attention layer to generate channel enhanced features; and performing spatial weight focusing processing on the channel enhanced features according to the spatial attention layer to generate multi-scale fused features. Figure 5 , input features After channel attention processing and spatial attention processing, the output features are obtained .

[0037] Step S4: Identify and extract the text content based on the trained text recognition model to generate a recognition result.

[0038] The text recognition model of the present invention includes a geometric correction layer, a color correction layer, an image block embedding layer, a partial convolution layer, and a global block layer, and is used to identify and extract text content to generate a recognition result. The specific steps are as follows: according to the image correction layer, the attention representation of the curved text feature is geometrically corrected and color corrected to generate a corrected image; according to the first image block embedding layer, the first partial convolution layer, and the first global block layer, the corrected image is sequentially subjected to image embedding processing, partial convolution processing, and global block processing, and then merged to generate a first correction result; according to the second image block embedding layer, the second partial convolution layer, and the second global block layer, the first correction result is sequentially subjected to partial convolution processing and global block processing, and then merged to generate a second correction result; according to the third partial convolution layer and the third global block layer, the second correction result is sequentially subjected to partial convolution processing and global block processing, and then merged to generate a third correction result. The third correction result is decoded to generate a recognition result.

[0039] The image correction-based text recognition model consists of two main components: an image correction layer and a text recognition layer. The image correction layer includes geometric and color correction, which is responsible for straightening curved text in the image into a smoother shape and improving image quality. The text recognition component primarily involves image block embedding, feature extraction, and CTC decoding. Feature extraction is divided into three stages, with the processed input height decreasing sequentially to obtain multi-scale information. Partial convolution and global blocks are used during the extraction process to obtain local and global information, respectively, which are then processed by merging or combining blocks.

[0040] See also Figure 6 , which represents the overall structure diagram of the text recognition model based on image correction provided by the present invention, including an image correction layer (image correction module). The input is the attention representation of the curved text feature of "Shenzhen XX Technology Co., Ltd.". After the attention representation of the curved text feature is geometrically corrected and color corrected by the image correction module, it undergoes image block embedding, partial convolution, global block, merging, partial convolution, global block, merging, partial convolution, global block, combination, and decoding processing in sequence to generate a recognition result (text display of Shenzhen XX Technology Co., Ltd.).

[0041] See also Figure 7 , which represents the image correction schematic diagram provided by the present invention. The image correction layer IRM provided by the present invention includes a geometric correction part and a color correction part. First, the input image is geometrically adjusted to correct the curved text to a certain extent; the color correction part is responsible for correcting image quality problems such as blur and improper contrast, improving a certain degree of clarity, and bringing convenience to the subsequent recognition process.

[0042] The text recognition layer proposed in this paper is primarily a three-stage network with gradually decreasing height. For the input text image, progressive image block embedding is first performed. Then, multi-scale features are obtained through three stages. In each stage, partial convolution and global blocks are used to extract local and global features. Merging and combining blocks are used to comprehensively process these features. Finally, CTC decoding is used to obtain a character sequence.

[0043] Step S5: Generate text content based on the marking results and the recognition results.

[0044] Specifically, after the text recognition model and the text detection model are trained, an image is input. The trained text detection model is first used to locate and mark the text. After marking is completed, the trained text recognition model is used to extract the content of the marked area, and finally the recognized text content is obtained.

[0045] It should be noted that the deep learning-based curved text detection and recognition method provided in the embodiments of the present application can be executed by a deep learning-based curved text detection and recognition system, or a control module in a deep learning-based curved text detection and recognition system for executing and loading a deep learning-based curved text detection and recognition method. In the embodiments of the present application, a deep learning-based curved text detection and recognition system is used to execute and load a method for curved text detection and recognition based on feature enhancement and image correction, to illustrate the deep learning-based curved text detection and recognition method provided in the embodiments of the present application.

[0046] Example 2 See also Figure 8 , which shows the internal structure of a deep learning-based curved text detection and recognition system provided by the present invention, including: A text detection model generation module 100 is used to generate a text detection model based on the context attention mechanism layer, the dynamic snake convolution layer, the feature fusion layer and the global attention mechanism layer; A text recognition model generation module 200 is used to generate a text recognition model based on the geometry correction layer, the color correction layer, the image block embedding layer, the partial convolution layer and the global block layer; A training module 300 is used to train the text detection module and the text recognition module respectively, and obtain a trained text detection module and a trained text recognition module respectively; The marking result generating module 400 is used to mark the position of the text area using the trained text detection module to generate a marking result; The recognition result generating module 500 is used to identify and extract text content using the trained text recognition module to generate a recognition result; The text content generation module 600 is used to generate text content according to the marking results and the recognition results.

[0047] The beneficial effects of the deep learning-based curved text detection and recognition system provided by the present invention are: first, the present invention accurately captures the local curved features of the text through the dynamic snake-shaped convolution layer in the text detection model generation module 100; enhances the semantic relevance and compensates for the neglect of overall information by local features through the contextual attention mechanism layer; realizes cross-dimensional and multi-scale feature fusion through the global attention mechanism layer; deforms and optimizes the input image through the geometric correction layer and color correction layer in the text recognition model generation module 200, and uses partial convolution layers and global block layers to collaboratively extract local details and global semantic information, showing significant performance advantages in curved text detection and recognition tasks; second, it is suitable for high-precision positioning and content extraction of curved text in natural scenes, and has broad application value in smart cities, autonomous driving, document digitization, industrial quality inspection and other fields.

[0048] In the embodiments of the present application, a deep learning-based curved text detection and recognition system can be a device, or a component, integrated circuit, or chip in a terminal. The device can be a mobile electronic device or a non-mobile electronic device. For example, the mobile electronic device can be a mobile phone, tablet computer, laptop computer, PDA, in-vehicle electronic device, wearable device, ultra-mobile personal computer (UMPC), netbook, or personal digital assistant (PDA), etc. The non-mobile electronic device can be a server, network attached storage (NAS), personal computer (PC), television (TV), ATM, or self-service machine, etc., and the embodiments of the present application do not specifically limit this.

[0049] In the embodiments of the present application, a deep learning-based curved text detection and recognition system can be a device having an operating system. The operating system can be an Android operating system, an iOS operating system, or other possible operating systems, which are not specifically limited in the embodiments of the present application.

[0050] The embodiment of the present application provides a deep learning-based curved text detection and recognition system that can achieve Figures 1 to 7 In the method embodiment, each process implemented by a curved text detection and recognition system based on deep learning is not described here to avoid repetition.

[0051] Optionally, an embodiment of the present application also provides an electronic device, including a processor, a memory, and a program or instruction stored in the memory and executable on the processor. When the program or instruction is executed by the processor, each process of the above-mentioned embodiment of the deep learning-based curved text detection and recognition method is implemented, and the same technical effect can be achieved. To avoid repetition, it will not be described here.

[0052] An embodiment of the present application also provides a readable storage medium, on which a program or instruction is stored. When the program or instruction is executed by a processor, the various processes of the above-mentioned embodiment of the curved text detection and recognition method based on deep learning are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0053] The processor is the processor in the electronic device described in the above embodiment. The readable storage medium includes a computer-readable storage medium, such as a computer read-only memory (ROM), random access memory (RAM), a magnetic disk, or an optical disk.

[0054] It should be noted that, in this article, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprises a ..." does not exclude the presence of other identical elements in the process, method, article or device comprising the element. In addition, it should be noted that the scope of the methods and devices in the embodiments of the present application is not limited to performing functions in the order shown or discussed, and may also include performing functions in a substantially simultaneous manner or in the opposite order according to the functions involved. For example, the described method may be performed in an order different from that described, and various steps may also be added, omitted, or combined. In addition, the features described with reference to certain examples may be combined in other examples.

[0055] Through the description of the above embodiments, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus the necessary general hardware platform. Of course, they can also be implemented by hardware, but in many cases the former is a more preferred embodiment. Based on this understanding, the technical solution of this application, or the part that contributes to the existing technology, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of this application.

[0056] The embodiments of the present application are described above in conjunction with the accompanying drawings, but the present application is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of this application, ordinary technicians in this field can also make many forms without departing from the purpose of this application and the scope of protection of the claims, all of which are within the protection of this application.

Claims

1. A method for detecting and recognizing curved text based on deep learning, characterized in that: include: Build text detection model and text recognition model respectively; Training the text detection model and the text recognition model respectively to obtain a trained text detection model and a trained text recognition model; Mark the text area based on the trained text detection model to generate a marking result; Identify and extract text content based on the trained text recognition model to generate recognition results; Generating text content based on the marking result and the recognition result.

2. The method for detecting and recognizing curved text based on deep learning according to claim 1, characterized in that: The text detection model includes a contextual attention mechanism layer, a dynamic snake convolution layer, a feature fusion layer and a global attention mechanism layer, and the text recognition model includes a geometric correction layer, a color correction layer, an image block embedding layer, a partial convolution layer and a global block layer.

3. The method for detecting and recognizing curved text based on deep learning according to claim 1, wherein: The process of generating the marking result includes: Perform a convolution downsampling process on the input image to generate a first feature map, a second feature map, a third feature map, and a fourth feature map. The size of the first feature map is half of the input image, the size of the second feature map is one-quarter of the input image, the size of the third feature map is one-eighth of the input image, and the size of the fourth feature map is one-sixteenth of the input image. Performing convolution processing on the fourth feature map according to the dynamic snake convolution layer to generate a snake convolution feature; Adaptively weighting the snake-shaped convolutional features according to a contextual attention mechanism layer to generate an attention representation of the curved text features; Concatenating the attention representation of the curved text feature with the second feature map, the third feature map, and the fourth feature map to obtain a concatenated feature; According to the global attention mechanism layer, the spliced ​​features are subjected to feature fusion processing to generate multi-scale fusion features; The multi-scale fusion features are respectively converted into a probability map and a threshold map to generate a labeling result.

4. The method for detecting and recognizing curved text based on deep learning according to claim 3, wherein: The specific process of generating the attention representation of curved text features includes: The snake-shaped convolution features are input into three parallel distributed convolution layers for convolution processing to generate the first snake-shaped convolution feature, the second snake-shaped convolution feature and the third snake-shaped convolution feature; Transforming and transposing the first serpentine convolution feature to generate a query matrix, and transforming the second serpentine convolution feature to generate a key matrix; Multiplying the key matrix and the query matrix and normalizing them to generate an attention matrix; The attention matrix is ​​bitwise multiplied by the third snake-shaped convolution feature to generate an attention representation of the curved text feature.

5. The method for detecting and recognizing curved text based on deep learning according to claim 3, wherein: When performing feature fusion processing on the spliced ​​features to generate multi-scale fusion features according to the global attention mechanism layer, the global attention mechanism layer includes a channel attention layer and a spatial attention layer, wherein the specific process of generating the multi-scale fusion features includes: According to the channel attention layer, performing channel weight calibration processing on the splicing features to generate channel enhancement features; According to the spatial attention layer, spatial weight focusing processing is performed on the channel enhancement features to generate multi-scale fusion features.

6. The method for detecting and recognizing curved text based on deep learning according to claim 1, wherein: The process of generating the recognition result includes: According to the image correction layer, the attention representation of the curved text features is geometrically and color-corrected to generate a corrected image; performing image embedding processing, partial convolution processing, and global block processing on the corrected image in sequence according to the first image block embedding layer, the first partial convolution layer, and the first global block layer, and then merging the processed images to generate a first correction result; performing partial convolution processing and global block processing on the first correction result in sequence according to the second image block embedding layer, the second partial convolution layer, and the second global block layer, and then merging the resulting data to generate a second correction result; performing partial convolution processing and global block processing on the second correction result in sequence according to a third partial convolution layer and a third global block layer, and then merging the resulting data to generate a third correction result; The third correction result is decoded to generate a recognition result.

7. The method for detecting and recognizing curved text based on deep learning according to claim 1, wherein: The process of obtaining a trained text detection model includes: Based on the binary cross entropy loss, we get the probability graph loss function and the binary graph loss function; Obtain the threshold map loss function based on the absolute difference between the threshold map prediction value and the threshold map label; Constructing a loss function of a text detection model based on the probability graph loss function, the binary graph loss function, and the threshold graph loss function; The text detection model is trained based on the loss function of the text detection model to obtain a trained text detection model.

8. A curved text detection and recognition system based on deep learning, characterized in that: include: A text detection model generation module is used to generate a text detection model based on the contextual attention mechanism layer, dynamic snake convolution layer, feature fusion layer, and global attention mechanism layer; A text recognition model generation module is used to generate a text recognition model based on the geometry correction layer, color correction layer, image block embedding layer, partial convolution layer and global block layer; A training module is used to train the text detection module and the text recognition module respectively, and obtain a trained text detection module and a trained text recognition module respectively; A marking result generating module, configured to mark the position of the text area using the trained text detection module to generate a marking result; A recognition result generation module, configured to use the trained text recognition module to identify and extract text content to generate a recognition result; A text content generation module is used to generate text content according to the marking result and the recognition result.

9. An electronic device, characterized in that: The invention comprises a processor, a memory, and a program or instruction stored in the memory and executable on the processor, wherein when the program or instruction is executed by the processor, the steps of a method for detecting and recognizing curved text based on deep learning are implemented as described in any one of claims 1 to 7.

10. A readable storage medium, characterized in that: The readable storage medium stores a program or instruction, and when the program or instruction is executed by the processor, the steps of the deep learning-based curved text detection and recognition method as described in any one of claims 1 to 7 are implemented.

Images