A method for dense scene text detection and recognition

Abstract

The present invention discloses a method for dense scene text detection and recognition, comprising: obtaining a public dataset and a dense text image dataset; establishing a deep learning network model based on word length-aware Transformer, pre-training the deep learning network model using the public dataset to generate an initial recognition model; training the initial recognition model using the dense text image dataset to optimize the initial recognition model, and then generating a dense scene text detection and recognition collaborative model; obtaining a dense scene text image to be recognized, inputting the dense scene text image to be recognized into the dense scene text detection and recognition collaborative model to obtain a recognition result. The present invention solves the technical problems of low accuracy and easy missed detection and false detection in the existing dense scene text detection and recognition methods.

Description

Technical Field

[0001] The present invention relates to the technical field of text recognition, and in particular to a method for detecting and recognizing text in dense scenes. Background Art

[0002] Scene text detection and recognition technology is a rapidly developing branch of artificial intelligence. Conventional scene text detection and recognition tasks aim to identify all instances of text within complex natural scene images through detection and recognition. This not only locates these instances within the image but also transcribes the text content to obtain the text information. This approach is crucial for a wide range of real-world applications, including image retrieval, autonomous driving, and assistance for the blind.

[0003] With the powerful representation capabilities of deep neural networks, scene text detection and recognition have made great progress in recent years. However, due to the characteristics of text in dense scenes, such as random location distribution, variable size and shape, rich fonts and colors, and diverse language types, the different patterns revealed by data changes, and the demanding conditions required for practical applications, text recognition in dense scenes remains a huge challenge and one of the core problems in the field of computer vision.

[0004] Currently, there are two approaches to detecting and recognizing text in dense scenes. One approach treats detection and recognition as two independent tasks based on deep learning networks, following the traditional pipeline approach to separate the text detection and recognition processes. A trained text detector is used to generate text proposals, which are then fed into a text recognizer to produce the result. However, because the recognition result is extremely dependent on the accuracy of the detection, error accumulation is a serious problem. Another approach leverages the global modeling advantages of the Transformer to accurately detect and recognize text in natural scenes end-to-end. However, for dense scenes with extreme aspect ratios, large numbers of text, curvature, and other challenging conditions, missed detections and false detections are common, making it difficult to achieve satisfactory results. Summary of the Invention

[0005] The purpose of the present invention is to overcome the above technical deficiencies and provide a method for dense scene text detection and recognition to solve the technical problems in the existing technology of dense scene text detection and recognition methods with low accuracy and prone to missed detection and false detection.

[0006] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0007] In a first aspect, the present invention provides a method for detecting and recognizing dense scene text, comprising the following steps:

[0008] Obtain public datasets and dense text image datasets;

[0009] Establishing a deep learning network model based on a word-length-aware Transformer, and pre-training the deep learning network model using the public dataset to generate an initial recognition model; wherein the deep learning network model consists of an encoder and a decoder containing multiple dynamic heads, and is used to extract text detection features and recognition features, fuse multi-scale semantic information, predict text bounding box information, and predict character sequences. The deep learning network model uses a character count prior customized for different word lengths to effectively constrain the region of interest, and retains a word-length-aware instance segmentation suggestion head;

[0010] The dense text image dataset is used to train the initial recognition model to optimize the initial recognition model and then generate a dense scene text detection and recognition collaborative model;

[0011] A dense scene text image to be recognized is obtained, and the dense scene image to be recognized is input into the dense scene text detection and recognition collaborative model to obtain a recognition result.

[0012] In some embodiments, the data sources of the dense text image dataset include:

[0013] Self-made dataset images after image processing, dense scene text images selected from public datasets, synthetic dataset images, and images after data enhancement processing.

[0014] In some embodiments, the deep learning network model uses the same decoder with multiple dynamic heads to decode the same set of text queries in the detection and recognition stages.

[0015] In some embodiments, the deep learning network model is divided into three parts according to its functions: a feature fusion layer, a query iteration layer, and a matching output layer. The feature fusion layer is used to extract multi-scale features from the input image, and use the initialized text bounding box information to align the multi-scale features with the region of interest, and fuse image features of different scales; the query iteration layer is used to perform multiple rounds of iterative updates on the text query using the fused features through a decoder with a dynamic head to obtain refined text query information; the matching output layer is used to use the text query information and multi-scale fusion information of the recognizer to predict the character sequence of the best matching text instance proposal, and obtain the final output of dense scene text image detection and recognition.

[0016] In some embodiments, the deep learning network model further includes a text word length prior count information prediction loss function, and the calculation formula of the text word length prior count information prediction loss function is:

[0017]

[0018] Among them, L prior The loss is predicted by the prior count information of the text word length, is the predicted value set of the text word length prior information, G p The labels required for predicting branches are counted a priori for the length of text words.

[0019] In some embodiments, the deep learning network model further includes a word length-aware instance segmentation loss function, and the calculation formula of the word length-aware instance segmentation loss function is:

[0020]

[0021] Among them, L map is the word length-aware instance segmentation loss, is the predicted value set of the word length segmentation map, G m is the word length aware segmentation map label, It's G m and The intersection between |G m | and Represents G m and The number of elements.

[0022] In some embodiments, the dense scene text detection and recognition method further includes:

[0023] After establishing a deep learning network model based on word length-aware Transformer, the hyperparameters of the deep learning network model are adjusted; wherein the hyperparameters include at least the number of text query iterations, the learning rate, the number of heads of the dynamic decoder, the dropout rate, and the loss weight factor.

[0024] In a second aspect, the present invention further provides a dense scene text detection and recognition device, comprising:

[0025] Data acquisition module, used to obtain public datasets and dense text image datasets;

[0026] A first model training module is configured to establish a deep learning network model based on a word-length-aware Transformer, and pre-train the deep learning network model using the public dataset to generate an initial recognition model; wherein the deep learning network model is composed of an encoder and a decoder containing multiple dynamic heads, and is configured to extract text detection features and recognition features, fuse multi-scale semantic information, predict text bounding box information, and predict character sequences. The deep learning network model uses a character count prior customized for different word lengths to effectively constrain regions of interest, and retains a word-length-aware instance segmentation suggestion head;

[0027] A second model training module is used to train the initial recognition model using the dense text image dataset to optimize the initial recognition model and generate a dense scene text detection and recognition collaborative model;

[0028] The recognition module is used to obtain a dense scene text image to be recognized, and input the dense scene text image to be recognized into the dense scene text detection and recognition collaborative model to obtain a recognition result.

[0029] In a third aspect, the present invention further provides an electronic device, comprising: a processor and a memory;

[0030] The memory stores a computer program executable by the processor;

[0031] When the processor executes the computer program, the steps in the above-mentioned dense scene text detection and recognition method are implemented.

[0032] In a fourth aspect, the present invention also provides a computer-readable storage medium, which stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the dense scene text detection and recognition method as described above.

[0033] Compared with the prior art, the dense scene text detection and recognition method provided by the present invention utilizes an end-to-end collaborative method for scene text detection and recognition based on a word-length-aware Transformer architecture. This method utilizes an image encoder to extract multi-scale image features from the input scene graph and uses a decoder with multiple dynamic heads to iteratively update the text query, achieving information refinement in the detection phase and loss gradient reflow in the recognition phase, thereby achieving the goal of collaboratively optimizing detection and recognition. Furthermore, the design can extract features from image data and effectively constrain regions of interest using a character count prior customized for different word lengths. A dedicated word-length-aware instance segmentation proposal head is also retained, enhancing the network model's ability to capture sample features in imbalanced data distributions. The designed collaborative architecture for dense scene text detection and recognition can stably implement dense scene text detection and recognition feature extraction, multi-scale semantic information fusion, text bounding box information prediction, and character sequence prediction, improving the accuracy of dense scene text detection and recognition. The design is suitable for dense text detection and recognition in a variety of real-world scenarios, particularly for situations requiring accurate detection and recognition of densely distributed text. It not only reduces the interference of complex backgrounds in images and focuses on scene text positioning, but also has a good positioning capability for texts with extreme word lengths in unbalanced text data. It has important guiding significance for achieving application fields such as accurate image retrieval, safe autonomous driving, and efficient blind assistance. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 is a flowchart of a dense scene text detection and recognition method provided by an embodiment of the present invention;

[0035] Figure 2 This is a complete flow chart for preparing a data set for the high-precision dense scene text detection and recognition method according to an embodiment of the present invention;

[0036] Figure 3 This is a process flow chart for building and training a deep network model for a high-precision dense scene text detection and recognition method according to an embodiment of the present invention;

[0037] Figure 4 is a flow chart of the model processing steps of the high-precision dense scene text detection and recognition method according to an embodiment of the present invention;

[0038] Figure 5 1 is a model architecture diagram of a high-precision dense scene text detection and recognition method according to an embodiment of the present invention;

[0039] Figure 6 Schematic diagram of the functional modules of the dense scene text detection and recognition device provided by an embodiment of the present invention;

[0040] Figure 7 It is a schematic diagram of the hardware structure of the electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0042] See also Figure 1 The dense scene text detection and recognition method provided by the present invention includes the following steps:

[0043] S100, obtaining a public dataset and a dense text image dataset;

[0044] S200, establishing a deep learning network model based on a word length-aware Transformer, and pre-training the deep learning network model using the public dataset to generate an initial recognition model; wherein the deep learning network model consists of an encoder and a decoder containing multiple dynamic heads, and is used to realize the extraction of text detection features and recognition features, the fusion of multi-scale semantic information, the prediction of text bounding box information, and the prediction of character sequences. The deep learning network model uses a character count prior customized for different word lengths to effectively constrain the region of interest, and retains a word length-aware instance segmentation suggestion head;

[0045] S300, training the initial recognition model using the dense text image dataset to optimize the initial recognition model and then generating a dense scene text detection and recognition collaborative model;

[0046] S400 : Acquire a dense scene text image to be recognized, and input the dense scene text image to be recognized into the dense scene text detection and recognition collaborative model to obtain a recognition result.

[0047] In this embodiment, an end-to-end collaborative method for scene text detection and recognition based on a word-length-aware Transformer architecture is designed. An image encoder is used to extract multi-scale image features from the input scene graph, and a decoder containing multiple dynamic heads is used to iteratively update the text query, thereby achieving information refinement in the detection phase and loss gradient reflow in the recognition phase, thereby achieving the purpose of collaborative optimization of detection and recognition. At the same time, the design can extract features from image data and use character count priors customized for different word lengths to effectively constrain the region of interest; a dedicated word-length-aware instance segmentation proposal head is also retained, which enhances the network model's ability to capture sample features in an unbalanced data distribution. The designed collaborative architecture for dense scene text detection and recognition can stably implement dense scene text detection feature and recognition feature extraction, multi-scale semantic information fusion, text bounding box information prediction, and character sequence prediction, thereby improving the accuracy of dense scene text detection and recognition. It is suitable for dense text detection and recognition in a variety of real-world scenarios, especially for situations where accurate detection and recognition of densely distributed text is required. It not only reduces the interference of complex backgrounds in images and focuses on scene text positioning, but also has a good positioning capability for texts with extreme word lengths in unbalanced text data. It has important guiding significance for achieving application fields such as accurate image retrieval, safe autonomous driving, and efficient blind assistance.

[0048] In some embodiments, in step S100, the data sources of the dense text image dataset include:

[0049] Self-made dataset images after image processing, dense scene text images selected from public datasets, synthetic dataset images, and images after data enhancement processing.

[0050] In some embodiments, the data enhancement processing method includes one or more of random scaling, random rotation, random cropping, and brightness enhancement.

[0051] In some embodiments, the method for creating the synthetic dataset image is:

[0052] A large amount of text in different fonts is synthesized into a regular scene image using the PIL toolkit, and the synthesized image is annotated to obtain a synthetic dataset image.

[0053] In this embodiment, image synthesis and data enhancement methods can be used to prepare a scene image dataset containing dense text, complete the division of positive and negative samples of the dataset image, content annotation of scene text image instances and recording of real box coordinate information, and the designed text detection and recognition collaborative architecture can be fully trained on the synthetically prepared dense scene text image dataset.

[0054] Specifically, a certain number of synthetic images containing dense text can be prepared through scientific image synthesis methods, or real dense text scene images can be obtained by using paper documents and high-definition industrial cameras to collect and crop them according to specifications, or data enhancement can be used to obtain a larger number of dense text images, thereby forming the dense scene text image dataset.

[0055] Furthermore, the preparation process of the dense scene text image dataset includes: preparation of conventional reading e-books, preparation of paper reading materials, dense text image acquisition, and dense text image processing.

[0056] For example, embodiments of the present invention capture dense text images by capturing text information in various physical reading scenarios using industrial cameras or cameras. Alternatively, high-quality text images that meet the requirements can be selected from the currently available public text image dataset, TotalText. It is important to note that the characters contained in the text instances in the prepared text images should be clear and rich. Alternatively, the PIL toolkit can be used to synthesize a large amount of text in different fonts into a regular scene image to obtain a synthetic dense text image of a certain scale.

[0057] Furthermore, in order to expand the size of the prepared dense scene text image dataset and to enable pre-training and training of deep network models on a large amount of image data, the embodiments of the present invention require the use of data augmentation methods. Specifically, the number of images in the dataset can be expanded through different image processing methods, such as random scaling, where the short size is selected from 640 to 896 (interval 32), and the long size does not exceed 1600; random rotation, where we rotate the image between plus and minus 90 degrees; random cropping, where we ensure that the cropped image does not cut off the text instance; brightness enhancement, etc.

[0058] Furthermore, the dataset prepared by the embodiment of the present invention also needs to complete the division of the dataset image positive and negative samples of the dense scene text image dataset. The division of positive and negative samples is determined in the dataset prepared by the above method, and the image samples include: positive samples, which include all text instances in the image with clearly determined coordinate information and character content, and are regarded as the foreground target part of the dense scene text image; negative samples, which include extremely blurred and indistinguishable text and non-text background parts in the scene image.

[0059] Furthermore, the dataset prepared by the embodiment of the present invention also needs to complete the task of Ground-Truth annotation of appropriately sized areas of the synthetic dataset image. The embodiment of the present invention provides two annotation methods: one is to automatically generate and record the real box coordinate position information and text content transcription that need to be annotated based on the position information of the synthetic area modified by the dense scene text image; the other is to manually annotate the coordinate position and character transcription of all positive samples based on the text detection and recognition image data annotation tool PPOCRLabel, and record and save the said Ground-Truth.

[0060] It should be noted that the construction rules of data labels in the embodiment of the present invention are divided into two categories, one is to construct word length prior information labels, and the other is to construct text instance word length segmentation map labels.

[0061] In this embodiment, given an image I containing Y text instances, its text word length prior count label G I The aspect ratio N of all text instances I and the number of characters N I Composition, described as:

[0062]

[0063] Further, in the embodiment of the present invention, The length can be calculated from the transcription label in the data annotation. For a text instance in image I, its transcription annotation is So:

[0064]

[0065] Among them, len() is the character length calculation function.

[0066] Furthermore, the embodiment of the present invention calculates the aspect ratio R of a text instance. The embodiment of the present invention divides the calculation of the aspect ratio into two categories. One category is horizontal text or multi-directional text, which contains four coordinate points. The other category is curved or long text, which contains 2n coordinate points, where n∈{3,4,5,6,7,8}. For the convenience of calculation, the embodiment of the present invention marks the coordinate points in a clockwise order starting from the upper left corner of the text instance according to the annotation convention. For horizontal text or multi-directional text, the calculation of the horizontal edge can be expressed as:

[0067]

[0068] Among them, L hs is the horizontal length of the text instance, P is the coordinate point of the text instance, and |*| is the distance between the two points. hs The average length of the two sides of the quadrilateral formed by the text instance annotation points that are in the same direction as the text characters. For any three points in the four coordinate points in the image plane, the triangle formed by them satisfies the basic mathematical theorem:

[0069]

[0070] sin 2 θ+cos 2 θ=1

[0071] Where a, b, and c are three edges formed by any three points. The four angles θ formed by the vertical and horizontal edges are calculated based on the coordinate values of the four points. i , we can calculate:

[0072]

[0073] Among them, L vs is the vertical length of the text instance, θ i For the quadrilateral at coordinate point P i The interior angle at . The length of the longitudinal side is L vs The average length of the sides of the quadrilateral formed by the annotation points of the text instance that are perpendicular to the arrangement direction of the text characters. The ratio of the horizontal length to the vertical length is recorded as the aspect ratio R:

[0074]

[0075] In this embodiment of the present invention, for a curved or long text containing 2n points, n∈{3,4,5,6,7,8}, all its coordinate points are divided into n-1 rectangles according to the rule that four adjacent coordinate points form a quadrilateral. Then, the horizontal and vertical lengths of the n-1 quadrilaterals are calculated using the horizontal or multi-directional text instance calculation method to obtain the horizontal length L of the n-1 quadrilaterals. hs and longitudinal length Lvs , and the summed ratio is recorded as the aspect ratio of the curved or long text:

[0076]

[0077] Furthermore, this embodiment takes into account the differences in sample data and normalizes the calculated labels to obtain the labels G required for the text word length prior count prediction branch. p .

[0078] Furthermore, in the embodiment of the present invention, for a given image I containing Y text instances, each text instance is composed of a polygon, and the number of vertices n of the polygon is determined by the characteristics of the dataset. The number of characters transcribed for each text instance is Determine word length-aware segmentation map label G m .

[0079] The specific flowchart for establishing a dense scene text image dataset is as follows: Figure 2 shown.

[0080] In some embodiments, in step S200 and step S300, an algorithm suitable for dense scene text detection and recognition is designed, a deep neural network model is built, and the constructed image encoder uses a Transformer architecture suitable for global feature modeling to extract multi-scale semantic features of the input image, and a decoder containing multiple dynamic heads is used to collaboratively optimize the text detection process and the text recognition process. The dense scene text detection and recognition collaborative algorithm uses bipartite matching to optimally match the prediction results generated by the text query and the learning data Ground-Truth. The bounding box loss, text mask loss, instance category loss, and word length prediction loss in the detection stage all participate in the optimal matching process as matching weights.

[0081] Specifically, embodiments of the present invention should design algorithms suitable for dense scene text detection and recognition based on scenario requirements. This paper provides an architecture for dense scene text detection and recognition collaboration using Transformer, based on an end-to-end collaborative method. This architecture can stably extract text detection and recognition features in dense scenes, fuse multi-scale semantic information, predict text bounding box information, and predict character sequences.

[0082] Furthermore, the collaborative architecture for dense scene text detection and recognition designed in the embodiment of the present invention regards the text detection task as a set prediction problem, and iteratively updates the text query in the text detector to achieve information refinement in the detection stage and loss gradient reflow in the recognition stage, thus optimizing the collaborative architecture for dense scene text detection and recognition end-to-end.

[0083] Optionally, the deep neural network framework designed in the embodiment of the present invention can be constructed by the common Transformer architecture and convolutional neural network, and through the serial sequential connection and synergy of a specific encoder and decoder combination, this network framework generally has an excellent ability to fuse local and global features of the image, and will show more significant advantages in processing dense scene text detection and recognition tasks.

[0084] In some embodiments, embodiments of the present invention constrain the word length prior that is clearly present in dense scene text images, so that the designed dense scene text detection and recognition collaborative architecture extracts high-level semantic features from image data and uses character count priors customized for different word lengths to effectively constrain the text target area of interest.

[0085] In addition, an embodiment of the present invention defines text with less than 4 or greater than 10 characters in dense scene text images as extreme word length text categories for segmentation. The designed dense scene text detection and recognition collaborative architecture retains a dedicated word length-aware instance segmentation suggestion head, which pays more attention to text instances with extreme word lengths, thereby enhancing the network model's ability to capture sample features in an unbalanced data distribution.

[0086] In some embodiments, the deep learning network model uses the same decoder with multiple dynamic heads to decode the same set of text queries in the detection and recognition stages.

[0087] In this embodiment, the architecture of collaborative dense scene text detection and recognition uses the same query decoder with multiple dynamic heads to decode the same set of text queries in the detection and recognition stages, reducing network model redundancy and enhancing the synergy between detection and recognition.

[0088] In some embodiments, the deep learning network model is divided into three parts according to its functions: a feature fusion layer, a query iteration layer, and a matching output layer. The feature fusion layer is used to extract multi-scale features from the input image, and use the initialized text bounding box information to align the multi-scale features with the region of interest, and fuse image features of different scales; the query iteration layer is used to perform multiple rounds of iterative updates on the text query using the fused features through a decoder with a dynamic head to obtain refined text query information; the matching output layer is used to use the text query information and multi-scale fusion information of the recognizer to predict the character sequence of the best matching text instance proposal, and obtain the final output of dense scene text image detection and recognition.

[0089] In this embodiment, the designed deep neural network is divided into three parts: a feature fusion layer, a query iteration layer, and a matching output layer. Specifically, this embodiment extracts multi-scale features from the input image and uses the initialized text bounding box information to align the multi-scale features with the region of interest, fusing image features at different scales. A decoder with a dynamic head uses the fused features to iteratively update the text query multiple times, obtaining refined text query information. The recognizer uses the text query information and the multi-scale fused information to predict the character sequence of the best-matching text instance proposal, obtaining the final output for dense scene text image detection and recognition.

[0090] In some embodiments, the method further comprises:

[0091] After establishing a deep learning network model based on word length-aware Transformer, the hyperparameters of the deep learning network model are adjusted; wherein the hyperparameters include at least the number of text query iterations, the learning rate, the number of heads of the dynamic decoder, the dropout rate, and the loss weight factor.

[0092] In this embodiment, after the design of the collaborative framework for text detection and recognition in dense scenes is completed, some hyperparameters in the deep network should be adjusted and set specifically, such as the number of text query iterations, learning rate, number of heads of the dynamic decoder, dropout rate, loss weight factor, etc.; and the number of training iterations should also be appropriately adjusted according to the size of the data set.

[0093] In some embodiments, steps S200 and S300 are used to implement model training. After obtaining a dense scene text image dataset, it is necessary to scientifically and reasonably divide the training set, validation set, and test set, and use the above-mentioned end-to-end dense scene text detection and recognition collaborative model to perform iterative training on the prepared training set of dense scene text image data.

[0094] Furthermore, the collaborative algorithm for dense scene text detection and recognition described in this embodiment requires feature learning of positive and negative samples for the unique data of dense text. Any supervised learning-based deep neural network object detection algorithm requires training the deep network with positive and negative samples to facilitate feature learning.

[0095] Furthermore, in the embodiment of the present invention, the GT boxes and text character transcriptions marked in the prepared dataset images are all positive sample data, and the background parts in the text images are all negative sample data, so as to provide feature learning for the dense scene text detection and recognition collaborative algorithm described in the embodiment of the present invention.

[0096] During specific training, training data is prepared first. After the deep network model is built and the positive and negative sample division of the network training is completed, the dense scene text detection and recognition collaborative model designed in the embodiment of the present invention can be trained on the prepared dense scene text image dataset.

[0097] In order to make the network model more effective for dense scene text detection and recognition, the selected image encoder network should be pre-trained on a large-scale image classification dataset before training the dense scene text detection and recognition collaborative model, and the pre-trained weights should be saved. Then, the pre-trained image encoder network weights should be used to start training and learning the constructed dense scene text detection and recognition collaborative model.

[0098] After several iterations of training on the prepared dense scene text image dataset, the designed deep network model is verified and tested. If the accuracy and recall of dense text detection and recognition do not show significant improvement compared to the previous dense scene text detection and recognition algorithm, the network model parameters of the designed dense scene text detection and recognition collaborative algorithm should be fine-tuned and repeated until the dense scene text detection and recognition effect is improved.

[0099] After training, the best weight parameters obtained from the training of the above network model are selected and saved, and then the trained network model is tested on the dense scene text image verification set and test set to detect and recognize real dense scene text in the image. If there is no strict requirement for real-time performance, the dense scene text detection and recognition collaborative method in this invention can be applied to the dense scene text detection and recognition task based on video. The complete training flow chart of the specific deep neural network is as follows: Figure 3 shown.

[0100] In some embodiments, step S400 is to detect and recognize dense scene text based on images or videos in real reading scenarios using the model obtained through the above training. For the specific example of dense scene text detection and recognition, the deep network model processing steps are as follows: Figure 4 shown.

[0101] It should be noted that the above description of this specification is based on specific examples. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims can be performed in an order different from that described in the embodiments and still achieve the desired results. In addition, the processes depicted in the accompanying drawings do not necessarily require the specific order shown or the sequential order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0102] Another embodiment of the present invention provides a specific architecture of a dense scene text detection and recognition model, please refer to Figure 5 ,The dense scene text detection and recognition architecture mainly includes image encoder, RoI alignment, text detector and text recognizer.

[0103] The designed network image encoder uses conventional deep neural networks, including but not limited to mainstream convolutional neural networks and Transformers, and utilizes a bidirectional feature pyramid network to enhance the ability to extract backbone features. RoI alignment uses a traditional multi-feature alignment module to align the relative positions of corresponding features at different scales. The text detector incorporates a spatial word length prior information predictor to constrain the relative shape of the text and generates a text word length segmentation map to refine the absolute position information of the text. The text recognizer uses the results of the final detection refinement in the iterative process to perform multi-scale feature fusion, obtaining detailed text feature information for text character classification and recognition.

[0104] The loss function of the optimization stage is expressed as the weighted sum of the loss of the detection stage and the loss of the recognition stage. It should be noted that the text word length prior count information prediction loss L designed in the embodiment of the invention is prior Using Smooth L1 loss, the calculation formula is as follows:

[0105]

[0106] in, is the predicted value set of the text word length prior information, G p The label required for the branch is predicted by the prior count of the length of the text word. Another embodiment of the present invention designs the word length-aware instance segmentation loss L map The dice loss is also used, and the calculation formula is as follows:

[0107]

[0108] in, is the predicted value set of the word length segmentation map, G m is the word length aware segmentation map label, It's G m and The intersection between |G m | and Represents G m and The number of elements.

[0109] Another embodiment of the present invention provides a dense scene text detection and recognition device, see Figure 6The dense scene text detection and recognition device includes a data acquisition module 11, a first model training module 12, a second model training module 13 and a recognition module 14.

[0110] The data acquisition module 11 is used to acquire a public dataset and a dense text image dataset.

[0111] The first model training module 12 is used to establish a deep learning network model based on the word length-aware Transformer, and pre-train the deep learning network model using the public dataset to generate an initial recognition model; wherein, the deep learning network model is composed of an encoder and a decoder containing multiple dynamic heads, and is used to realize the extraction of text detection features and recognition features, the fusion of multi-scale semantic information, the prediction of text bounding box information and the prediction of character sequences. The deep learning network model uses character count priors customized for different word lengths to effectively constrain the area of interest, and retains a word length-aware instance segmentation suggestion head.

[0112] The second model training module 13 is used to train the initial recognition model using the dense text image dataset to optimize the initial recognition model and then generate a dense scene text detection and recognition collaborative model.

[0113] The recognition module 14 is used to obtain a dense scene text image to be recognized, and input the dense scene text image to be recognized into the dense scene text detection and recognition collaborative model to obtain a recognition result.

[0114] It should be noted that the module referred to in the present invention refers to a series of computer program instruction segments that can perform specific functions, which is more suitable for describing the execution process of dense scene text detection and recognition than a program. For the specific implementation of each module, please refer to the corresponding method embodiment above, which will not be repeated here.

[0115] In some embodiments, the data sources of the dense text image dataset include:

[0116] Self-made dataset images after image processing, dense scene text images selected from public datasets, synthetic dataset images, and images after data enhancement processing.

[0117] In some embodiments, the data enhancement processing method includes one or more of random scaling, random rotation, random cropping, and brightness enhancement.

[0118] In some embodiments, the method for creating the synthetic dataset image is:

[0119] A large amount of text in different fonts is synthesized into a regular scene image using the PIL toolkit, and the synthesized image is annotated to obtain a synthetic dataset image.

[0120] In some embodiments, the deep learning network model uses the same decoder with multiple dynamic heads to decode the same set of text queries in the detection and recognition stages.

[0121] In some embodiments, the deep learning network model is divided into three parts according to its functions: a feature fusion layer, a query iteration layer, and a matching output layer. The feature fusion layer is used to extract multi-scale features from the input image, and use the initialized text bounding box information to align the multi-scale features with the region of interest, and fuse image features of different scales; the query iteration layer is used to perform multiple rounds of iterative updates on the text query using the fused features through a decoder with a dynamic head to obtain refined text query information; the matching output layer is used to use the text query information and multi-scale fusion information of the recognizer to predict the character sequence of the best matching text instance proposal, and obtain the final output of dense scene text image detection and recognition.

[0122] In some embodiments, the dense scene text detection and recognition device also includes an adjustment module for adjusting the hyperparameters of the deep learning network model after establishing the deep learning network model based on the word length-aware Transformer; wherein the hyperparameters include at least the number of text query iterations, the learning rate, the number of heads of the dynamic decoder, the dropout rate, and the loss weight factor.

[0123] Another embodiment of the present invention provides an electronic device, such as Figure 7 As shown, the electronic device 10 includes:

[0124] One or more processors 110 and memory 120, Figure 7 In the description, a processor 110 is used as an example. The processor 110 and the memory 120 may be connected via a bus or other means. Figure 7 The bus connection is taken as an example.

[0125] The processor 110 is used to implement various control logics of the electronic device 10. It can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a single-chip microcomputer, an ARM (Acorn RISC Machine) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination of these components. In addition, the processor 110 can also be any traditional processor, microprocessor, or state machine. The processor 110 can also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP, and / or any other such configuration.

[0126] Memory 120, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as program instructions corresponding to the small molecule drug design method in the embodiments of the present invention. Processor 110 executes the non-volatile software programs, instructions, and modules stored in memory 120 to execute various functional applications and data processing of electronic device 10, thereby implementing the small molecule drug design method in the above-mentioned method embodiment.

[0127] The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store an operating platform and applications required for at least one function; the data storage area may store data created based on the use of the electronic device 10, etc. In addition, the memory 120 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 120 may optionally include a memory remotely located relative to the processor 110, and these remote memories may be connected to the electronic device 10 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] One or more units are stored in the memory 120, and when executed by one or more processors 110, perform the small molecule drug design method in any of the above method embodiments, for example, perform the above described Figure 1 Method steps S100 to S400.

[0129] Another embodiment of the present invention provides a computer-readable storage medium storing computer-executable instructions, which are executed by one or more processors, for example, to perform the above-described Figure 1 Method steps S100 to S400.

[0130] In summary, the dense scene text detection and recognition method provided by the present invention, through the designed end-to-end scene text detection and recognition collaborative method based on the word length-aware Transformer architecture, uses an image encoder to extract multi-scale image features from the input scene graph, and uses a decoder containing multiple dynamic heads to iteratively update the text query, realizing information refinement in the detection stage and loss gradient reflow in the recognition stage, so as to achieve the purpose of collaborative optimization of detection and recognition. At the same time, the design can extract features from image data and use character count priors customized for different word lengths to effectively constrain the region of interest; it also retains a dedicated word length-aware instance segmentation proposal head, which enhances the ability of the network model to capture sample features in unbalanced data distribution. The designed dense scene text detection and recognition collaborative architecture can stably realize dense scene text detection feature and recognition feature extraction, multi-scale semantic information fusion, text bounding box information prediction and character sequence prediction, thereby improving the accuracy of dense scene text detection and recognition. It is suitable for dense text detection and recognition in a variety of real-world scenarios, especially for situations where accurate detection and recognition of densely distributed text is required. It not only reduces the interference of complex backgrounds in images and focuses on scene text positioning, but also has a good positioning capability for texts with extreme word lengths in unbalanced text data. It has important guiding significance for achieving application fields such as accurate image retrieval, safe autonomous driving, and efficient blind assistance.

[0131] The specific embodiments of the present invention described above do not limit the scope of protection of the present invention. Any other corresponding changes and modifications made based on the technical concept of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A dense scene text detection and recognition method, characterized in that: The steps include: Obtain public datasets and dense text image datasets; Establishing a deep learning network model based on a word-length-aware Transformer, and pre-training the deep learning network model using the public dataset to generate an initial recognition model; wherein the deep learning network model consists of an encoder and a decoder containing multiple dynamic heads, and is used to extract text detection features and recognition features, fuse multi-scale semantic information, predict text bounding box information, and predict character sequences. The deep learning network model uses a character count prior customized for different word lengths to effectively constrain the region of interest, and retains a word-length-aware instance segmentation suggestion head; The dense text image dataset is used to train the initial recognition model to optimize the initial recognition model and then generate a dense scene text detection and recognition collaborative model; A dense scene text image to be recognized is obtained, and the dense scene text image to be recognized is input into the dense scene text detection and recognition collaborative model to obtain a recognition result.

2. The dense scene text detection and recognition method according to claim 1, characterized in that The data sources of the dense text image dataset include: Self-made dataset images after image processing, dense scene text images selected from public datasets, synthetic dataset images, and images after data enhancement processing.

3. The dense scene text detection and recognition method according to claim 1, characterized in that The deep learning network model uses the same decoder with multiple dynamic heads to decode the same set of text queries in the detection and recognition stages.

4. The dense scene text detection and recognition method according to claim 1, characterized in that The deep learning network model is divided into three parts according to its functions: a feature fusion layer, a query iteration layer, and a matching output layer. The feature fusion layer is used to extract multi-scale features from the input image, align the multi-scale features with the region of interest using the initialized text bounding box information, and fuse image features of different scales; the query iteration layer is used to perform multiple rounds of iterative updates on the text query using the fused features through a decoder with a dynamic head to obtain refined text query information; the matching output layer is used to use the text query information and multi-scale fusion information of the recognizer to predict the character sequence of the best matching text instance proposal to obtain the final output of dense scene text image detection and recognition.

5. The dense scene text detection and recognition method according to claim 4, characterized in that: The deep learning network model also includes a text word length prior count information prediction loss function, and the calculation formula of the text word length prior count information prediction loss function is: Among them, L prior The loss is predicted by the prior count information of the text word length, is the predicted value set of the text word length prior information, G p The labels required for predicting branches are counted a priori for the length of text words.

6. The dense scene text detection and recognition method according to claim 4, characterized in that: The deep learning network model also includes a word length-aware instance segmentation loss function, and the calculation formula of the word length-aware instance segmentation loss function is: Among them, L map is the word length-aware instance segmentation loss, is the predicted value set of the word length segmentation map, G m is the word length aware segmentation map label, It's G m and The intersection between |G m | and Represents G m and The number of elements.

7. The dense scene text detection and recognition method according to claim 1, characterized in that: Also includes: After establishing a deep learning network model based on word length-aware Transformer, the hyperparameters of the deep learning network model are adjusted; wherein the hyperparameters include the number of text query iterations, the learning rate, the number of decoder heads, the dropout rate, and the loss weight factor.

8. A dense scene text detection and recognition device, characterized in that: include: Data acquisition module, used to obtain public datasets and dense text image datasets; A first model training module is configured to establish a deep learning network model based on a word-length-aware Transformer, and pre-train the deep learning network model using the public dataset to generate an initial recognition model; wherein the deep learning network model is composed of an encoder and a decoder containing multiple dynamic heads, and is configured to extract text detection features and recognition features, fuse multi-scale semantic information, predict text bounding box information, and predict character sequences. The deep learning network model uses a character count prior customized for different word lengths to effectively constrain regions of interest, and retains a word-length-aware instance segmentation suggestion head; A second model training module is used to train the initial recognition model using the dense text image dataset to optimize the initial recognition model and generate a dense scene text detection and recognition collaborative model; The recognition module is used to obtain a dense scene text image to be recognized, and input the dense scene text image to be recognized into the dense scene text detection and recognition collaborative model to obtain a recognition result.

9. An electronic device, characterized in that: include: processor and memory; The memory stores a computer program executable by the processor; When the processor executes the computer program, the steps of the dense scene text detection and recognition method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that include: The computer-readable storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the steps in the dense scene text detection and recognition method as described in any one of claims 1 to 7.

Images