Model training method, method for detecting and recognizing text, and related device

Abstract

The present application provides a model training method, a device for detecting and recognizing text, an electronic device, a computer-readable medium, and a computer program product. The method of the present application includes: obtaining a training image set corresponding to a target task, wherein the target task is to simultaneously detect and recognize text in an image; using a denoising training method based on a Bezier curve to train a target model, wherein the method freezes target parameters to train the vocabulary mapping layer of the target model and the portion corresponding to the target task; after unfreezing the frozen parameters, fine-tuning the target model based on the training image set. The present application trains a model using a denoising method based on a Bezier curve, and then uses the trained model to simultaneously detect and recognize text contained in an input image, thereby improving the accuracy of text detection and recognition and improving efficiency.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, a device for detecting and recognizing text, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] Scene text-related tasks are an important research direction in computer vision, involving multiple subtasks of detecting, recognizing, and linking text in natural scenes. These tasks are of great significance to fields such as image search, instant translation, robot navigation, and industrial automation. Traditional scene text detection models, such as DBNet, strive to improve text detection performance through a multi-step approach based on specialized representations of text components. Recognition models, such as ParSeq, attempt to improve text recognition performance by using deep learning techniques, particularly models based on CNNs and RNNs.

[0003] However, traditional scene text detection and text recognition tasks generally use a detection-first-then-recognition approach, with detection and recognition as separate steps. In reality, detection and recognition have a synergistic effect: features from detection can aid recognition, and features from recognition can also feed back into detection. Summary of the Invention

[0004] Various aspects of the present application provide a model training method, a device for detecting and recognizing text, an electronic device, a computer-readable medium, and a computer program product.

[0005] In one aspect of the present application, a training method is provided, wherein the method comprises:

[0006] Obtaining a training image set corresponding to a target task, wherein the target task is to simultaneously detect and recognize text in an image;

[0007] Training the target model using a Bezier curve-based denoising training method, wherein the method trains the vocabulary mapping layer of the target model and the portion corresponding to the target task by freezing the target parameters;

[0008] After unfreezing the frozen parameters, fine-tune the target model based on the training image set.

[0009] In one aspect of the present application, a method for detecting and recognizing text is provided, wherein the method comprises:

[0010] Acquire the target scene image to be processed;

[0011] Using the trained target model to perform detection and recognition processing on the target scene image, the detection and recognition processing simultaneously performs text position detection and text content recognition on the text contained in the target scene image;

[0012] The target model is obtained by training using the model training method of the embodiment of the present application.

[0013] In one aspect of the present application, a model training device is provided, wherein the device comprises:

[0014] Means for obtaining a training image set corresponding to a target task, wherein the target task is to simultaneously detect and recognize text in an image;

[0015] An apparatus for training a target model using a Bezier curve-based denoising training method, wherein the method trains a vocabulary mapping layer of the target model and a portion corresponding to a target task by freezing target parameters;

[0016] A device for fine-tuning a target model based on a training image set after unfreezing the frozen parameters.

[0017] In one aspect of the present application, a device for detecting and recognizing text is provided, wherein the device comprises

[0018] Device for acquiring a target scene image to be processed;

[0019] A device for performing detection and recognition processing on a target scene image using a trained target model, wherein the detection and recognition processing simultaneously performs text position detection and text content recognition on text contained in the target scene image;

[0020] The target model is obtained by training using the model training method of the embodiment of the present application.

[0021] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0022] In another aspect of the present application, a computer program product is provided, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0023] In the solution provided in the embodiment of the present application, a denoising method based on Bezier curves is adopted to train the model, and then the trained model is used to simultaneously detect and recognize the text contained in the input image, thereby improving the accuracy of text detection and recognition and improving efficiency; in the training phase, parameters other than the vocabulary and the parameters related to the text scene task to be executed are frozen, so that only the vocabulary mapping layer and the task-related parts are trained, and in the fine-tuning phase, all learnable parameters are unfrozen to fine-tune the model based on the dataset corresponding to the task. This training method reduces the cost of training and fine-tuning on the target dataset, and can complete the detection of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction is given below to the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0025] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0026] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown;

[0027] Figure 2 A flowchart of a method for detecting and recognizing text according to an embodiment of the present application is shown.

[0028] Figure 3 shows an exemplary model structure schematic diagram according to an embodiment of the present application;

[0029] Figure 4 A schematic diagram of the structure of the model training device provided in an embodiment of the present application is shown;

[0030] Figure 5 A schematic structural diagram of an apparatus for detecting and recognizing text according to an embodiment of the present application is shown;

[0031] Figure 6 A structural diagram of a device suitable for implementing the solution in the embodiments of the present application is shown.

[0032] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0033] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0034] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0035] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0036] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for information storage. The information can be computer program instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc-read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0037] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown, wherein the method comprises at least step S101, step S102 and step S103.

[0038] In practical scenarios, the execution subject of this method can be a network device or an application running on a network device. The network device includes, but is not limited to, a network host, a single network server, a set of multiple network servers, or a collection of computers based on cloud computing, and can be used to implement some of the processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a group of loosely coupled computers forming a virtual computer.

[0039] The concepts involved in the embodiments of this application are explained below.

[0040] OCR: Optical Character Recognition.

[0041] Transformer Architecture: Models based on the Transformer architecture are a type of deep learning model. Key features of the Transformer model include: Self-Attention: This mechanism allows the model to process a sequence so that each element interacts directly with other elements in the sequence, regardless of their position within the sequence. This mechanism enables the model to capture the relationship between any two elements in the sequence, regardless of their distance in the sequence. Parallel Processing: Models based on the Transformer architecture can process all elements in a sequence in parallel, significantly improving training efficiency. Encoder-Decoder Architecture: Models based on the Transformer architecture use an encoder-decoder architecture for sequence-to-sequence tasks, such as machine translation. The encoder processes the input sequence, and the decoder generates the output sequence. Each encoder and decoder consists of multiple identical layers, each incorporating a self-attention mechanism and a feedforward neural network. Positional Encoding: Because the Transformer model itself does not capture positional information within a sequence, a positional encoding is introduced to provide information about the position of each element in the sequence. This positional encoding is typically a vector of the same length as the input sequence. It is added to the input sequence through a learned or predefined method to help the model understand the order of elements in the sequence. Multi-head attention: The self-attention mechanism in models based on the Transformer architecture can be expanded to multi-head attention, which captures information from different representation subspaces simultaneously. This increases the expressive power of the model, allowing it to capture different information in different representation subspaces. Layer normalization and residual connections: The Transformer model applies layer normalization to the output of each sublayer (self-attention layer and feedforward neural network layer) and uses residual connections. These techniques help avoid the problem of vanishing or exploding gradients in deep network training.

[0042] Bezier curve: The Bezier curve method is a technology used for natural scene text detection and recognition (STR), which uses Bezier curves to represent and fit the shape of text areas instead of traditional rectangles or polygons.

[0043] In some embodiments, the model of the embodiment of the present application is an end-to-end model based on the Transformer architecture, which is used to simultaneously perform text position detection and text content recognition on the text contained in the target scene image.

[0044] In some embodiments, when training the model of the present application, all parameters other than the vocabulary and the parameters related to the text scene task to be performed are frozen, so that only the vocabulary mapping layer and the task-related text classification head are trained. During the fine-tuning phase, all learnable parameters are unfrozen to fine-tune the model based on the dataset corresponding to the text scene task.

[0045] exist Figure 1 Before step S101 shown, the method further includes step S104.

[0046] In step S104, the target model is pre-trained so that the target model can learn a universal feature representation.

[0047] Specifically, pre-training on a large public dataset allows the target model to learn universal feature representations that can capture the basic patterns in images and general properties of objects.

[0048] During the pre-training phase, the model loads a pre-trained vocabulary (also called a vocabulary or embedding layer). This embedding layer contains pre-trained vocabulary weights. These weights are learned from a large amount of data and help the model better understand and process text data. Another important benefit of pre-trained weights is that they reduce the time and resource consumption of model training. Because the model has already learned some common features, subsequent tasks only require fine-tuning these features, without having to train from scratch.

[0049] Refer to the following Figure 1 To illustrate, in step S101 , a training image set corresponding to a target task is obtained.

[0050] The target task is to simultaneously detect and recognize text in an image.

[0051] The training image set includes annotation data. The annotation information includes the location and content of the text, for example, the coordinates of the four corners of the text box and the text content within the text box.

[0052] In step S102, a Bezier curve-based noise reduction training method is used to train the target model, wherein the method freezes parameters other than the target parameters in the target model to train the corresponding parts of the target parameters.

[0053] According to one embodiment, the target parameters include parameters related to the vocabulary and parameters related to the target task.

[0054] The vocabulary-related parameters include parameters related to the model's embedding layer or other parameters directly related to the vocabulary. Optionally, the vocabulary-related parameters include parameters of the word embedding layer of the expanded new vocabulary. This approach, after the vocabulary is expanded, can achieve excellent detection results for special characters with a small sample size.

[0055] The parameters related to the target task include parameters related to the detection structure of the model. The detection structure is used to detect and locate text contained in the input image.

[0056] By freezing all parameters in the target model except the target parameters, the model can focus on training the vocabulary and the parts related to the target task to be performed. Based on this training method, the model does not need to learn all parameters at the same time, thus improving the model training efficiency.

[0057] According to one embodiment, the method may determine which parameters need to be frozen by obtaining pre-stored freezing parameter indication information. Alternatively, the method may determine the parameters need to be frozen in real time based on user input.

[0058] According to one embodiment, the method includes steps S1021 to S1025 in which the method uses a Bezier curve-based denoising training method to train the target model in S102.

[0059] The Bezier curve-based denoising training method is targeted at text recognition tasks based on the Transformer architecture. By improving traditional denoising training, the method makes the model more stable and efficient when processing text of arbitrary shapes.

[0060] In step S1021 , the query of the noise reduction part is decomposed into a noise position query and a noise content query.

[0061] In step S1022 , a noise position query is generated using the four control points of the Bezier curve, and after adding noise to the Bezier control points, a new set of Bezier control points is obtained.

[0062] In step S1023 , the coordinates of multiple points are uniformly sampled from the generated Bezier curve using the Bezier control points to which noise is added.

[0063] In step S1024, the obtained coordinates are position-encoded and processed by a two-layer multilayer perceptron (MLP) to obtain a corresponding noise position query.

[0064] In step S1025 , a noise content query is initialized by a mask character sliding method to assist in the alignment of text content and position.

[0065] The Mask Character Sliding (MCS) method is used to assist in aligning text content and position.

[0066] Specifically, we first determine the number of valid characters t, which refers to the number of characters in the input sequence that actually make sense. Then, we perform an integer division operation To calculate the number of times each valid character should be cloned so that the total length T of the sequence is evenly distributed to each valid character. In addition, since T may not be completely divisible by t, there will be a remainder Indicates that there are k additional spaces to be allocated. In order to distribute these additional spaces fairly, an additional clone will be allocated to each of the first k valid characters in the sequence to ensure that each character is allocated as evenly as possible.

[0067] After obtaining the sliding characters, a masking operation is used to control the number of consecutive characters. This means that a portion of the consecutive characters are flipped to background labels with a certain probability. After processing the positive part, noise is added to the characters in the positive and negative parts, causing these characters to flip to other characters with a probability of λ. These characters are then embedded to obtain the noise content query, where all the characters initialized in the negative part are background.

[0068] According to one embodiment, when performing a task using a trained model, the portion of applying the Bezier curve-based denoising training method is removed during model inference.

[0069] Among them, the part that applies the Bezier curve-based denoising training method can be removed during model inference by setting the denoising training part to an inactive state in the model's configuration file or code or directly disabling the layer in the model that applies denoising training.

[0070] By removing the part of the model that applies the Bezier curve-based denoising training method during the model inference phase, the efficiency of the inference process is ensured without adding additional computational burden.

[0071] According to one embodiment, the target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder, and a decoder. This embodiment trains the model by performing the following steps:

[0072] The backbone network extracts global features from the input image to obtain corresponding global and local features. Optionally, the input image is preprocessed before feature extraction. This may include resizing, normalization, contrast enhancement, and other operations to ensure that the model receives input in a consistent format. The global features obtained by the backbone network help the model understand the global context of the image.

[0073] Next, the encoder further extracts global features of the input image. The global features obtained by the backbone network help the model understand the global context of the image.

[0074] Next, the decoder further extracts local features from the input image based on the global features to obtain the corresponding local features. To gain a more detailed understanding of the image content, the model needs to further extract local features. This is usually achieved by using deeper network structures that can identify detailed information such as edges, corners, and textures in the image. Local features are particularly important for text detection because they help the model recognize the outline and shape of text.

[0075] Next, a Bezier curve-based denoising training method is applied to the decoder to train the model to effectively handle noise and interference in the image. During model inference, this Bezier curve-based denoising training is removed to ensure efficient inference without adding additional computational burden.

[0076] The local features output by the decoder layer are used to locate the text regions in the image and identify the corresponding text content.

[0077] The model based on the Transformer architecture of this embodiment further includes a detection head and a recognition head. The method performs text detection through the detection head and performs text recognition through the recognition head, thereby achieving simultaneous text detection and text recognition in one model.

[0078] Specifically, the detection head is responsible for accurately identifying the position and shape of the text area in the image, while the recognition head is responsible for in-depth analysis of the detected text area to achieve accurate recognition of the text content.

[0079] This embodiment applies a Bezier curve-based denoising training method in a model based on the Transformer architecture, which alleviates the instability of bipartite graph matching used in the Transformer framework based on DETR (Detection Transformer), and greatly promotes model convergence during training.

[0080] Continue to refer to Figure 1To illustrate, in step S103 , after the frozen parameters are unfrozen, the target model is fine-tuned based on the target training image set.

[0081] Fine-tuning is a process that preserves the useful features learned by the model during pre-training while adapting it to the new task with a small amount of adjustments. This process can significantly improve the performance of the model on a specific task while reducing training cost and time.

[0082] Optionally, during fine-tuning, the pre-trained target model can be structurally adjusted based on the requirements of the target task. For example, in an image recognition task, the final classification layer may need to be replaced to match the number of categories in the target dataset.

[0083] The method provided in the embodiment of the present application adopts a Bezier curve-based denoising method to train the model, and then uses the trained model to simultaneously detect and recognize the text contained in the input image, thereby improving the accuracy of text detection and recognition and improving efficiency; freezing parameters other than the vocabulary and the parameters related to the text scene task to be executed during the training phase, so that only the vocabulary mapping layer and the task-related parts are trained, and unfreezing all learnable parameters during the fine-tuning phase to fine-tune the model based on the dataset corresponding to the task. This training method reduces the cost of training and fine-tuning on the target dataset, and can complete the detection of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training.

[0084] Figure 2 A flowchart of a method for detecting and recognizing text according to an embodiment of the present application is shown. Figure 2 The steps shown include step S201 and step S202.

[0085] Reference Figure 2 , in step S201, the target scene image to be processed is obtained.

[0086] The target scene images may include photos or video frames of various types of scenes. For example, for urban scenes, the target scene images may include images showing urban buildings, streets, traffic, road signs, etc. For indoor scenes, the target scene images may include images showing indoor environments such as homes, offices, and stores.

[0087] In step S202, the trained target model is used to perform detection and recognition processing on the target scene image.

[0088] The detection and recognition process simultaneously performs text position detection and text content recognition on the text contained in the target scene image.

[0089] The trained target model is obtained by training and fine-tuning the above steps S101 to S103 based on the data set corresponding to the task.

[0090] Specifically, when performing text location detection, the trained target model is used to perform boundary detection to predict the vertices or boundary points of the text region. The detected boundary points are then converted into the location information of one or more text regions. For example, the detected boundary points can be converted into adaptive text region representations, such as polygons or curves.

[0091] Specifically, when performing text content recognition, for each detected text area, the boundaries between characters are determined through horizontal and vertical projection analysis, and the detected text area is segmented into individual characters. Then, each segmented character is recognized using .

[0092] The text detection and recognition method of the embodiment of the present application realizes simultaneous detection and recognition of text in an image, thereby improving the accuracy and robustness of scene text detection and recognition.

[0093] Refer to the following Figure 3 The exemplary model structure shown is used to illustrate the method of the embodiment of the present application. The model of this example is a model based on the transformer structure.

[0094] Reference Figure 3 The model in this example mainly consists of three parts: the backbone network, the encoder, and the decoder. The encoder contains multiple encoder layers, and the decoder contains multiple decoder layers. These three parts are introduced below.

[0095] 1) Backbone network: The backbone network in this example is the ViTAE-v2 backbone network, which processes the input image and extracts global features. These features contain the basic visual information of the image and provide the basis for subsequent text detection and recognition.

[0096] 2) Encoder: The encoder further extracts global features of the input image.

[0097] 3) Decoder: The decoder extracts local features of the input image. The local features output by the decoder layer are used to locate the text area in the input image and recognize the text content.

[0098] The Bezier curve-based denoising training method is applied at the decoder layer to train the model to learn how to effectively handle noise and interference in images, thereby further optimizing the model's performance. Furthermore, the denoising training method helps the model better understand and handle changes in text shape, such as curved or tilted text, thereby providing more reliable detection or recognition results in practical applications. When performing model inference, the application of the Bezier curve-based denoising training method is removed to ensure the efficiency of the inference process without adding additional computational burden.

[0099] The Transformer-based model in this example also includes a detection head and a recognition head. These heads are used to locate and identify text content, respectively, enabling simultaneous text detection and recognition within a single model. The detection head accurately identifies the location and shape of text regions within an image, while the recognition head conducts in-depth analysis of the detected text regions to accurately identify the text content.

[0100] This example uses a hybrid training method to train Figure 3 The model shown. The process of this hybrid training method includes:

[0101] 1) Freeze the parameters other than the vocabulary and the parameters related to the target task to be executed, and train the parameters of the word embedding layer of the expanded new vocabulary and the parameters of the detection head and recognition head related to the target task;

[0102] The main function of the vocabulary mapping layer (Embedding layer) is to map discrete words into a continuous vector space. This mapping places similar words close together in the vector space, thereby capturing the semantic relationships between words. Through training, the Embedding layer learns the contextual information of each word, enabling the model to better understand semantics when processing text.

[0103] 2) Obtain the target data set for the task to be performed;

[0104] 3) Unfreeze all learnable parameters and fine-tune the model based on the target dataset.

[0105] After unfreezing all learnable parameters, the model recalibrates all parameters, including the weights of the embedding layers, detection heads, and recognition heads. This process allows the model to better adapt to the specific target dataset. Fine-tuning typically requires several thousand steps of training to ensure good performance on the new dataset. Through fine-tuning, the model not only leverages the knowledge learned during pre-training but also further optimizes based on the characteristics of the target dataset, thereby improving classification accuracy.

[0106] When performing text detection and recognition tasks, the input image is input into the trained model of this example. The detection head and recognition head of the model predict the location of the text contained in the input image based on the local features output by the decoder and simultaneously identify the corresponding text content. The location information of the corresponding one or more text areas and the identified text content are used as the text detection and recognition results.

[0107] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown. The device includes: a device for acquiring a training image set corresponding to a target task (hereinafter referred to as "training image acquisition device 101"), a device for acquiring a training image set corresponding to a target task (hereinafter referred to as "noise reduction training device 102"), and a device for fine-tuning a target model based on the training image set after unfreezing frozen parameters (hereinafter referred to as "model fine-tuning device 103").

[0108] According to one embodiment, the device further comprises a pre-training device.

[0109] The pre-training device pre-trains the target model so that the target model can learn a universal feature representation.

[0110] Specifically, the pre-training device pre-trains the target model on a large public dataset, allowing the target model to learn universal feature representations that can capture the basic patterns in images and general properties of objects.

[0111] During the pre-training phase, the model loads a pre-trained vocabulary (also called a vocabulary or embedding layer). This embedding layer contains pre-trained vocabulary weights. These weights are learned from a large amount of data and help the model better understand and process text data. Another important benefit of pre-trained weights is that they reduce the time and resource consumption of model training. Because the model has already learned some common features, subsequent tasks only require fine-tuning these features, without having to train from scratch.

[0112] Reference Figure 4 , the training image acquisition device 101 acquires a training image set corresponding to the target task.

[0113] The target task is to simultaneously detect and recognize text in an image.

[0114] The training image set includes annotation data. The annotation information includes the location and content of the text, for example, the coordinates of the four corners of the text box and the text content within the text box.

[0115] The noise reduction training device 102 uses a noise reduction training method based on Bezier curves to train the target model.

[0116] The noise reduction training device 102 performs training on the part corresponding to the target parameter by freezing the parameters other than the target parameter in the target model.

[0117] According to one embodiment, the target parameters include parameters related to the vocabulary and parameters related to the target task.

[0118] The vocabulary-related parameters include parameters related to the model's embedding layer or other parameters directly related to the vocabulary. Optionally, the vocabulary-related parameters include parameters of the word embedding layer of the expanded new vocabulary. This approach, after the vocabulary is expanded, can achieve excellent detection results for special characters with a small sample size.

[0119] The parameters related to the target task include parameters related to the detection structure of the model. The detection structure is used to detect and locate text contained in the input image.

[0120] By freezing all parameters in the target model except the target parameters, the model can focus on training the vocabulary and the parts related to the target task to be performed. Based on this training method, the model does not need to learn all parameters at the same time, thus improving the model training efficiency.

[0121] According to one embodiment, the device may determine which parameters need to be frozen by acquiring pre-stored freezing parameter indication information. Alternatively, the device may determine the parameters need to be frozen in real time based on user input.

[0122] According to one embodiment, the noise reduction training device 102 further includes a query decomposition device, a control point generation device, a coordinate point sampling device, a position query acquisition device, and a noise initialization device.

[0123] The Bezier curve-based denoising training method is targeted at text recognition tasks based on the Transformer architecture. By improving traditional denoising training, the method makes the model more stable and efficient when processing text of arbitrary shapes.

[0124] The query decomposition device decomposes the query of the noise reduction part into a noise position query and a noise content query.

[0125] The control point generating device generates a noise position query using four control points of the Bezier curve, and obtains a set of new Bezier control points after adding noise to the Bezier control points.

[0126] The coordinate point sampling device uses Bezier control points with added noise to uniformly sample multiple point coordinates from the generated Bezier curve.

[0127] The location query acquisition device obtains the corresponding noise location query by performing location encoding and two-layer multilayer perceptron (MLP) processing on the obtained coordinates.

[0128] The noise initialization device initializes the noise content query through the mask character sliding method.

[0129] The noise device is initialized by using the Mask Character Sliding (MCS) method to assist in the alignment of text content and position.

[0130] According to one embodiment, in the stage of performing a task using a trained model, when performing model inference, the apparatus removes the portion of applying a Bezier curve-based noise reduction training method.

[0131] Among them, the part that applies the Bezier curve-based denoising training method can be removed during model inference by setting the denoising training part to an inactive state in the model's configuration file or code or directly disabling the layer in the model that applies denoising training.

[0132] By removing the part of the model that applies the Bezier curve-based denoising training method during the model inference phase, the efficiency of the inference process is ensured and additional computational burden is avoided.

[0133] According to one embodiment, the target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder, and a decoder. This embodiment trains the model by performing the following operations:

[0134] The backbone network extracts global features from the input image to obtain corresponding global and local features. Optionally, the input image is preprocessed before feature extraction. This may include resizing, normalization, contrast enhancement, and other operations to ensure that the model receives input in a consistent format. The global features obtained by the backbone network help the model understand the global context of the image.

[0135] Next, the encoder further extracts global features of the input image. The global features obtained by the backbone network help the model understand the global context of the image.

[0136] Next, the decoder further extracts local features from the input image based on the global features to obtain the corresponding local features. To gain a more detailed understanding of the image content, the model needs to further extract local features. This is usually achieved by using deeper network structures that can identify detailed information such as edges, corners, and textures in the image. Local features are particularly important for text detection because they help the model recognize the outline and shape of text.

[0137] Next, a Bezier curve-based denoising training method is applied to the decoder to train the model to effectively handle noise and interference in the image. During model inference, this Bezier curve-based denoising training is removed to ensure efficient inference without adding additional computational burden.

[0138] The local features output by the decoder layer are used to locate the text regions in the image and identify the corresponding text content.

[0139] The model based on the Transformer architecture in this embodiment also includes a detection head and a recognition head. The method performs text detection through the detection head and performs text recognition through the recognition head, thereby achieving simultaneous text detection and text recognition in one model.

[0140] Specifically, the detection head is responsible for accurately identifying the position and shape of the text area in the image, while the recognition head is responsible for in-depth analysis of the detected text area to achieve accurate recognition of the text content.

[0141] This embodiment applies a Bezier curve-based denoising training method in a model based on the Transformer architecture, which alleviates the instability of bipartite graph matching used in the DETR-based Transformer framework and greatly promotes model convergence during training.

[0142] Continue to refer to Figure 4 To explain, after unfreezing the frozen parameters, the model fine-tuning device 103 fine-tunes the target model based on the target training image set.

[0143] Fine-tuning is a process that preserves the useful features learned by the model during pre-training while adapting it to the new task with a small amount of adjustments. This process can significantly improve the performance of the model on a specific task while reducing training cost and time.

[0144] Optionally, during the fine-tuning process, the model fine-tuning device 103 performs structural adjustments on the pre-trained target model according to the requirements of the target task to be performed. For example, in an image recognition task, the last classification layer may need to be replaced to match the number of categories in the target dataset.

[0145] According to the device of the present application, a denoising method based on Bezier curves is adopted to train the model, and then the trained model is used to simultaneously detect and recognize the text contained in the input image, thereby improving the accuracy of text detection and recognition and improving efficiency; in the training phase, parameters other than the vocabulary and the parameters related to the text scene task to be executed are frozen, so that only the vocabulary mapping layer and the task-related parts are trained, and in the fine-tuning phase, all learnable parameters are unfrozen to fine-tune the model based on the dataset corresponding to the task. This training method reduces the cost of training and fine-tuning on the target dataset, and can complete the detection of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training.

[0146] Figure 5 A schematic structural diagram of an apparatus for detecting and recognizing text according to an embodiment of the present application is shown.

[0147] The device includes: a device for acquiring a target scene image to be processed (hereinafter referred to as "input image acquisition device 201") and a device for detecting and identifying the target scene image using a trained target model (hereinafter referred to as "detection and identification device 202").

[0148] Reference Figure 5 , the input image acquisition device 201 acquires the target scene image to be processed.

[0149] The target scene images may include photos or video frames of various types of scenes. For example, for urban scenes, the target scene images may include images showing urban buildings, streets, traffic, road signs, etc. For indoor scenes, the target scene images may include images showing indoor environments such as homes, offices, and stores.

[0150] The detection and recognition device 202 uses the trained target model to perform detection and recognition processing on the target scene image.

[0151] The detection and recognition process simultaneously performs text position detection and text content recognition on the text contained in the target scene image.

[0152] The trained target model is obtained by training and fine-tuning the training image acquisition device 101, the noise reduction training device 102 and the model fine-tuning device 103 based on the data set corresponding to the task.

[0153] Specifically, when detecting text positions, the detection and recognition device 202 uses the trained target model to perform boundary detection to predict the vertices or boundary points of the text region. The detected boundary points are then converted into position information of one or more text regions. For example, the detected boundary points are converted into adaptive text region representations, such as polygons or curves.

[0154] Specifically, when the detection and recognition device 202 performs text content recognition, for each detected text region, the boundaries between characters are determined by horizontal and vertical projection analysis, and the detected text region is segmented into individual characters. Then, each segmented character is recognized using .

[0155] The apparatus for text detection and recognition in the embodiment of the present application realizes simultaneous detection and recognition of text in an image, thereby improving the accuracy and robustness of scene text detection and recognition.

[0156] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application. The method corresponding to the electronic device can be the model training method or the method for detecting and recognizing text in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0157] The electronic device may be a user device, or a device formed by integrating a user device and a network device via a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablets, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer collections, and can be used to implement some of the processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computers.

[0158] Figure 6The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown. The device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 into the random access memory (RAM) 1203. Various programs and data required for system operation are also stored in RAM 1203. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through a bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0159] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, and a speaker; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, and a semiconductor memory; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 1209 performs communication processing via a network such as the Internet.

[0160] In particular, the methods and / or embodiments of the present application can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above-mentioned functions defined in the method of the present application are performed.

[0161] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application.

[0162] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0163] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0164] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0165] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0166] The flow chart or block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the equipment, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code include one or more executable instructions for realizing the logical function of the specification. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs the function or operation of the specification, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0169] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0170] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0171] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0173] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

Claims

1. A model training method, comprising: Obtaining a training image set corresponding to a target task, wherein the target task is to simultaneously detect and recognize text in an image; Training the target model using a Bezier curve-based denoising training method, wherein the method freezes parameters other than the target parameters in the target model to train the portion corresponding to the target parameters; After unfreezing the frozen parameters, fine-tune the target model based on the training image set; The method further comprises: Pre-training the target model so that the target model can learn a universal feature representation; The target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder, and a decoder. The method trains the model by the following steps: Perform global feature extraction on the input image in the backbone network to obtain corresponding global features and local features; Further extract global features of the input image in the encoder; In the decoder, based on the global features, the input image is further subjected to local feature extraction processing to obtain corresponding local features; Apply the Bezier curve-based denoising training method in the decoder to train the model; The Transformer architecture-based model further includes a detection head and a recognition head, wherein text detection is performed by the detection head and text recognition is performed by the recognition head.

2. The method according to claim 1, wherein The method of using the Bezier curve-based denoising training method to train the target model includes: Decompose the noise reduction part query into noise location query and noise content query; Generate noise position query using the four control points of the Bezier curve, and obtain a new set of Bezier control points after adding noise to the Bezier control points; Using the Bezier control points with added noise, uniformly sample multiple point coordinates from the generated Bezier curve; By performing position encoding and two-layer MLP processing on the obtained coordinates, the corresponding noise position query is obtained; Noisy content queries are initialized through a masked character sliding method to assist in the alignment of text content and location.

3. A method for detecting and recognizing text, wherein: The method comprises: Acquire the target scene image to be processed; Using the trained target model to perform detection and recognition processing on the target scene image, the detection and recognition processing simultaneously performs text position detection and text content recognition on the text contained in the target scene image; Wherein, the target model is obtained by training using the model training method described in claim 1 or 2.

4. The method according to claim 3, wherein: The method further comprises: When performing model inference, remove the part that applies the Bezier curve-based denoising training method.

5. A model training device, wherein: The device comprises: Means for obtaining a training image set corresponding to a target task, wherein the target task is to simultaneously detect and recognize text in an image; A device for training a target model using a Bezier curve-based noise reduction training method, wherein the device performs training on a portion corresponding to the target parameter by freezing parameters other than the target parameter in the target model; A device for fine-tuning the target model based on the training image set after unfreezing the frozen parameters; The device further includes a pre-training device, which pre-trains the target model so that the target model can learn a universal feature representation; The target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder, and a decoder. The device trains the model by performing the following operations: Perform global feature extraction on the input image in the backbone network to obtain corresponding global features and local features; Further extract global features of the input image in the encoder; In the decoder, based on the global features, the input image is further subjected to local feature extraction processing to obtain corresponding local features; Apply the Bezier curve-based denoising training method in the decoder to train the model; The Transformer architecture-based model further includes a detection head and a recognition head, wherein text detection is performed by the detection head and text recognition is performed by the recognition head.

6. A device for text detection, wherein: The device comprises: Device for acquiring a target scene image to be processed; A device for performing detection and recognition processing on a target scene image using a trained target model, wherein the detection and recognition processing simultaneously performs text position detection and text content recognition on text contained in the target scene image; Wherein, the target model is obtained by training using the model training method described in claim 1 or 2.

7. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method of claim 1 or 2, or the method of claim 3 or 4.

8. A computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to claim 1 or 2, or to perform the method according to claim 3 or 4.

9. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to claim 1 or 2, or performs the method according to claim 3 or 4.

Images