Model training method and device, text detection method and device, medium and program product

Abstract

The invention provides a model training method and device, a text detection method and device, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps of obtaining a training image set corresponding to a target task; the target model is trained by using a Bezier curve-based noise reduction training method, and according to the method, parameters except target parameters in the target model are frozen to train corresponding parts of the target parameters; and after unfreezing the frozen parameters, performing fine adjustment on the target model based on the training image set. According to the method, the model is trained by adopting the noise reduction method based on the Bezier curve, and then the trained model is used for detecting the text which may have an irregular shape in the input image, so that the accuracy of text detection is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, a text detection method, a device, an electronic device, a computer-readable medium, and a computer program product. Background Art

[0002] Scene text-related tasks are an important research direction in the field of computer vision, which involves multiple subtasks of detecting, recognizing and linking text from natural scenes. These tasks are of great significance to image search, instant translation, robot navigation, industrial automation and other fields.

[0003] Traditional scene text detection models, such as DBNet, are committed to improving the performance of text detection through a multi-step approach based on special representations of text components, while recognition models, such as ParSeq, try to improve text recognition performance by using deep learning techniques, especially CNN and RNN-based models.

[0004] However, solutions based on existing technologies still face challenges in handling text detection in complex scenes, especially the detection of irregularly shaped text. Summary of the invention

[0005] Multiple aspects of the present application provide a model training method, a text detection method, an apparatus, an electronic device, a computer-readable medium, and a computer program product.

[0006] In one aspect of the present application, a training method is provided, wherein the method comprises:

[0007] Obtain the training image set corresponding to the target task;

[0008] Using a Bezier curve-based denoising training method to train a target model, wherein the method freezes parameters other than the target parameters in the target model to train a portion corresponding to the target parameters;

[0009] After unfreezing the frozen parameters, fine-tune the target model based on the training image set.

[0010] In one aspect of the present application, a text detection method is provided, wherein the method comprises:

[0011] Acquire the target scene image to be processed;

[0012] Use the trained target model to perform text detection and obtain the location information of one or more text regions output by the target model;

[0013] Wherein, the target model is obtained by training using the model training method according to the embodiment of the present application.

[0014] In one aspect of the present application, a model training device is provided, wherein the device comprises:

[0015] Means for obtaining a training image set corresponding to a target task;

[0016] A device for training a target model using a Bezier curve-based denoising training method, wherein the device performs training on a portion corresponding to the target parameter by freezing parameters other than the target parameter in the target model;

[0017] A device for fine-tuning a target model based on a training image set after unfreezing the frozen parameters.

[0018] In one aspect of the present application, a device for text detection is provided, wherein the device comprises:

[0019] A device for acquiring a target scene image to be processed;

[0020] A device for performing text detection using a trained target model to obtain position information of one or more text regions output by the target model;

[0021] Wherein, the target model is obtained by training using the model training method according to the embodiment of the present application.

[0022] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0023] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0024] In the solution provided in the embodiment of the present application, a denoising method based on Bezier curves is used to train the model, and then the trained model is used to detect text that may have irregular shapes in the input image, thereby improving the accuracy of text detection; in the training phase, parameters other than the vocabulary and parameters related to the target task to be executed are frozen, so that only the vocabulary mapping layer and the task-related detection head are trained, and in the fine-tuning phase, all learnable parameters are unfrozen to fine-tune the model based on the data set corresponding to the target task. This training method reduces the cost of training and fine-tuning on the target data set, and can achieve the detection of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0026] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0027] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown;

[0028] Figure 2 A flowchart of a text detection method according to an embodiment of the present application is shown.

[0029] Figure 3 A schematic diagram of an exemplary model structure according to an embodiment of the present application is shown;

[0030] Figure 4 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown;

[0031] Figure 5 A schematic diagram of the structure of a device for detecting text according to an embodiment of the present application is shown;

[0032] Figure 6 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0033] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0035] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0036] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0037] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device.

[0038] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown. The method at least includes step S101, step S102 and step S103.

[0039] In actual scenarios, the execution subject of the method can be a network device, or an application running on a network device, wherein the network device includes but is not limited to a network host, a single network server, a plurality of network server sets, or a collection of computers based on cloud computing, and can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0040] The concepts involved in the embodiments of the present application are explained below.

[0041] Transformer architecture: Models based on the Transformer architecture are a type of deep learning model. The core features of the Transformer model include: Self-Attention: The self-attention mechanism allows the model to process a sequence in such a way that each element can interact directly with other elements in the sequence without relying on the position in the sequence. This mechanism enables the model to capture the relationship between any two elements in the sequence, no matter how far apart they are in the sequence. Parallel processing: Models based on the Transformer architecture can process all elements in the sequence in parallel, which greatly improves the training efficiency of the model. Encoder-Decoder Architecture: Models based on the Transformer architecture use the encoder-decoder architecture for sequence-to-sequence tasks such as machine translation. The encoder processes the input sequence and the decoder generates the output sequence. Each encoder and decoder consists of multiple identical layers, each of which contains a self-attention mechanism and a feedforward neural network. Positional encoding: Since the Transformer model itself does not have the ability to capture positional information in a sequence, positional encoding needs to be introduced to provide positional information for each element in the sequence. Positional encoding is usually a vector of the same length as the input sequence, which is added to the input sequence in a learned or predefined way to help the model understand the order of elements in the sequence. Multi-head attention: The self-attention mechanism in models based on the Transformer architecture can be extended to multi-head attention, that is, to capture information from different representation subspaces at the same time. This increases the expressiveness of the model, allowing it to capture different information in different representation subspaces. Layer normalization and residual connections: The Transformer model applies layer normalization on the output of each sublayer (self-attention layer and feedforward neural network layer) and uses residual connections. These techniques help avoid the problem of vanishing or exploding gradients in deep network training.

[0042] Bezier curve: The Bezier curve method is a technology used for natural scene text detection and recognition (STR), which uses Bezier curves to represent and fit the shape of text areas instead of traditional rectangles or polygons.

[0043] In some embodiments, the model of the embodiment of the present application is a model based on the Transformer architecture, which is used to detect text in an input image.

[0044] In some embodiments, when training the model of the present application, parameters other than the vocabulary and the parameters related to the text scene task to be performed are frozen, so that only the vocabulary mapping layer and the task-related text classification head are trained. In the fine-tuning stage, all learnable parameters are unfrozen, so that the model is fine-tuned based on the data set corresponding to the text scene task.

[0045] Refer to the following Figure 1 To illustrate, in step S101, a training image set corresponding to a target task is obtained.

[0046] The target task is to detect text contained in an image.

[0047] Optionally, the image includes text with irregular shapes. The training image set includes annotation data for boundaries of irregularly shaped text regions.

[0048] According to one embodiment, before step S101, the method further includes step S104.

[0049] In step S104, the target model is pre-trained so that the target model can learn a universal feature representation.

[0050] Specifically, pre-training on a large public dataset allows the target model to learn universal feature representations that can capture the basic patterns in images and general properties of objects.

[0051] During the pre-training phase, the model will load a pre-trained word list (also called a vocabulary or embedding layer). This embedding layer contains the pre-trained weights of the vocabulary, which are learned from a large amount of data and can help the model better understand and process text data. Another important function of pre-trained weights is to reduce the time and resource consumption of model training. Since the model has learned some common features, in subsequent tasks, only these features need to be fine-tuned without training from scratch.

[0052] In step S102, a Bezier curve-based denoising training method is used to train the target model.

[0053] The method freezes the parameters other than the target parameters in the target model to perform training on the corresponding parts of the target parameters.

[0054] According to one embodiment, the target parameters include parameters related to the vocabulary and parameters related to the target task.

[0055] The parameters related to the vocabulary include the embedding layer of the model or other parameters directly related to the vocabulary. Optionally, the parameters related to the vocabulary include the parameters of the word embedding layer of the expanded new vocabulary. After the vocabulary is expanded, this method can obtain excellent detection effect for special characters with a small sample size.

[0056] The parameters related to the target task include parameters related to the detection structure of the model. The detection structure is used to detect and locate text contained in the input image.

[0057] By freezing all parameters other than the target parameters in the target model, we can focus on the vocabulary part of the training model and the part related to the target task to be performed. Based on this training method, the model training efficiency is improved because it does not need to learn all parameters at the same time.

[0058] According to one embodiment, the method may determine which parameters need to be frozen by acquiring pre-stored freezing parameter indication information. Alternatively, the method may determine the parameters that need to be frozen in real time based on user input.

[0059] According to one embodiment, the method uses a Bezier curve-based denoising training method to train a target model in S102, including steps S1021 to S1025.

[0060] The Bezier curve-based denoising training method is targeted at text recognition tasks based on the Transformer architecture. This method improves traditional denoising training to make the model more stable and efficient when processing text of any shape.

[0061] In step S1021, the query of the noise reduction part is decomposed into a noise position query and a noise content query.

[0062] In step S1022, a noise position query is generated using four control points of the Bezier curve, and after adding noise to the Bezier control points, a new set of Bezier control points is obtained.

[0063] For the four Bezier control points of the Bezier center curve of any text instance, assuming that an image contains N text instances, the set of all instances can be expressed as S = {S j |j=1,2,…,N},S j is a set of 4 Bezier control points of the Bezier center curve of a text instance. Each instance can be represented as:

[0064] S j ={(x ij ,y ij )|i=0,1,2,3} (1)

[0065] Each point consists of a pair of coordinates (x ij ,y ij ), and the value of i ranges from 0 to 3, representing the 4 Bezier control points from left to right. Adding random noise to these points gives:

[0066] S′ j ={(x ij +Δx ij ,y ij +Δyij )|i=0,1,2,3} (2)

[0067] Where Δx ij with Δy ij It is obtained by calculating the Euclidean distance between the 4 Bezier control points of the center curve and the 4 Bezier control points of the upper boundary. After adding noise, a new Bezier control point set S' is obtained.

[0068] In step S1023, the Bezier control points with added noise are used to uniformly sample the coordinates of multiple points from the generated Bezier curve.

[0069] On the Bezier curve obtained by these Bezier control points, T points are uniformly taken, with a dimension of (N, T, 2).

[0070] In step S1024, the corresponding noise position query is obtained by performing position encoding and two-layer multilayer perceptron (MLP) processing on the obtained coordinates.

[0071] The coordinates obtained in step S1023 are subjected to position encoding (PE) and two layers of MLP to obtain a noise position query Q with a dimension of (N, T, 256) N . Q N It is expressed as follows:

[0072] Q N =MLP(PE(Points))(3)

[0073] In step S1025, the noise content query is initialized by the mask character sliding method.

[0074] The Mask Character Sliding (MCS) method is used to assist in aligning text content and position.

[0075] Specifically, we first determine the number of valid characters t, which refers to the number of characters in the input sequence that actually make sense. Then, we perform an integer division operation To calculate the number of times each valid character should be cloned so that the total length T of the sequence is evenly distributed to each valid character. In addition, since T may not be completely divisible by t, there will be a remainder Indicates that there are k additional spaces to be allocated. In order to distribute these additional spaces fairly, an additional clone will be allocated to each of the first k valid characters in the sequence to ensure that each character is allocated as evenly as possible.

[0076] After getting the sliding characters, use the mask operation to control the number of consecutive characters, that is, flip a part of the consecutive characters into background labels with a certain probability. After processing the positive part, add noise to the characters in the positive part and the negative part, so that these characters are flipped into other characters with a probability of λ. After embedding, these characters are obtained to obtain the noise content query, in which all the characters initialized in the negative part are background.

[0077] According to one embodiment, at the stage of performing a task using a trained model, when performing model inference, the portion of applying a Bezier curve-based denoising training method is removed.

[0078] Among them, the part that applies the Bezier curve-based denoising training method can be removed when performing model inference by setting the denoising training part to an inactive state in the configuration file or code of the model or directly disabling the layer that applies denoising training in the model.

[0079] By removing the part of the model that applies the Bezier curve-based denoising training method during the model inference phase, the efficiency of the inference process is ensured and additional computational burden is avoided.

[0080] According to one embodiment, the target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder and a decoder. This embodiment trains the model by performing the following steps:

[0081] The backbone network performs global feature extraction on the input image to obtain corresponding global features and local features. Optionally, the input image is preprocessed before feature extraction, which may include operations such as resizing, normalizing, and enhancing contrast to ensure that the model receives input in a consistent format. The global features obtained through the backbone network help the model understand the global context of the image.

[0082] Next, the encoder further extracts global features of the input image. The global features obtained through the backbone network help the model understand the global context of the image.

[0083] Next, in the decoder, based on the global features, the input image is further processed for local feature extraction to obtain the corresponding local features. In order to understand the image content more finely, the model needs to further extract local features. This is usually achieved by using deeper network structures that can identify detailed information such as edges, corners, and textures in the image. Local features are particularly important for text detection because they help the model recognize the outline and shape of the text.

[0084] Next, a Bezier curve-based denoising training method is applied in the decoder to train the model to learn how to effectively deal with noise and interference in the image.

[0085] Among them, the local features output by the decoder layer are used to locate the text area in the image.

[0086] The Transformer architecture-based model further includes a detection head, and the method uses the detection head to predict the position of the text contained in the input image based on the local features output by the decoder layer.

[0087] Optionally, the position information of one or more texts contained in the input image predicted by the detection head is used as the text detection result. The text position information includes but is not limited to information for indicating the position coordinates of the corner points and bounding boxes of the predicted text.

[0088] Optionally, the text position information may also include shape, direction information or tilt angle information of the predicted text.

[0089] Continue to refer to Figure 1 To explain, in step S103, after the frozen parameters are unfrozen, the target model is fine-tuned based on the target training image set.

[0090] Among them, fine-tuning is performed to retain the useful features that the model has learned in the pre-training phase, while adapting it to new tasks through a small amount of adjustment. The fine-tuning process can significantly improve the performance of the model on specific tasks while reducing training costs and time.

[0091] Optionally, during the fine-tuning process, the pre-trained target model is structurally adjusted according to the requirements of the target task to be performed. For example, in an image recognition task, the last classification layer may need to be replaced to match the number of categories in the target dataset.

[0092] According to the method of the present application, a denoising method based on Bezier curves is used to train the model, and then the trained model is used to detect text that may have irregular shapes in the input image, thereby improving the accuracy of text detection; in the training phase, parameters other than the vocabulary and parameters related to the target task to be executed are frozen, so that only the vocabulary mapping layer and the task-related detection head are trained, and in the fine-tuning phase, all learnable parameters are unfrozen to fine-tune the model based on the data set corresponding to the target task. This training method reduces the cost of training and fine-tuning on the target data set, and can achieve the detection of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training.

[0093] Figure 2A flowchart of a text detection method according to an embodiment of the present application is shown. Figure 2 The steps shown include step S201 and step S202.

[0094] Reference Figure 2 , in step S201, the target scene image to be processed is obtained.

[0095] The target scene image may include photos or video frames of various types of scenes, etc. For example, for an urban scene, the target scene image may include images showing urban buildings, streets, traffic, road signs, etc., and for an indoor scene, the target scene image may include images showing indoor environments such as homes, offices, and stores.

[0096] In step S202, text detection is performed using the trained target model to obtain position information of one or more text regions output by the target model.

[0097] The position information includes various information that can be used to indicate the position of the detected text area, such as the coordinates of the vertices and boundary points of the text area.

[0098] The text area may have any shape.

[0099] The trained target model is obtained by training and fine-tuning the above steps 101 to S103 based on the data set corresponding to the task.

[0100] Specifically, by using the trained target model to perform boundary detection, the vertices or boundary points of the text area are predicted. Then, the detected boundary points are converted into position information of one or more text areas. For example, the detected boundary points are converted into adaptive text area representations, such as polygons or curves.

[0101] Optionally, the method applies an attention mechanism or other methods to suppress background noise around the text region.

[0102] The text detection method of the embodiment of the present application can effectively detect text that may have irregular shapes in an image, thereby improving the accuracy and robustness of scene text detection.

[0103] Refer to the following Figure 3 The exemplary model structure shown is used to illustrate the method of the embodiment of the present application. The model of this example is a model based on a Transformer structure, which is used to detect text that may have irregular shapes in an input image.

[0104] Reference Figure 3The model in this example mainly includes three parts: backbone network, encoder and decoder. The encoder contains multiple encoder layers, and the decoder contains multiple decoder layers. The following introduces these three parts respectively.

[0105] 1) Backbone network: The backbone network in this example is the ViTAE-v2 backbone network, which is used to process the input image and extract the global features of the image. These features contain the basic visual information of the image and provide a basis for subsequent text detection and recognition;

[0106] 2) Encoder: The encoder further extracts the global features of the input image.

[0107] 3) Decoder: The decoder extracts local features of the input image. The local features output by the decoder layer are used to locate the text area in the input image and identify the text content.

[0108] The above-mentioned Bezier curve-based denoising training method is applied at the decoder layer to train the model to learn how to effectively handle noise and interference in the image, thereby further optimizing the performance of the model. In addition, the denoising training method helps the model better understand and handle the shape changes of text, such as curved or tilted text, thereby providing more reliable detection or recognition results in practical applications.

[0109] This example uses a hybrid training method to train Figure 2 The model shown. The process of this hybrid training method includes:

[0110] 1) Freeze the parameters other than the vocabulary and the target task to be executed, and train the vocabulary mapping layer and the structure related to the target task;

[0111] In this example, the structure related to the target task is the detection head. Through training, the Embedding layer can learn the context information of each word, so that the model can better understand the semantics when processing text.

[0112] 2) Obtaining the target data set of the target task to be performed;

[0113] 3) Unfreeze all learnable parameters and fine-tune the model based on the target dataset.

[0114] After unfreezing all learnable parameters, the model will readjust all parameters, including the weights of the embedding layer and the detection head. This process allows the model to better adapt to the specific target dataset. Fine-tuning usually requires thousands of steps of training to ensure that the model performs well on the new dataset. Through fine-tuning, the model can not only use the knowledge learned during pre-training, but also further optimize according to the characteristics of the target dataset, thereby improving the classification accuracy.

[0115] When performing the text detection task, the input image is input into the trained model of this example. The detection head of the model is used to predict the location of the text contained in the input image based on the local features output by the decoder, and the location information of the corresponding one or more text areas is used as the text detection result.

[0116] Figure 4 The schematic diagram of the structure of the model training device provided in the embodiment of the present application is shown. The device includes: a device for acquiring a training image set corresponding to a target task (hereinafter referred to as "training image acquisition device 101"), a device for acquiring a training image set corresponding to a target task (hereinafter referred to as "noise reduction training device 102"), and a device for fine-tuning a target model based on the training image set after unfreezing frozen parameters (hereinafter referred to as "model fine-tuning device 103").

[0117] Reference Figure 4 , the training image acquisition device 101 acquires a training image set corresponding to the target task.

[0118] The target task is to detect text contained in an image.

[0119] Optionally, the image includes text with irregular shapes. The training image set includes annotation data for boundaries of irregularly shaped text regions.

[0120] According to one embodiment, the device further comprises a pre-training device.

[0121] The pre-training device pre-trains the target model so that the target model can learn a universal feature representation.

[0122] Specifically, the pre-training device pre-trains the target model on a large public dataset, allowing the target model to learn universal feature representations that can capture the basic patterns in images and general properties of objects.

[0123] During the pre-training phase, the model will load a pre-trained word list (also called a vocabulary or embedding layer). This embedding layer contains the pre-trained weights of the vocabulary, which are learned from a large amount of data and can help the model better understand and process text data. Another important function of pre-trained weights is to reduce the time and resource consumption of model training. Since the model has learned some common features, in subsequent tasks, only these features need to be fine-tuned without training from scratch.

[0124] The denoising training device 102 uses a denoising training method based on Bezier curves to train the target model.

[0125] The noise reduction training device 102 performs training on the part corresponding to the target parameter by freezing the parameters other than the target parameter in the target model.

[0126] According to one embodiment, the target parameters include parameters related to the vocabulary and parameters related to the target task.

[0127] The parameters related to the vocabulary include the embedding layer of the model or other parameters directly related to the vocabulary. Optionally, the parameters related to the vocabulary include the parameters of the word embedding layer of the expanded new vocabulary. After the vocabulary is expanded, this method can obtain excellent detection effect for special characters with a small sample size.

[0128] The parameters related to the target task include parameters related to the detection structure of the model. The detection structure is used to detect and locate text contained in the input image.

[0129] By freezing all parameters other than the target parameters in the target model, we can focus on the vocabulary part of the training model and the part related to the target task to be performed. Based on this training method, the model training efficiency is improved because it does not need to learn all parameters at the same time.

[0130] According to one embodiment, the device may determine which parameters need to be frozen by acquiring pre-stored freezing parameter indication information. Alternatively, the device may determine the parameters that need to be frozen in real time based on user input.

[0131] According to one embodiment, the noise reduction training device 102 further includes a query decomposition device, a control point generation device, a coordinate point sampling device, a position query acquisition device and a noise initialization device.

[0132] The Bezier curve-based denoising training method is targeted at text recognition tasks based on the Transformer architecture. This method improves traditional denoising training to make the model more stable and efficient when processing text of any shape.

[0133] The query decomposition device decomposes the query of the noise reduction part into a noise position query and a noise content query.

[0134] The control point generating device generates a noise position query using four control points of the Bezier curve, and obtains a set of new Bezier control points after adding noise to the Bezier control points.

[0135] The coordinate point sampling device uses Bezier control points with added noise to uniformly sample multiple point coordinates from the generated Bezier curve.

[0136] The location query acquisition device obtains the corresponding noise location query by performing location encoding and two-layer multilayer perceptron (MLP) processing on the obtained coordinates.

[0137] The noise initialization device initializes the noise content query through the mask character sliding method.

[0138] The noise device is initialized by using the Mask Character Sliding (MCS) method to assist in the alignment of text content and position.

[0139] According to one embodiment, at the stage of executing a task using a trained model, when performing model inference, the device removes the portion of applying a Bezier curve-based denoising training method.

[0140] Among them, the part that applies the Bezier curve-based denoising training method can be removed when performing model inference by setting the denoising training part to an inactive state in the configuration file or code of the model or directly disabling the layer that applies denoising training in the model.

[0141] By removing the part of the model that applies the Bezier curve-based denoising training method during the model inference phase, the efficiency of the inference process is ensured and additional computational burden is avoided.

[0142] After unfreezing the frozen parameters, the model fine-tuning device 103 fine-tunes the target model based on the target training image set.

[0143] Among them, fine-tuning is performed to retain the useful features that the model has learned in the pre-training phase, while adapting it to new tasks through a small amount of adjustment. The fine-tuning process can significantly improve the performance of the model on specific tasks while reducing training costs and time.

[0144] Optionally, during the fine-tuning process, the model fine-tuning device 103 performs structural adjustment processing on the pre-trained target model according to the requirements of the target task to be performed. For example, in an image recognition task, the last classification layer may need to be replaced to match the number of categories in the target data set.

[0145] According to the device of the present application, a denoising method based on Bezier curves is used to train the model, and then the trained model is used to detect text that may have irregular shapes in the input image, thereby improving the accuracy of text detection; in the training phase, parameters other than the vocabulary and parameters related to the target task to be executed are frozen, so that only the vocabulary mapping layer and the task-related detection head are trained, and in the fine-tuning phase, all learnable parameters are unfrozen to fine-tune the model based on the data set corresponding to the target task. This training method reduces the cost of training and fine-tuning on the target data set, and can achieve the detection of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training.

[0146] Figure 5 A schematic structural diagram of a device for detecting text according to an embodiment of the present application is shown.

[0147] The device includes: a device for acquiring a target scene image to be processed (hereinafter referred to as "input image acquisition device 201") and a device for performing text detection using a trained target model to obtain position information of one or more text areas output by the target model (hereinafter referred to as "text detection device 202").

[0148] Reference Figure 5 , the input image acquisition device 201 acquires the target scene image to be processed.

[0149] The target scene image may include photos or video frames of various types of scenes, etc. For example, for an urban scene, the target scene image may include images showing urban buildings, streets, traffic, road signs, etc., and for an indoor scene, the target scene image may include images showing indoor environments such as homes, offices, and stores.

[0150] The text detection device 202 uses the trained target model to perform text detection and obtains the position information of one or more text regions output by the target model.

[0151] The position information includes various information that can be used to indicate the position of the detected text area, such as the coordinates of the vertices and boundary points of the text area.

[0152] The text area may have any shape.

[0153] The trained target model is obtained by training and fine-tuning the training image acquisition device 101, the noise reduction training device 102 and the model fine-tuning device 103 based on the data set corresponding to the task.

[0154] Specifically, the model fine-tuning device 103 predicts the vertices or boundary points of the text area by performing boundary detection using the trained target model. Then, the detected boundary points are converted into position information of one or more text areas. For example, the detected boundary points are converted into adaptive text area representations, such as polygons or curves.

[0155] Optionally, the device applies an attention mechanism or other methods to suppress background noise around the text region.

[0156] The device for text detection in the embodiment of the present application can effectively detect text that may have irregular shapes in an image, thereby improving the accuracy and robustness of scene text detection.

[0157] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the model training method or the method for text detection in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0158] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0159] Figure 6The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0160] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0161] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0162] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0163] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0164] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0165] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0166] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0167] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0168] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0169] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0170] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0171] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0172] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0174] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A model training method, comprising: Obtain the training image set corresponding to the target task; Using a Bezier curve-based denoising training method to train a target model, wherein the method freezes parameters other than the target parameters in the target model to train a portion corresponding to the target parameters; After unfreezing the frozen parameters, fine-tune the target model based on the training image set.

2. The method according to claim 1, wherein: The method of using the Bezier curve-based denoising training method to train the target model includes: Decompose the query of the noise reduction part into noise location query and noise content query; Generate a noise position query using the four control points of the Bezier curve, and obtain a new set of Bezier control points after adding noise to the Bezier control points; Using the Bezier control points with added noise, uniformly sample multiple point coordinates from the generated Bezier curve; By performing position encoding and two-layer MLP processing on the obtained coordinates, the corresponding noise position query is obtained; Noisy content queries are initialized through a masked character sliding method to assist in the alignment of text content and location.

3. The method according to claim 1 or 2, wherein: The target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder and a decoder. The method trains the model by the following steps: Perform global feature extraction on the input image in the backbone network to obtain corresponding global features and local features; Further extracting global features of the input image in the encoder; In the decoder, based on the global features, further performing local feature extraction processing on the input image to obtain corresponding local features; A Bezier curve-based denoising training method is applied in the decoder to train the model.

4. The method according to claim 3, wherein: The Transformer architecture-based model also includes a detection head, and the method further includes: The detection head predicts the location of text contained in the input image based on the local features output by the decoder layer.

5. The method according to claim 1, wherein: The method further comprises: The target model is pre-trained so that the target model can learn a universal feature representation.

6. A text detection method, wherein: The method comprises: Acquire the target scene image to be processed; Use the trained target model to perform text detection and obtain the location information of one or more text regions output by the target model; Wherein, the target model is obtained by training using the model training method described in any one of claims 1 to 5.

7. The method according to claim 2, wherein: The method further comprises: When performing model inference, remove the part that applies the Bezier curve-based denoising training method.

8. A model training device, wherein: The device comprises: Means for obtaining a training image set corresponding to a target task; A device for training a target model using a Bezier curve-based denoising training method, wherein the device performs training on a portion corresponding to the target parameter by freezing parameters other than the target parameter in the target model; A device for fine-tuning a target model based on a training image set after unfreezing the frozen parameters.

9. A device for text detection, wherein: The device comprises: A device for acquiring a target scene image to be processed; A device for performing text detection using a trained target model to obtain position information of one or more text regions output by the target model; Wherein, the target model is obtained by training using the model training method described in any one of claims 1 to 5.

10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method of any one of claims 1 to 5, or the method of claim 6 or 7.

11. A computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 5, or to execute the method according to claim 6 or 7.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented, or the method according to claim 6 or 7 is executed.

Images