Model training method and device, medium and program product

Abstract

The invention provides a model training method and device, electronic equipment, a computer readable medium and a computer program product. The method comprises the steps that after pre-training of a target model is completed, parameters except target parameters in the target model are frozen, and corresponding parts of the target parameters are trained; obtaining a target training image set corresponding to the to-be-executed detection task; and after unfreezing the frozen parameters, performing fine adjustment on the target model based on the target training image set. According to the method, related parameters are frozen in the model training stage, so that only parameters related to the word list mapping layer and the scene text task to be executed are trained, and all learnable parameters are unfrozen in the fine tuning stage to perform fine tuning on the model based on the data set corresponding to the scene text task; the training mode reduces the cost of training fine tuning on the target data set.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, device, electronic device, computer-readable medium and computer program product. Background Art

[0002] Scene text-related tasks are an important research direction in the field of computer vision, which involves various types of tasks related to detecting and recognizing text information from natural scenes. These tasks are of great significance to image search, instant translation, robot navigation, industrial automation and other fields. Existing research mainly focuses on text detection and recognition.

[0003] Traditional scene text detection models, such as DBNet, are committed to improving the performance of text detection through a multi-step approach based on special representations of text components. On the other hand, text recognition models, such as ParSeq, try to improve the recognition performance of text by using deep learning techniques, especially CNN and RNN-based models.

[0004] However, solutions based on existing technologies still face challenges in handling text detection or text recognition in complex scenes. Summary of the invention

[0005] Multiple aspects of the present application provide a model training method, apparatus, electronic device, computer-readable medium, and computer program product.

[0006] In one aspect of the present application, a model training method is provided, wherein the method comprises:

[0007] After completing the pre-training of the target model, the parameters other than the target parameters in the target model are frozen to train the corresponding parts of the target parameters;

[0008] Obtain a target training image set corresponding to the detection task to be performed;

[0009] After unfreezing the frozen parameters, the target model is fine-tuned based on the target training image set.

[0010] In one aspect of the present application, a model training device is provided, wherein the device comprises:

[0011] After completing the pre-training of the target model, a device is used to train the corresponding part of the target parameter by freezing the parameters other than the target parameters in the target model;

[0012] Means for acquiring a target training image set corresponding to a detection task to be performed;

[0013] A device for fine-tuning a target model based on a target training image set after unfreezing the frozen parameters.

[0014] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0015] Another aspect of the present application provides a computer program product, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0016] In the solution provided in the embodiment of the present application, by freezing the relevant parameters in the model training stage, only the parameters related to the vocabulary mapping layer and the scene text task to be executed are trained, and all learnable parameters are unfrozen in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set, and can achieve detection and recognition of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the model training efficiency; by constructing and training a model based on the Transformer architecture including a backbone network, an encoder layer and a decoder layer, multiple tasks such as detection, recognition and linking are integrated into a unified framework, so that it can adapt to different target data sets and vocabularies, and has good flexibility and adaptability; the embodiment of the present application uses a denoising training method based on Bezier curves to optimize the performance of the model, thereby improving the accuracy and robustness of the model on the target data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0018] Other features, objects and advantages of the present application will become more apparent by reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown;

[0020] Figure 2 A schematic diagram of an exemplary model structure according to an embodiment of the present application is shown;

[0021] Figure 3 A schematic diagram of the structure of a model training device provided in an embodiment of the present application is shown;

[0022] Figure 4 A schematic diagram of the structure of a device suitable for implementing the solution in the embodiment of the present application is shown.

[0023] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0024] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0025] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPU), input / output interface, network interface and memory.

[0026] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0027] Computer readable media include permanent and non-permanent, removable and non-removable media, and can be implemented by any method or technology to store information. Information can be computer program instructions, data structures, modules of programs or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, read-only compact disk (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0028] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown. The method at least includes step S101, step S102 and step S103.

[0029] In actual scenarios, the execution subject of the method can be a network device, or an application running on a network device, wherein the network device includes but is not limited to a network host, a single network server, a plurality of network server sets, or a collection of computers based on cloud computing, and can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, wherein cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0030] The concepts involved in the embodiments of the present application are explained below.

[0031] Transformer architecture: A model based on the Transformer architecture is a deep learning model. The core features of the Transformer model include:

[0032] Self-Attention: The self-attention mechanism allows the model to process a sequence in such a way that each element can interact directly with other elements in the sequence without relying on the position in the sequence. This mechanism enables the model to capture the relationship between any two elements in the sequence, no matter how far apart they are in the sequence. Parallel processing: Models based on the Transformer architecture can process all elements in the sequence in parallel, which greatly improves the training efficiency of the model.

[0033] Encoder-Decoder Architecture: Models based on the Transformer architecture use an encoder-decoder architecture for sequence-to-sequence tasks such as machine translation. The encoder processes the input sequence and the decoder generates the output sequence. Each encoder and decoder consists of multiple identical layers, each of which contains a self-attention mechanism and a feed-forward neural network.

[0034] Position encoding: Since the Transformer model itself does not have the ability to capture position information in the sequence, position encoding needs to be introduced to provide the position information of each element in the sequence. Position encoding is usually a vector of the same length as the input sequence, which is added to the input sequence through learning or predefined methods to help the model understand the order of elements in the sequence.

[0035] Multi-head attention: The self-attention mechanism in the Transformer-based model can be extended to multi-head attention, that is, capturing information from different representation subspaces at the same time. This increases the expressive power of the model, allowing it to capture different information in different representation subspaces.

[0036] Layer Normalization and Residual Connections: The Transformer model applies layer normalization on the output of each sublayer (self-attention layer and feed-forward neural network layer) and uses residual connections. These techniques help avoid the vanishing or exploding gradient problem in deep network training.

[0037] Bezier curve: The Bezier curve method is a technology used for natural scene text detection and recognition (STR), which uses Bezier curves to represent and fit the shape of the text area instead of traditional rectangles or polygons.

[0038] In some embodiments, the model of the embodiment of the present application is a model based on the Transformer architecture, which is used to perform various scene text tasks, such as text detection, text link detection, and text recognition.

[0039] In some embodiments, when training the model of the present application, parameters other than the vocabulary and the parameters related to the scene text task to be performed are frozen, so that only the vocabulary mapping layer and the task-related text classification head are trained. In the fine-tuning stage, all learnable parameters are unfrozen, so that the model is fine-tuned based on the data set corresponding to the scene text task.

[0040] exist Figure 1 Before step S101 shown, the method further includes step S104.

[0041] In step S104, the target model is pre-trained so that the target model can learn a universal feature representation.

[0042] Specifically, pre-training on a large public dataset allows the target model to learn universal feature representations that can capture the basic patterns in images and general properties of objects.

[0043] During the pre-training phase, the model loads a pre-trained word list (also called a vocabulary or embedding layer). This embedding layer contains the pre-trained weights of the vocabulary, which are learned from a large amount of data and can help the model better understand and process text data. Another important role of pre-trained weights is to reduce the time and resource consumption of model training. Since the model has learned some common features, when it needs to perform scene text tasks, it only needs to fine-tune these features without training from scratch.

[0044] Refer to the following Figure 1 To explain, in step S101, after completing the pre-training of the target model, the parameters other than the target parameters in the target model are frozen to perform training on the parts corresponding to the target parameters.

[0045] According to one embodiment, the target parameters include parameters related to the vocabulary and parameters related to the scene text task to be performed.

[0046] The parameters related to the vocabulary include the embedding layer of the model or other parameters directly related to the vocabulary. Optionally, the parameters related to the vocabulary include the parameters of the word embedding layer of the expanded new vocabulary. After the vocabulary is expanded, this method can obtain excellent detection effect for special characters with a small sample size.

[0047] By freezing all parameters other than the target parameters in the target model, we can focus on the vocabulary part of the training model and the part related to the scene text task to be performed. Based on this training method, the model training efficiency is improved because it does not need to learn all parameters at the same time.

[0048] According to one embodiment, the target parameter may further include other preset parameters, and the step S101 further includes the following steps:

[0049] Load the model weights of the target model pre-trained on a predetermined large image dataset.

[0050] Next, determine the parameters that need to be frozen.

[0051] The method may determine which parameters need to be frozen by acquiring pre-stored freezing parameter indication information. Alternatively, the method may determine the parameters need to be frozen in real time based on user input.

[0052] For example, by setting the requires_grad attribute of these parameters to False. Based on this setting, the gradients of these parameters will not be calculated during backpropagation, so their values ​​will not change, thus preventing these parameters from being updated during training.

[0053] Then, after freezing the determined parameters, continue training the rest of the model.

[0054] According to one embodiment, the method uses a Bezier curve-based noise reduction training method in S101 to perform training on a portion corresponding to the target parameter. The training process includes steps S1011 to S1015.

[0055] The Bezier curve-based denoising training method is targeted at text recognition tasks based on the Transformer architecture. This method improves traditional denoising training to make the model more stable and efficient when processing text of any shape.

[0056] In step S1011, the query of the noise reduction part is decomposed into a noise position query and a noise content query.

[0057] In step S1012, a noise position query is generated using four control points of the Bezier curve. After adding noise to the Bezier control points, a new set of control points is obtained.

[0058] In step S1013, the Bezier control points with added noise are used to uniformly sample the coordinates of multiple points from the generated Bezier curve.

[0059] In step S1014, the corresponding noise position query is obtained by performing position encoding and two-layer multilayer perceptron (MLP) processing on the obtained coordinates.

[0060] In step S1015, the noise content query is initialized by a mask character sliding method to assist in the alignment of text content and position.

[0061] Specifically, we first determine the number of valid characters t, which refers to the number of characters in the input sequence that actually make sense. Then, we perform an integer division operation To calculate the number of times each valid character should be cloned so that the total length T of the sequence is evenly distributed to each valid character. In addition, since T may not be completely divisible by t, there will be a remainder Indicates that there are k additional spaces to be allocated. In order to distribute these additional spaces fairly, an additional clone will be allocated to each of the first k valid characters in the sequence to ensure that each character is allocated as evenly as possible.

[0062] After getting the sliding characters, use the mask operation to control the number of consecutive characters, that is, flip a part of the consecutive characters into background labels with a certain probability. After processing the positive part, add noise to the characters in the positive part and the negative part, so that these characters flip into other characters with a probability of λ. After embedding, these characters get the noise content query, in which all the characters initialized in the negative part are background.

[0063] According to one embodiment, when performing model inference, the portion of applying the Bezier curve-based denoising training method is removed.

[0064] Among them, the part that applies the Bezier curve-based denoising training method can be removed when performing model inference by setting the denoising training part to an inactive state in the configuration file or code of the model or directly disabling the layer that applies denoising training in the model.

[0065] By removing the part of the model that applies the Bezier curve-based denoising training method during the model inference phase, the efficiency of the inference process is ensured and additional computational burden is avoided.

[0066] Continue to refer to Figure 1 To illustrate, in step S102, a target training image set corresponding to the scene text task to be performed is obtained.

[0067] The scene text task may include various tasks for detecting, identifying and understanding text information in scene images, such as text detection, text link detection and text recognition tasks.

[0068] Optionally, the method cleans, annotates and preprocesses the acquired data set for use in training.

[0069] In step S103, after the frozen parameters are unfrozen, the target model is fine-tuned based on the target training image set.

[0070] Among them, fine-tuning is performed to retain the useful features that the model has learned in the pre-training phase and adapt it to new tasks through a small amount of adjustment. The fine-tuning process can significantly improve the performance of the model on specific tasks, thereby reducing training costs and time.

[0071] Optionally, during the fine-tuning process, the pre-trained target model is structurally adjusted according to the requirements of the scene text task to be performed. For example, in a text recognition task, the last classification layer may need to be replaced to match the number of categories in the target dataset.

[0072] According to one embodiment, the model is a model based on a Transformer architecture, the structure of which includes a backbone network, an encoder and a decoder. This embodiment trains the model by executing the following steps S1031 to S1033:

[0073] In step S1031, global feature extraction processing is performed on the input image in the backbone network to obtain corresponding global features and local features.

[0074] Optionally, the input image is preprocessed before feature extraction, which may include operations such as resizing, normalizing, and enhancing contrast to ensure that the model receives input in a consistent format.

[0075] Among them, the global features obtained through the backbone network help the model understand the global context of the image.

[0076] In step S1032, global features of the input image are further extracted in the encoder.

[0077] Among them, the global features obtained through the backbone network help the model understand the global context of the image.

[0078] In step S1033, the decoder further performs local feature extraction processing on the input image based on the global feature to obtain corresponding local features.

[0079] To get a more detailed understanding of the image content, the model needs to further extract local features. This is achieved by using deeper network structures that are able to identify details such as edges, corners, and textures in the image. These local features help the model recognize the outline and shape of text, and can detect text even if the text has different orientations and sizes in the image.

[0080] In step S1034, a Bezier curve-based noise reduction training method is applied in the decoder to train the model to learn how to effectively process noise and interference in the image.

[0081] The local features output by the decoder layer are used to locate the text area in the image and identify the text content.

[0082] According to one embodiment, the method further includes steps 106 to 109 .

[0083] In step S106, the image to be processed corresponding to the scene text task to be executed is obtained.

[0084] The images to be processed may include photos or video frames of various types of scenes, etc. For example, for urban scenes, the images to be processed may include images showing urban buildings, streets, traffic, road signs, etc., and for indoor scenes, the images to be processed may include images showing indoor environments such as homes, offices, and stores.

[0085] In step S107, the image to be processed is input into the trained target model.

[0086] The trained target model is trained through the above step 101 and is obtained by fine-tuning based on the data set corresponding to the task through the above step S103.

[0087] In step S108, feature extraction is performed based on the target model to obtain image features corresponding to the image to be processed, wherein the image features include global features and local features.

[0088] For the model based on the Transformer architecture, feature extraction is performed based on the encoder and decoder to obtain the image features corresponding to the image to be processed.

[0089] In step S109, the result of the scene text task output by the target model based on the image features is obtained.

[0090] The method provided in the embodiment of the present application freezes the relevant parameters in the training stage, so that only the parameters related to the vocabulary mapping layer and the scene text task to be executed are trained, and all learnable parameters are unfrozen in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set, and can complete the detection and recognition of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training; constructing and training a model based on the Transformer architecture including a backbone network, an encoder layer and a decoder layer, integrating multiple tasks such as detection, recognition and linking into a unified framework, which can adapt to different target data sets and vocabularies and has good flexibility and adaptability; using a denoising training method to optimize the performance of the unified model, improving the accuracy and robustness of the model on the target data set.

[0091] Refer to the following Figure 2 The exemplary model structure shown is used to illustrate the method of the embodiment of the present application. The model of this example is a model based on the Transformer architecture, which can be used to perform a variety of scene text tasks, such as text detection, text link detection and text recognition tasks.

[0092] Reference Figure 2 The model in this example mainly includes three parts: backbone network, encoder and decoder. The encoder contains multiple encoder layers, and the decoder contains multiple decoder layers. The following introduces these three parts respectively.

[0093] 1) Backbone network: The backbone network in this example is the ViTAE-v2 backbone network, which is used to process the input image and extract the global features of the image. These features contain the basic visual information of the image and provide a basis for subsequent text detection and recognition;

[0094] 2) Encoder: The encoder further extracts the global features of the input image.

[0095] 3) Decoder: The decoder extracts local features of the input image. The local features are used to capture detailed information such as the shape, size, and direction of the text.

[0096] The Bezier curve-based denoising training method described in step S105 is applied at the decoder layer to train the model to learn how to effectively process noise and interference in the image, thereby further optimizing the performance of the model. In addition, the Bezier curve-based denoising training method helps the model better understand and process the shape changes of the text, thereby providing more reliable detection or recognition results in practical applications.

[0097] This example uses a hybrid training method to train the model. The process of this hybrid training method includes:

[0098] 1) Freeze the parameters other than the vocabulary and the parameters related to the scene text task to be performed, and train the vocabulary mapping layer and the structure related to the scene text task;

[0099] Through training, the Embedding layer can learn the contextual information of each word, so that the model can better understand the semantics when processing text.

[0100] Optionally, the structure related to the scene text task may include a classification head. The classification head is usually a linear layer, which converts the high-dimensional vector output by the Embedding layer into a specific category label. By performing a linear transformation on the Embedding vector, the model can output the probability distribution of each category, thereby achieving the classification task.

[0101] 2) Obtain the target data set of the task to be performed;

[0102] 3) Unfreeze all learnable parameters and fine-tune the model based on the target dataset.

[0103] After unfreezing all learnable parameters, the model will readjust all parameters, including the weights of the embedding layer and the classification head. This process allows the model to better adapt to the specific target dataset. Fine-tuning usually requires thousands of steps of training to ensure that the model performs well on the new dataset. Through fine-tuning, the model can not only use the knowledge learned during pre-training, but also further optimize according to the characteristics of the target dataset, thereby improving the classification accuracy.

[0104] Through the classification head structure, the final binary classification prediction result can be output. For example, assuming that the task to be performed is a text detection task, the prediction result output by the model is the text detection result.

[0105] The method provided in the embodiment of the present application freezes the relevant parameters in the model training stage, so that only the parameters related to the vocabulary mapping layer and the scene text task to be executed are trained, and all learnable parameters are unfrozen in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set, and can complete the detection and recognition of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the model training efficiency; by constructing and training a model based on the Transformer architecture including a backbone network, an encoder layer and a decoder layer, multiple tasks such as detection, recognition and linking are integrated into a unified framework, so that it can adapt to different target data sets and vocabularies, and has good flexibility and adaptability; the embodiment of the present application uses a denoising training method based on Bezier curves to optimize the performance of the model, thereby improving the accuracy and robustness of the model on the target data set.

[0106] In addition, the present application also provides a model training device, the structure of which is as follows: Figure 3 The device includes: a device for training the corresponding part of the target parameters by freezing the parameters other than the target parameters in the target model after completing the pre-training of the target model (hereinafter referred to as "parameter training device 101"), a device for acquiring a target training image set corresponding to the detection task to be performed (hereinafter referred to as "data acquisition device 102"), and a device for fine-tuning the target model based on the target training image set after unfreezing the frozen parameters (hereinafter referred to as "model fine-tuning device 103").

[0107] Wherein, the device also includes a pre-training device.

[0108] The pre-training device pre-trains the target model so that the target model can learn a universal feature representation.

[0109] Specifically, the pre-training device is pre-trained on a large public dataset to allow the target model to learn universal feature representations that can capture the basic patterns in images and general properties of objects.

[0110] During the pre-training phase, the model loads a pre-trained word list (also called a vocabulary or embedding layer). This embedding layer contains the pre-trained weights of the vocabulary, which are learned from a large amount of data and can help the model better understand and process text data. Another important role of pre-trained weights is to reduce the time and resource consumption of model training. Since the model has learned some common features, when it needs to perform scene text tasks, it only needs to fine-tune these features without training from scratch.

[0111] Refer to the following Figure 3 To explain, after completing the pre-training of the target model, the parameter training device 101 freezes the parameters other than the target parameters in the target model to train the corresponding parts of the target parameters.

[0112] According to one embodiment, the target parameters include parameters related to the vocabulary and parameters related to the scene text task to be performed.

[0113] The parameters related to the vocabulary include the embedding layer of the model or other parameters directly related to the vocabulary. Optionally, the parameters related to the vocabulary include the parameters of the word embedding layer of the expanded new vocabulary. After the vocabulary is expanded, this method can obtain excellent detection effect for special characters with a small sample size.

[0114] By freezing all parameters other than the target parameters in the target model, we can focus on the vocabulary part of the training model and the part related to the scene text task to be performed. Based on this training method, the model training efficiency is improved because it does not need to learn all parameters at the same time.

[0115] According to one embodiment, the target parameters may also include other preset parameters, and the parameter training device 101 includes a weight loading device, a parameter determining device and a freezing training device.

[0116] The weight loading device loads the model weights of the target model pre-trained on a predetermined large-scale image dataset.

[0117] Next, the parameter determination device determines the parameters that need to be frozen.

[0118] The parameter determination device may determine which parameters need to be frozen by acquiring pre-stored freezing parameter indication information. Alternatively, the parameter determination device may determine the parameters that need to be frozen in real time based on user input.

[0119] For example, by setting the requires_grad attribute of these parameters to False. Based on this setting, the gradients of these parameters will not be calculated during backpropagation, so their values ​​will not change, thus preventing these parameters from being updated during training.

[0120] Next, the freezing training device freezes the determined parameters and continues to train other parts of the model.

[0121] According to one embodiment, the parameter training device 101 uses a Bezier curve-based noise reduction training method to train the corresponding part of the target parameter, and the parameter training device 101 further includes a query decomposition device, a control point generation device, a coordinate point sampling device, a position query acquisition device and an initialization noise device.

[0122] The Bezier curve-based denoising training method is targeted at text recognition tasks based on the Transformer architecture. This method improves traditional denoising training to make the model more stable and efficient when processing text of any shape.

[0123] The query decomposition device decomposes the query of the noise reduction part into a noise position query and a noise content query.

[0124] The control point generating device generates a noise position query using four control points of the Bezier curve, and obtains a set of new Bezier control points after adding noise to the Bezier control points.

[0125] The coordinate point sampling device uses Bezier control points with added noise to uniformly sample multiple point coordinates from the generated Bezier curve.

[0126] The location query acquisition device obtains the corresponding noise location query by performing location encoding and two-layer multilayer perceptron (MLP) processing on the obtained coordinates.

[0127] The noise initialization device initializes the noise content query through the mask character sliding method.

[0128] Specifically, we first determine the number of valid characters t, which refers to the number of characters in the input sequence that actually make sense. Then, we perform an integer division operation To calculate the number of times each valid character should be cloned so that the total length T of the sequence is evenly distributed to each valid character. In addition, since T may not be completely divisible by t, there will be a remainder Indicates that there are k additional spaces to be allocated. In order to distribute these additional spaces fairly, an additional clone will be allocated to each of the first k valid characters in the sequence to ensure that each character is allocated as evenly as possible.

[0129] After getting the sliding characters, use the mask operation to control the number of consecutive characters, that is, flip a part of the consecutive characters into background labels with a certain probability. After processing the positive part, add noise to the characters in the positive part and the negative part, so that these characters flip into other characters with a probability of λ. After embedding, these characters get the noise content query, in which all the characters initialized in the negative part are background.

[0130] According to one embodiment, when performing model inference, the device removes the portion of applying the Bezier curve-based denoising training method.

[0131] Among them, the part that applies the Bezier curve-based denoising training method can be removed when performing model inference by setting the denoising training part to an inactive state in the configuration file or code of the model or directly disabling the layer that applies denoising training in the model.

[0132] By removing the part of the model that applies the Bezier curve-based denoising training method during the model inference phase, the efficiency of the inference process is ensured and additional computational burden is avoided.

[0133] Continue to refer to Figure 3 To illustrate, the data acquisition device 102 acquires a target training image set corresponding to a scene text task to be performed.

[0134] The scene text task may include various tasks for detecting, identifying and understanding text information in scene images, such as text detection, text link detection and text recognition tasks.

[0135] Optionally, the data acquisition device 102 cleans, annotates and preprocesses the acquired data set for use in training.

[0136] The model fine-tuning device 103 thaws the frozen parameters and then fine-tunes the target model based on the target training image set.

[0137] The model fine-tuning device 103 retains the useful features that the model has learned in the pre-training stage by fine-tuning, and adapts it to new tasks through a small amount of adjustment. The fine-tuning process can significantly improve the performance of the model on a specific task, thereby reducing training costs and time.

[0138] Optionally, during the fine-tuning process, the model fine-tuning device 103 performs structural adjustment processing on the pre-trained target model according to the requirements of the scene text task to be performed. For example, in a text recognition task, the last classification layer may need to be replaced to match the number of categories in the target data set.

[0139] According to one embodiment, the model is a model based on a Transformer architecture, the structure of which includes a backbone network, an encoder and a decoder. The device of this embodiment trains the model by performing the following operations:

[0140] In the backbone network, the input image is processed for global feature extraction to obtain the corresponding global features and local features.

[0141] Optionally, the input image is preprocessed before feature extraction, which may include operations such as resizing, normalizing, and enhancing contrast to ensure that the model receives input in a consistent format. Among them, the global features obtained through the backbone network help the model understand the global context of the image.

[0142] Next, the encoder further extracts global features of the input image. The global features obtained through the backbone network help the model understand the global context of the image.

[0143] Next, in the decoder, based on the global features, local feature extraction processing is further performed on the input image to obtain corresponding local features.

[0144] To get a more detailed understanding of the image content, the model needs to further extract local features. This is achieved by using deeper network structures that are able to identify details such as edges, corners, and textures in the image. These local features help the model recognize the outline and shape of text, and can detect text even if the text has different orientations and sizes in the image.

[0145] Next, a Bezier curve-based denoising training method is applied in the decoder to train the model to learn how to effectively deal with noise and interference in the image.

[0146] The local features output by the decoder layer are used to locate the text area in the image and identify the text content.

[0147] According to one embodiment, the device further includes an image acquisition device, an image input device, a feature extraction device and a result output device.

[0148] The image acquisition device acquires the to-be-processed image corresponding to the scene text task to be executed.

[0149] The images to be processed may include photos or video frames of various types of scenes, etc. For example, for urban scenes, the images to be processed may include images showing urban buildings, streets, traffic, road signs, etc., and for indoor scenes, the images to be processed may include images showing indoor environments such as homes, offices, and stores.

[0150] The image input device inputs the image to be processed into the trained target model.

[0151] The trained target model is trained by the parameter training device 101 and is fine-tuned based on the data set corresponding to the task through the operation of the model fine-tuning device 103.

[0152] The feature extraction device extracts features based on the target model to obtain image features corresponding to the image to be processed, wherein the image features include global features and local features.

[0153] For the model based on the Transformer architecture, feature extraction is performed based on the encoder and decoder to obtain the image features corresponding to the image to be processed.

[0154] The result output device obtains the result of the scene text task output by the target model based on the image features.

[0155] The device provided in the embodiment of the present application freezes the relevant parameters in the model training stage, so that only the parameters related to the vocabulary mapping layer and the scene text task to be executed are trained, and all learnable parameters are unfrozen in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set, and can complete the detection and recognition of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the model training efficiency; by constructing and training a model based on the Transformer architecture including a backbone network, an encoder layer and a decoder layer, multiple tasks such as detection, recognition and linking are integrated into a unified framework, so that it can adapt to different target data sets and vocabularies, and has good flexibility and adaptability; the embodiment of the present application uses a denoising training method based on Bezier curves to optimize the performance of the model, thereby improving the accuracy and robustness of the model on the target data set.

[0156] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application, and the method corresponding to the electronic device may be the model training method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0157] The electronic device may be a user device, or a device formed by integrating a user device and a network device through a network, or may be an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablet computers, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer sets, which can be used to implement some processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing, where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computer sets.

[0158] Figure 4The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown, and the device 1200 includes a central processing unit (CPU, Central Processing Unit) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM, Read Only Memory) 1202 or the program loaded from the storage part 1208 to the random access memory (RAM, Random Access Memory) 1203. In RAM1203, various programs and data required for system operation are also stored. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through bus 1204. Input / output (I / O, Input / Output) interface 1205 is also connected to bus 1204.

[0159] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, etc.; an output section 1207 including a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, etc., and a speaker, etc.; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, a semiconductor memory, etc.; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 1209 performs communication processing via a network such as the Internet.

[0160] In particular, the methods and / or embodiments in the embodiments of the present application may be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above functions defined in the method of the present application are executed.

[0161] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application described above.

[0162] Specifically, the present embodiment may adopt any combination of one or more computer-readable media. The computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device, or device.

[0163] Computer readable signal media may include a data signal propagated in baseband or as part of a carrier wave, which carries a computer readable program code. Such propagated data signals may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination of the above. Computer readable signal media may also be any computer readable medium other than a computer readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0164] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0165] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0166] The flow chart or block diagram in the accompanying drawings shows the possible architecture, function and operation of the equipment, method and computer program product according to various embodiments of the present application. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0168] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0169] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0170] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0171] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store program codes.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0173] In addition, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices stated in a device claim can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, and do not indicate any particular order.

Claims

1. A model training method, wherein: The method comprises: After completing the pre-training of the target model, the parameters other than the target parameters in the target model are frozen to train the corresponding parts of the target parameters; Obtain a target training image set corresponding to the detection task to be performed; After unfreezing the frozen parameters, the target model is fine-tuned based on the target training image set.

2. The method according to claim 1, wherein: The step of freezing parameters other than the target parameters in the target model to train the corresponding part of the target parameters includes: Load the model weights of the target model pre-trained on a predetermined large image dataset; Determine the parameters that need to be frozen; After freezing the determined parameters, continue training the rest of the model.

3. The method according to claim 1, wherein: The freezing of parameters other than the target parameters to train the corresponding part of the target parameters includes: A Bezier curve-based denoising training method is used to train the corresponding parts of the target parameters.

4. The method according to claim 3, wherein: The method of using the Bezier curve-based denoising training method to train the corresponding part of the target parameter includes: Decompose the query of the noise reduction part into noise location query and noise content query; Generate a noise position query using the four control points of the Bezier curve, and obtain a new set of Bezier control points after adding noise to the Bezier control points; Using the Bezier control points with added noise, uniformly sample multiple point coordinates from the generated Bezier curve; By performing position encoding and two-layer MLP processing on the obtained coordinates, the corresponding noise position query is obtained; Noisy content queries are initialized through a masked character sliding method to assist in the alignment of text content and location.

5. The method according to claim 3 or 4, wherein: The method further comprises: When performing model inference, remove the part that applies the Bezier curve-based denoising training method.

6. The method according to any one of claims 1 to 5, wherein: The target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder and a decoder. The method trains the model by the following steps: Perform global feature extraction on the input image in the backbone network to obtain corresponding global features and local features; Further extracting global features of the input image in the encoder; In the decoder, based on the global features, further performing local feature extraction processing on the input image to obtain corresponding local features; A Bezier curve-based denoising training method is applied in the decoder to train the model to learn how to effectively deal with noise and interference in images.

7. The method according to claim 1, wherein: The method further comprises: The target model is pre-trained so that the target model can learn a universal feature representation.

8. The method according to claim 1, wherein: The method further comprises: Obtaining an image to be processed corresponding to a scene text task to be performed; Input the image to be processed into the trained target model; Perform feature extraction based on the target model to obtain image features corresponding to the image to be processed, wherein the image features include global features and local features; Get the result of the scene text task based on the output of the target model based on the image features.

9. A model training device, wherein: The device comprises: After completing the pre-training of the target model, a device is used to train the corresponding part of the target parameter by freezing the parameters other than the target parameters in the target model; Means for acquiring a target training image set corresponding to a detection task to be performed; A device for fine-tuning a target model based on a target training image set after unfreezing the frozen parameters.

10. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

11. A computer readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 8.

12. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images