Model training method, device, medium and program product

Abstract

The present application provides a model training method, apparatus, electronic device, computer-readable medium and computer program product. The method of the present application includes: after completing the pre-training of the target model, training the corresponding part of the target parameters by freezing the parameters other than the target parameters in the target model; obtaining the target training image set corresponding to the detection task to be performed; after unfreezing the frozen parameters, fine-tuning the target model based on the target training image set. The present application freezes the relevant parameters in the model training stage, thereby only training the vocabulary mapping layer and the parameters related to the scene text task to be performed, and unfreezes all learnable parameters in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a model training method, device, electronic device, computer-readable medium, and computer program product. Background Art

[0002] Scene text-related tasks are an important research direction in computer vision. They involve various tasks related to detecting and recognizing textual information in natural scenes. These tasks are of great significance to image search, instant translation, robotic navigation, industrial automation, and other fields. Existing research primarily focuses on text detection and recognition.

[0003] Traditional scene text detection models, such as DBNet, focus on improving text detection performance through a multi-step approach based on specialized representations of text components. On the other hand, text recognition models, such as ParSeq, attempt to improve text recognition performance by using deep learning techniques, particularly CNN and RNN-based models.

[0004] However, solutions based on existing technologies still face challenges in handling text detection or text recognition in complex scenarios. Summary of the Invention

[0005] Various aspects of the present application provide a model training method, apparatus, electronic device, computer-readable medium, and computer program product.

[0006] In one aspect of the present application, a model training method is provided, wherein the method comprises:

[0007] After completing the pre-training of the target model, the parameters other than the target parameters in the target model are frozen to train the corresponding parts of the target parameters.

[0008] Obtain the target training image set corresponding to the detection task to be performed;

[0009] After unfreezing the frozen parameters, the target model is fine-tuned based on the target training image set.

[0010] In one aspect of the present application, a model training device is provided, wherein the device comprises:

[0011] After completing the pre-training of the target model, a device is used to train the part corresponding to the target parameters by freezing the parameters other than the target parameters in the target model;

[0012] means for acquiring a target training image set corresponding to a detection task to be performed;

[0013] A device for fine-tuning a target model based on a target training image set after unfreezing the frozen parameters.

[0014] In another aspect of the present application, a computer-readable storage medium is provided, on which computer program instructions are stored. The computer program instructions can be executed by a processor to implement the method of the embodiment of the present application.

[0015] In another aspect of the present application, a computer program product is provided, including a computer program, which implements the method of the embodiment of the present application when executed by a processor.

[0016] In the solution provided in the embodiment of the present application, by freezing the relevant parameters in the model training stage, only the parameters related to the vocabulary mapping layer and the scene text task to be executed are trained, and all learnable parameters are unfrozen in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set, and can complete the detection and recognition of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training; by constructing and training a model based on the Transformer architecture including a backbone network, an encoder layer and a decoder layer, multiple tasks such as detection, recognition and linking are integrated into a unified framework, so that it can adapt to different target data sets and vocabularies, and has good flexibility and adaptability; the embodiment of the present application uses a denoising training method based on Bezier curves to optimize the performance of the model, thereby improving the accuracy and robustness of the model on the target data set. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, a brief introduction is given below to the drawings required for use in the embodiments or descriptions of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Other features, objects and advantages of the present application will become more apparent upon reading the detailed description of non-limiting embodiments made with reference to the following drawings:

[0019] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown;

[0020] Figure 2 shows an exemplary model structure schematic diagram according to an embodiment of the present application;

[0021] Figure 3 A schematic diagram of the structure of the model training device provided in an embodiment of the present application is shown;

[0022] Figure 4 A structural diagram of a device suitable for implementing the solution in the embodiments of the present application is shown.

[0023] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0024] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0025] In a typical configuration of the present application, the terminal and the equipment of the service network each include one or more processors (CPUs), input / output interfaces, network interfaces and memories.

[0026] Memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0027] Computer-readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology for information storage. The information can be computer program instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc-read only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassettes, magnetic tape disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information that can be accessed by a computing device.

[0028] Figure 1 A flow chart of a model training method provided in an embodiment of the present application is shown, wherein the method comprises at least step S101, step S102 and step S103.

[0029] In practical scenarios, the execution subject of this method can be a network device or an application running on a network device. The network device includes, but is not limited to, a network host, a single network server, a set of multiple network servers, or a collection of computers based on cloud computing, and can be used to implement some of the processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing. Cloud computing is a type of distributed computing, consisting of a group of loosely coupled computers forming a virtual computer.

[0030] The concepts involved in the embodiments of this application are explained below.

[0031] Transformer architecture: A model based on the Transformer architecture is a deep learning model. The core features of the Transformer model include:

[0032] Self-Attention: The self-attention mechanism allows the model to process a sequence so that each element can interact directly with other elements in the sequence, regardless of their position in the sequence. This mechanism enables the model to capture the relationship between any two elements in the sequence, regardless of their distance in the sequence. Parallel Processing: Models based on the Transformer architecture can process all elements in a sequence in parallel, which greatly improves model training efficiency.

[0033] Encoder-Decoder Architecture: Models based on the Transformer architecture use an encoder-decoder architecture for sequence-to-sequence tasks such as machine translation. The encoder processes the input sequence, and the decoder generates the output sequence. Each encoder and decoder consists of multiple identical layers, each of which includes a self-attention mechanism and a feedforward neural network.

[0034] Positional encoding: Because the Transformer model itself does not capture positional information in a sequence, positional encoding is required to provide the positional information of each element in the sequence. Positional encoding is typically a vector of the same length as the input sequence. It is added to the input sequence through a learned or predefined method to help the model understand the order of elements in the sequence.

[0035] Multi-head attention: The self-attention mechanism in the Transformer-based model can be extended to multi-head attention, which captures information from different representation subspaces at the same time. This increases the expressive power of the model and allows it to capture different information in different representation subspaces.

[0036] Layer normalization and residual connections: The Transformer model applies layer normalization on the output of each sublayer (self-attention layer and feedforward neural network layer) and uses residual connections. These techniques help avoid the gradient disappearance or explosion problem in deep network training.

[0037] Bezier curve: The Bezier curve method is a technology used for natural scene text detection and recognition (STR), which uses Bezier curves to represent and fit the shape of text areas instead of traditional rectangles or polygons.

[0038] In some embodiments, the model of the embodiment of the present application is a model based on the Transformer architecture, which is used to perform various scene text tasks, such as text detection, text link detection, and text recognition.

[0039] In some embodiments, when training the model of this application, all parameters other than the vocabulary and the parameters related to the scene text task to be performed are frozen, so that only the vocabulary mapping layer and the task-related text classification head are trained. During the fine-tuning phase, all learnable parameters are unfrozen to fine-tune the model based on the dataset corresponding to the scene text task.

[0040] exist Figure 1 Before step S101 shown, the method further includes step S104.

[0041] In step S104, the target model is pre-trained so that the target model can learn a universal feature representation.

[0042] Specifically, pre-training on a large public dataset allows the target model to learn universal feature representations that can capture the basic patterns in images and general properties of objects.

[0043] During the pre-training phase, the model loads a pre-trained vocabulary (also called a vocabulary or embedding layer). This embedding layer contains pre-trained vocabulary weights. These weights are learned from a large amount of data and help the model better understand and process text data. Another important effect of pre-trained weights is that they reduce the time and resource consumption of model training. Because the model has already learned some common features, when performing scene text tasks, only these features need to be fine-tuned, without having to train from scratch.

[0044] Refer to the following Figure 1 To illustrate, in step S101, after completing the pre-training of the target model, the parameters other than the target parameters in the target model are frozen to train the corresponding parts of the target parameters.

[0045] According to one embodiment, the target parameters include parameters related to the vocabulary and parameters related to the scene text task to be performed.

[0046] The vocabulary-related parameters include parameters related to the model's embedding layer or other parameters directly related to the vocabulary. Optionally, the vocabulary-related parameters include parameters of the word embedding layer of the expanded new vocabulary. This approach, after the vocabulary is expanded, can achieve excellent detection results for special characters with a small sample size.

[0047] By freezing all parameters except the target parameters in the target model, we can focus on training the vocabulary part of the model and the parts related to the scene text task to be performed. Based on this training method, the model does not need to learn all parameters at the same time, thus improving the model training efficiency.

[0048] According to one embodiment, the target parameters may further include other preset parameters, and step S101 further includes the following steps:

[0049] Load the target model's pre-trained model weights on a predetermined large image dataset.

[0050] Next, determine the parameters that need to be frozen.

[0051] The method may determine which parameters need to be frozen by obtaining pre-stored freezing parameter indication information, or may determine the parameters need to be frozen in real time based on user input.

[0052] For example, by setting the requires_grad attribute of these parameters to False. Based on this setting, the gradients of these parameters will not be calculated during backpropagation, so their values ​​will not change, thus preventing these parameters from being updated during training.

[0053] Then, after freezing the determined parameters, continue training other parts of the model.

[0054] According to one embodiment, the method uses a Bezier curve-based noise reduction training method in S101 to perform training on a portion corresponding to the target parameter. The training process includes steps S1011 to S1015.

[0055] The Bezier curve-based denoising training method is targeted at text recognition tasks based on the Transformer architecture. By improving traditional denoising training, the method makes the model more stable and efficient when processing text of arbitrary shapes.

[0056] In step S1011 , the query of the noise reduction part is decomposed into a noise position query and a noise content query.

[0057] In step S1012, a noise position query is generated using the four control points of the Bezier curve. After adding noise to the Bezier control points, a new set of control points is obtained.

[0058] In step S1013 , the coordinates of multiple points are uniformly sampled from the generated Bezier curve using the Bezier control points to which noise is added.

[0059] In step S1014, the obtained coordinates are position-encoded and processed by a two-layer multilayer perceptron (MLP) to obtain a corresponding noise position query.

[0060] In step S1015 , a noise content query is initialized by a mask character sliding method to assist in the alignment of text content and position.

[0061] Specifically, we first determine the number of valid characters t, which refers to the number of characters in the input sequence that actually make sense. Then, we perform an integer division operation To calculate the number of times each valid character should be cloned so that the total length T of the sequence is evenly distributed to each valid character. In addition, since T may not be completely divisible by t, there will be a remainder Indicates that there are k additional spaces to be allocated. In order to distribute these additional spaces fairly, an additional clone will be allocated to each of the first k valid characters in the sequence to ensure that each character is allocated as evenly as possible.

[0062] After obtaining the sliding characters, a masking operation is used to control the number of consecutive characters. This means that a portion of the consecutive characters are flipped to background labels with a certain probability. After processing the positive part, noise is added to the characters in the positive and negative parts, causing these characters to flip to other characters with a probability of λ. These characters are then embedded to obtain the noise content query, where all the characters initialized in the negative part are background.

[0063] According to one embodiment, when performing model inference, the portion of applying the Bezier curve-based denoising training method is removed.

[0064] Among them, the part that applies the Bezier curve-based denoising training method can be removed during model inference by setting the denoising training part to an inactive state in the model's configuration file or code or directly disabling the layer in the model that applies denoising training.

[0065] By removing the part of the model that applies the Bezier curve-based denoising training method during the model inference phase, the efficiency of the inference process is ensured and additional computational burden is avoided.

[0066] Continue to refer to Figure 1 To illustrate, in step S102 , a target training image set corresponding to the scene text task to be performed is obtained.

[0067] The scene text tasks may include various tasks for detecting, recognizing, and understanding text information in scene images, such as text detection, text link detection, and text recognition tasks.

[0068] Optionally, the method cleans, labels, and preprocesses the acquired dataset for use in training.

[0069] In step S103 , after the frozen parameters are unfrozen, the target model is fine-tuned based on the target training image set.

[0070] Fine-tuning is a process that preserves the useful features learned by the model during pre-training and adapts it to new tasks with minor adjustments. This process can significantly improve the performance of the model on specific tasks, thereby reducing training costs and time.

[0071] Optionally, during fine-tuning, the pre-trained target model can be structurally adjusted based on the requirements of the scene text task being performed. For example, in a text recognition task, the final classification layer may need to be replaced to match the number of categories in the target dataset.

[0072] According to one embodiment, the model is a model based on the Transformer architecture, and the structure of the model includes a backbone network, an encoder, and a decoder. This embodiment trains the model by performing the following steps S1031 to S1033:

[0073] In step S1031, global feature extraction processing is performed on the input image in the backbone network to obtain corresponding global features and local features.

[0074] Optionally, the input image is preprocessed before feature extraction. This may include operations such as resizing, normalizing, and enhancing contrast to ensure that the model receives input in a consistent format.

[0075] Among them, the global features obtained through the backbone network help the model understand the global context of the image.

[0076] In step S1032 , global features of the input image are further extracted in the encoder.

[0077] Among them, the global features obtained through the backbone network help the model understand the global context of the image.

[0078] In step S1033, the decoder further performs local feature extraction processing on the input image based on the global features to obtain corresponding local features.

[0079] To gain a more nuanced understanding of image content, the model needs to further extract local features. This is achieved by using deeper network structures that are capable of identifying detailed information such as edges, corners, and textures within the image. These local features help the model recognize the outlines and shapes of text, enabling text detection even when the text appears in a variety of orientations and sizes within the image.

[0080] In step S1034, a Bezier curve-based noise reduction training method is applied in the decoder to train the model to learn how to effectively process noise and interference in the image.

[0081] Among them, the local features output by the decoder layer are used to locate the text area in the image and identify the text content

[0082] According to one embodiment, the method further includes steps 106 to 109 .

[0083] In step S106 , an image to be processed corresponding to the scene text task to be executed is obtained.

[0084] The images to be processed may include photos or video frames of various types of scenes. For example, for urban scenes, the images to be processed may include images showing urban buildings, streets, traffic, road signs, etc. For indoor scenes, the images to be processed may include images showing indoor environments such as homes, offices, and stores.

[0085] In step S107, the image to be processed is input into the trained target model.

[0086] The trained target model is trained through the above step 101 and is obtained by fine-tuning based on the data set corresponding to the task through the above step S103.

[0087] In step S108, feature extraction is performed based on the target model to obtain image features corresponding to the image to be processed, wherein the image features include global features and local features.

[0088] For models based on the Transformer architecture, feature extraction is performed based on the encoder and decoder to obtain image features corresponding to the image to be processed.

[0089] In step S109, the result of the scene text task output by the target model based on the image features is obtained.

[0090] The method provided in the embodiment of the present application freezes the relevant parameters in the training stage, so that only the parameters related to the vocabulary mapping layer and the scene text task to be executed are trained, and all learnable parameters are unfrozen in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set, and can complete the detection and recognition of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training; constructing and training a model based on the Transformer architecture including a backbone network, an encoder layer and a decoder layer, integrating multiple tasks such as detection, recognition and linking into a unified framework, which can adapt to different target data sets and vocabularies and has good flexibility and adaptability; using a denoising training method to optimize the performance of the unified model, thereby improving the accuracy and robustness of the model on the target data set.

[0091] Refer to the following Figure 2 The exemplary model structure shown is used to illustrate the method of the embodiment of the present application. The model of this example is a model based on the Transformer architecture, which can be used to perform various scene text tasks, such as text detection, text link detection and text recognition tasks.

[0092] Reference Figure 2 The model in this example mainly consists of three parts: the backbone network, the encoder, and the decoder. The encoder contains multiple encoder layers, and the decoder contains multiple decoder layers. These three parts are introduced below.

[0093] 1) Backbone network: The backbone network in this example is the ViTAE-v2 backbone network, which processes the input image and extracts global features. These features contain the basic visual information of the image and provide the basis for subsequent text detection and recognition.

[0094] 2) Encoder: The encoder further extracts global features of the input image.

[0095] 3) Decoder: The decoder extracts local features of the input image. These local features are used to capture detailed information such as the shape, size, and orientation of the text.

[0096] The Bezier curve-based denoising training method described in step S105 is applied to the decoder layer to train the model to effectively handle noise and interference in the image, thereby further optimizing the model's performance. Furthermore, the Bezier curve-based denoising training method helps the model better understand and handle changes in text shape, thereby providing more reliable detection or recognition results in practical applications.

[0097] This example uses a hybrid training method to train the model. The process of this hybrid training method includes:

[0098] 1) Freeze all parameters except the vocabulary and the parameters related to the scene text task to be performed, and train the vocabulary mapping layer and the structure related to the scene text task;

[0099] Through training, the Embedding layer can learn the contextual information of each word, enabling the model to better understand semantics when processing text.

[0100] Optionally, the structure associated with the scene text task may include a classification head. This head is typically a linear layer that converts the high-dimensional vector output by the embedding layer into specific category labels. By performing a linear transformation on the embedding vector, the model can output a probability distribution for each category, thereby achieving the classification task.

[0101] 2) Obtain the target data set for the task to be performed;

[0102] 3) Unfreeze all learnable parameters and fine-tune the model based on the target dataset.

[0103] After unfreezing all learnable parameters, the model recalibrates all parameters, including the weights of the embedding layers and the classification head. This process allows the model to better adapt to the specific target dataset. Fine-tuning typically requires several thousand steps of training to ensure good performance on the new dataset. Through fine-tuning, the model not only leverages the knowledge learned during pre-training but also further optimizes based on the characteristics of the target dataset, thereby improving classification accuracy.

[0104] The classification head structure can output the final binary classification prediction result. For example, if the task to be performed is a text detection task, the prediction result output by the model is the text detection result.

[0105] The method provided in the embodiment of the present application freezes the relevant parameters in the model training stage, so that only the parameters related to the vocabulary mapping layer and the scene text task to be executed are trained, and all learnable parameters are unfrozen in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set, and can complete the detection and recognition of hundreds of targets in text images within a few hours. At the same time, the vocabulary can be easily expanded, further improving the efficiency of model training; by constructing and training a model based on the Transformer architecture including a backbone network, an encoder layer and a decoder layer, multiple tasks such as detection, recognition and linking are integrated into a unified framework, so that it can adapt to different target data sets and vocabularies, and has good flexibility and adaptability; the embodiment of the present application uses a denoising training method based on Bezier curves to optimize the performance of the model, thereby improving the accuracy and robustness of the model on the target data set.

[0106] In addition, the present invention also provides a model training device, the structure of which is as follows: Figure 3 The device includes: a device for training the corresponding parts of the target parameters by freezing the parameters other than the target parameters in the target model after completing the pre-training of the target model (hereinafter referred to as "parameter training device 101"), a device for acquiring a target training image set corresponding to the detection task to be performed (hereinafter referred to as "data acquisition device 102"), and a device for fine-tuning the target model based on the target training image set after unfreezing the frozen parameters (hereinafter referred to as "model fine-tuning device 103").

[0107] Wherein, the device also includes a pre-training device.

[0108] The pre-training device pre-trains the target model so that the target model can learn a universal feature representation.

[0109] Specifically, the pre-training mechanism is pre-trained on a large public dataset, allowing the target model to learn universal feature representations that can capture the basic patterns in images and general properties of objects.

[0110] During the pre-training phase, the model loads a pre-trained vocabulary (also called a vocabulary or embedding layer). This embedding layer contains pre-trained vocabulary weights. These weights are learned from a large amount of data and help the model better understand and process text data. Another important effect of pre-trained weights is that they reduce the time and resource consumption of model training. Because the model has already learned some common features, when performing scene text tasks, only these features need to be fine-tuned, without having to train from scratch.

[0111] Refer to the following Figure 3 To illustrate, after completing the pre-training of the target model, the parameter training device 101 freezes the parameters other than the target parameters in the target model to train the corresponding parts of the target parameters.

[0112] According to one embodiment, the target parameters include parameters related to the vocabulary and parameters related to the scene text task to be performed.

[0113] The vocabulary-related parameters include parameters related to the model's embedding layer or other parameters directly related to the vocabulary. Optionally, the vocabulary-related parameters include parameters of the word embedding layer of the expanded new vocabulary. This approach, after the vocabulary is expanded, can achieve excellent detection results for special characters with a small sample size.

[0114] By freezing all parameters except the target parameters in the target model, we can focus on training the vocabulary part of the model and the parts related to the scene text task to be performed. Based on this training method, the model does not need to learn all parameters at the same time, thus improving the model training efficiency.

[0115] According to one embodiment, the target parameters may also include other preset parameters, and the parameter training device 101 includes a weight loading device, a parameter determination device and a freeze training device.

[0116] The weight loading device loads the model weights of the target model pre-trained on a predetermined large image dataset.

[0117] Next, the parameter determination device determines the parameters that need to be frozen.

[0118] The parameter determination device may determine which parameters need to be frozen by acquiring pre-stored freezing parameter indication information. Alternatively, the parameter determination device may determine the parameters need to be frozen in real time based on user input.

[0119] For example, by setting the requires_grad attribute of these parameters to False. Based on this setting, the gradients of these parameters will not be calculated during backpropagation, so their values ​​will not change, thus preventing these parameters from being updated during training.

[0120] Next, the freeze training device freezes the determined parameters and continues to train other parts of the model.

[0121] According to one embodiment, the parameter training device 101 uses a Bezier curve-based noise reduction training method to train the corresponding part of the target parameter. The parameter training device 101 further includes a query decomposition device, a control point generation device, a coordinate point sampling device, a position query acquisition device and an initialization noise device.

[0122] The Bezier curve-based denoising training method is targeted at text recognition tasks based on the Transformer architecture. By improving traditional denoising training, the method makes the model more stable and efficient when processing text of arbitrary shapes.

[0123] The query decomposition device decomposes the query of the noise reduction part into a noise position query and a noise content query.

[0124] The control point generating device generates a noise position query using four control points of the Bezier curve, and obtains a set of new Bezier control points after adding noise to the Bezier control points.

[0125] The coordinate point sampling device uses Bezier control points with added noise to uniformly sample multiple point coordinates from the generated Bezier curve.

[0126] The location query acquisition device obtains the corresponding noise location query by performing location encoding and two-layer multilayer perceptron (MLP) processing on the obtained coordinates.

[0127] The noise initialization device initializes the noise content query through the mask character sliding method.

[0128] Specifically, we first determine the number of valid characters t, which refers to the number of characters in the input sequence that actually make sense. Then, we perform an integer division operation To calculate the number of times each valid character should be cloned so that the total length T of the sequence is evenly distributed to each valid character. In addition, since T may not be completely divisible by t, there will be a remainder Indicates that there are k additional spaces to be allocated. In order to distribute these additional spaces fairly, an additional clone will be allocated to each of the first k valid characters in the sequence to ensure that each character is allocated as evenly as possible.

[0129] After obtaining the sliding characters, a masking operation is used to control the number of consecutive characters. This means that a portion of the consecutive characters are flipped to background labels with a certain probability. After processing the positive part, noise is added to the characters in the positive and negative parts, causing these characters to flip to other characters with a probability of λ. These characters are then embedded to obtain the noise content query, where all the characters initialized in the negative part are background.

[0130] According to one embodiment, when performing model inference, the apparatus removes the portion of applying the Bezier curve-based denoising training method.

[0131] Among them, the part that applies the Bezier curve-based denoising training method can be removed during model inference by setting the denoising training part to an inactive state in the model's configuration file or code or directly disabling the layer in the model that applies denoising training.

[0132] By removing the part of the model that applies the Bezier curve-based denoising training method during the model inference phase, the efficiency of the inference process is ensured and additional computational burden is avoided.

[0133] Continue to refer to Figure 3 To illustrate, the data acquisition device 102 acquires a target training image set corresponding to the scene text task to be performed.

[0134] The scene text tasks may include various tasks for detecting, recognizing, and understanding text information in scene images, such as text detection, text link detection, and text recognition tasks.

[0135] Optionally, the data acquisition device 102 cleans, labels, and preprocesses the acquired data set for use in training.

[0136] The model fine-tuning device 103 unfreezes the frozen parameters and then fine-tunes the target model based on the target training image set.

[0137] The model fine-tuning device 103 performs fine-tuning to retain the useful features learned by the model during the pre-training phase and adapts it to the new task with a small amount of adjustment. The fine-tuning process can significantly improve the performance of the model on a specific task, thereby reducing training costs and time.

[0138] Optionally, during the fine-tuning process, the model fine-tuning device 103 performs structural adjustments on the pre-trained target model according to the requirements of the scene text task to be performed. For example, in a text recognition task, the last classification layer may need to be replaced to match the number of categories in the target dataset.

[0139] According to one embodiment, the model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder, and a decoder. The apparatus of this embodiment trains the model by performing the following operations:

[0140] The input image is processed for global feature extraction in the backbone network to obtain the corresponding global features and local features.

[0141] Optionally, the input image is preprocessed before feature extraction, which may include operations such as resizing, normalizing, and enhancing contrast to ensure that the model receives input in a consistent format. The global features obtained by the backbone network help the model understand the global context of the image.

[0142] Next, the encoder further extracts global features of the input image. The global features obtained by the backbone network help the model understand the global context of the image.

[0143] Next, in the decoder, local feature extraction processing is further performed on the input image based on the global features to obtain corresponding local features.

[0144] To gain a more nuanced understanding of image content, the model needs to further extract local features. This is achieved by using deeper network structures that are capable of identifying detailed information such as edges, corners, and textures within the image. These local features help the model recognize the outlines and shapes of text, enabling text detection even when the text appears in a variety of orientations and sizes within the image.

[0145] Next, a Bezier curve-based denoising training method is applied in the decoder to train the model to learn how to effectively deal with noise and interference in the image.

[0146] Among them, the local features output by the decoder layer are used to locate the text area in the image and identify the text content

[0147] According to one embodiment, the device further includes an image acquisition device, an image input device, a feature extraction device and a result output device.

[0148] The image acquisition device acquires an image to be processed corresponding to a scene text task to be executed.

[0149] The images to be processed may include photos or video frames of various types of scenes. For example, for urban scenes, the images to be processed may include images showing urban buildings, streets, traffic, road signs, etc. For indoor scenes, the images to be processed may include images showing indoor environments such as homes, offices, and stores.

[0150] The image input device inputs the image to be processed into the trained target model.

[0151] The trained target model is trained by the parameter training device 101 and is fine-tuned based on the data set corresponding to the task through the operation of the model fine-tuning device 103.

[0152] The feature extraction device extracts features based on the target model to obtain image features corresponding to the image to be processed, wherein the image features include global features and local features.

[0153] For models based on the Transformer architecture, feature extraction is performed based on the encoder and decoder to obtain image features corresponding to the image to be processed.

[0154] The result output device obtains the result of the scene text task output by the target model based on the image features.

[0155] The device provided in the embodiment of the present application freezes the relevant parameters in the model training stage, thereby only training the parameters related to the vocabulary mapping layer and the scene text task to be executed, and unfreezes all learnable parameters in the fine-tuning stage to fine-tune the model based on the data set corresponding to the scene text task. This training method reduces the cost of training and fine-tuning on the target data set, and can complete the detection and recognition of hundreds of targets in text images within a few hours. At the same time, it can easily expand the vocabulary, further improving the efficiency of model training; by constructing and training a model based on the Transformer architecture including a backbone network, an encoder layer and a decoder layer, multiple tasks such as detection, recognition and linking are integrated into a unified framework, so that it can adapt to different target data sets and vocabularies, and has good flexibility and adaptability; the embodiment of the present application uses a denoising training method based on Bezier curves to optimize the performance of the model, thereby improving the accuracy and robustness of the model on the target data set.

[0156] Based on the same inventive concept, an electronic device is also provided in an embodiment of the present application. The method corresponding to the electronic device can be the model training method in the aforementioned embodiment, and its principle of solving the problem is similar to that of the method. The electronic device provided in an embodiment of the present application includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the methods and / or technical solutions of the aforementioned multiple embodiments of the present application.

[0157] The electronic device may be a user device, or a device formed by integrating a user device and a network device via a network, or an application running on the above device. The user device includes but is not limited to various terminal devices such as computers, mobile phones, tablets, smart watches, and bracelets. The network device includes but is not limited to network hosts, single network servers, multiple network server sets, or cloud computing-based computer collections, and can be used to implement some of the processing functions when setting an alarm. Here, the cloud is composed of a large number of hosts or network servers based on cloud computing (Cloud Computing), where cloud computing is a type of distributed computing, a virtual computer composed of a group of loosely coupled computers.

[0158] Figure 4The structure of a device suitable for implementing the method and / or technical solution in the embodiment of the present application is shown. The device 1200 includes a central processing unit (CPU) 1201, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 1202 or the program loaded from the storage part 1208 into the random access memory (RAM) 1203. Various programs and data required for system operation are also stored in RAM 1203. CPU 1201, ROM 1202 and RAM 1203 are connected to each other through a bus 1204. Input / output (I / O) interface 1205 is also connected to bus 1204.

[0159] The following components are connected to the I / O interface 1205: an input section 1206 including a keyboard, a mouse, a touch screen, a microphone, an infrared sensor, and the like; an output section 1207 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), an LED display, an OLED display, and a speaker; a storage section 1208 including one or more computer-readable media such as a hard disk, an optical disk, a magnetic disk, and a semiconductor memory; and a communication section 1209 including a network interface card such as a LAN (Local Area Network) card, a modem, and the like. The communication section 1209 performs communication processing via a network such as the Internet.

[0160] In particular, the methods and / or embodiments of the present application can be implemented as computer software programs. For example, the embodiments disclosed in the present application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for executing the method shown in the flowchart. When the computer program is executed by the central processing unit (CPU) 1201, the above-mentioned functions defined in the method of the present application are performed.

[0161] Another embodiment of the present application further provides a computer-readable storage medium having computer program instructions stored thereon, which can be executed by a processor to implement the methods and / or technical solutions of any one or more embodiments of the present application.

[0162] Specifically, the present embodiment can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device.

[0163] A computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. This propagated data signal may take a variety of forms, including, but not limited to, electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0164] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0165] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0166] The flow chart or block diagram in the accompanying drawings illustrate the possible architecture, functions and operations of the equipment, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code include one or more executable instructions for realizing the logical function of the specification. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated system for hardware that performs the function or operation of the specification, or can be implemented with a combination of dedicated hardware and computer instructions.

[0167] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0168] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or page components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0169] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0170] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or in the form of hardware plus software functional units.

[0171] The above-mentioned integrated unit implemented in the form of a software functional unit can be stored in a computer-readable storage medium. The above-mentioned software functional unit is stored in a storage medium and includes a number of instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute some steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program code.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

[0173] Furthermore, it is clear that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. Multiple units or devices recited in a device claim may also be implemented by a single unit or device through software or hardware. Terms such as "first" and "second" are used to indicate names and do not imply any particular order.

Claims

1. A model training method, wherein: The method comprises: After completing the pre-training of the target model, the parameters other than the target parameters in the target model are frozen to train the corresponding parts of the target parameters. Obtain the target training image set corresponding to the detection task to be performed; After unfreezing the frozen parameters, fine-tune the target model based on the target training image set; The method further comprises: Pre-training the target model so that the target model can learn a universal feature representation; The target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder, and a decoder. The method trains the model by the following steps: Perform global feature extraction on the input image in the backbone network to obtain corresponding global features and local features; Further extract global features of the input image in the encoder; In the decoder, based on the global features, the input image is further subjected to local feature extraction processing to obtain corresponding local features; A Bezier curve-based denoising training method is applied in the decoder to train the model to learn how to effectively deal with noise and interference in images.

2. The method according to claim 1, wherein The step of freezing parameters other than the target parameters in the target model to train the corresponding portion of the target parameters includes: Load the model weights of the target model pre-trained on a predetermined large image dataset; Determine the parameters that need to be frozen; After freezing the determined parameters, continue training other parts of the model.

3. The method according to claim 1, wherein The freezing of parameters other than the target parameters to train the corresponding portion of the target parameters includes: A Bezier curve-based denoising training method is used to train the corresponding parts of the target parameters.

4. The method according to claim 3, wherein: The method of using the Bezier curve-based denoising training method to train the corresponding part of the target parameter includes: Decompose the noise reduction part query into noise location query and noise content query; Generate noise position query using four control points of Bezier curve; After adding noise to the Bezier control points, a new set of control points is obtained; Using these noisy Bezier control points, uniformly sample multiple point coordinates from the generated Bezier curve; By performing position encoding and two-layer MLP processing on the obtained coordinates, the corresponding noise position query is obtained; Noisy content queries are initialized through a masked character sliding method to assist in the alignment of text content and location.

5. The method according to claim 3 or 4, wherein: The method further comprises: When performing model inference, remove the part that applies the Bezier curve-based denoising training method.

6. The method according to claim 1, wherein The method further comprises: Obtain the image to be processed corresponding to the scene text task to be executed; Input the image to be processed into the trained target model; Perform feature extraction based on the target model to obtain image features corresponding to the image to be processed, wherein the image features include global features and local features; Get the results of the scene text task based on the output of the target model based on image features.

7. A model training device, wherein: The device comprises: After completing the pre-training of the target model, a device is used to train the part corresponding to the target parameters by freezing the parameters other than the target parameters in the target model; means for acquiring a target training image set corresponding to a detection task to be performed; A device for fine-tuning a target model based on a target training image set after unfreezing the frozen parameters; The device further includes a pre-training device, which pre-trains the target model so that the target model can learn a universal feature representation; The target model is a model based on the Transformer architecture, the structure of which includes a backbone network, an encoder, and a decoder. The device trains the model by performing the following operations: Perform global feature extraction on the input image in the backbone network to obtain corresponding global features and local features; Further extract global features of the input image in the encoder; In the decoder, based on the global features, the input image is further subjected to local feature extraction processing to obtain corresponding local features; A Bezier curve-based denoising training method is applied in the decoder to train the model to learn how to effectively deal with noise and interference in images.

8. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

9. A computer-readable medium having computer program instructions stored thereon, wherein the computer program instructions can be executed by a processor to implement the method according to any one of claims 1 to 6.

10. A computer program product comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images