Data enhancement strategy searching method and device

Abstract

The invention provides a data enhancement strategy searching method and device. The method comprises the following steps: acquiring an enhancement operation set comprising at least two types of enhancement operations; different types of enhancement operations correspond to different execution probability ranges; obtaining feature distribution of a to-be-enhanced training set, and searching a target enhancement operation according to the feature distribution of the training set and an intensity parameter of each enhancement operation in the enhancement operation set; searching a target data enhancement strategy for enhancing the training set according to the feature distribution of the training set, a preset length and the intensity parameter of each target enhancement operation; wherein the preset length is used for randomly combining different target enhancement operations in the same type of enhancement operations into a data enhancement strategy. According to the embodiment of the invention, the search process of the data enhancement strategy can be simplified, and the calculation complexity is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to a data enhancement strategy search method and device. Background Art

[0002] Deep learning tasks usually require a large amount of training data. In order to obtain a large amount of training data, data enhancement strategies can be used to enhance the existing training data to improve the performance and effect of the model.

[0003] In related technologies, data augmentation strategy settings are mostly based on human prior knowledge. For example, when training a visual model, data augmentation operations are artificially added based on human prior knowledge. This artificial addition of data augmentation operations requires cumbersome parameter adjustments. Since the search space for these parameters is large, it is difficult to obtain the optimal set of parameters, resulting in poor search results for the data augmentation strategy. Summary of the invention

[0004] The embodiments of the present disclosure provide a data enhancement strategy search method and device.

[0005] According to a first aspect of an embodiment of the present disclosure, a data enhancement strategy search method is provided, the method comprising:

[0006] Obtaining an enhancement operation set including at least two types of enhancement operations, wherein different types of enhancement operations correspond to different execution probability ranges;

[0007] Obtaining the feature distribution of the training set to be enhanced;

[0008] Searching for a target enhancement operation according to the feature distribution of the training set and the strength parameter of each enhancement operation in the enhancement operation set;

[0009] According to the feature distribution of the training set, the preset length and the strength parameters of each target enhancement operation, a target data enhancement strategy for enhancing the training set is searched; wherein the preset length is used to randomly combine different target enhancement operations in the same type of enhancement operations into a data enhancement strategy.

[0010] In one embodiment, the obtaining of an enhancement operation set including at least two types of enhancement operations includes:

[0011] Determining a plurality of enhancement operations and an execution probability and a strength parameter of each enhancement operation according to differences in sample attributes between the training set and the unlabeled test set;

[0012] At least two types of enhancement operations forming the enhancement operation set are determined according to a comparison result between the execution probability of each enhancement operation and a preset probability.

[0013] In one embodiment, obtaining the feature distribution of the training set to be enhanced includes:

[0014] Acquire a feature distribution of the training set according to a first loss function of the pre-training model and a feature description extracted by the pre-training model from each sample image in the training set;

[0015] The first loss function includes: taking the feature center of the feature description of all sample images in the training set as the center of a sphere and taking the distance threshold as the radius, and the distance from the feature description outside the spherical surface to the spherical surface.

[0016] In one embodiment, searching for a target enhancement operation according to the feature distribution of the training set and the strength parameter of each enhancement operation in the enhancement operation set includes:

[0017] Adjusting the execution probabilities of all enhancement operations in the enhancement operation set to the same probability;

[0018] For each enhancement operation in the enhancement operation set, perform the following operations:

[0019] According to the intensity parameter of the enhancement operation, data enhancement is performed on each sample image in the training set respectively;

[0020] Obtaining a feature description extracted by the pre-trained model for each of the sample images enhanced by the enhancement operation;

[0021] Obtaining a first distance between a center of a feature distribution of the training set and a feature description of each of the sample images enhanced by the enhancement operation;

[0022] It is determined whether the enhancement operation is the target enhancement operation according to the plurality of first distances and the distance threshold.

[0023] In one embodiment, determining whether the enhancement operation is the target enhancement operation according to the plurality of first distances and the distance thresholds includes:

[0024] Determine the number of first distances whose absolute values ​​of differences from the distance threshold are smaller than a preset difference;

[0025] When the ratio of the determined number of the first distances to the total number of the first distances exceeds a first preset ratio, determining the enhancement operation as the target enhancement operation;

[0026] When the ratio of the determined number of the first distances to the total number of the first distances does not exceed the first preset ratio, it is determined that the enhancement operation is not the target enhancement operation.

[0027] In one embodiment, the target enhancement operation includes: a first target enhancement operation belonging to a first type of enhancement operation and a second target enhancement operation belonging to a second type of enhancement operation; wherein a lower limit value of an execution probability range corresponding to the first type of enhancement operation is greater than an upper limit value of an execution probability range corresponding to the second type of enhancement operation;

[0028] The step of searching for a target data enhancement strategy for enhancing the training set according to the feature distribution of the training set, the preset length, and the strength parameters of each target enhancement operation comprises:

[0029] Determine each data enhancement strategy in the first data enhancement set as a first target data enhancement strategy; wherein the first data enhancement set includes: data enhancement strategies obtained by performing a random combination of the preset length on different first target enhancement operations according to the strength parameters of each first target enhancement operation;

[0030] and / or,

[0031] According to the feature distribution of the training set, a second target data enhancement strategy is searched in the second data enhancement set; wherein the second data enhancement set includes: a data enhancement strategy obtained by randomly combining different second target enhancement operations of the preset length according to the strength parameters of each second target enhancement operation.

[0032] In one embodiment, searching for a second target data enhancement strategy in a second data enhancement set according to the feature distribution of the training set includes:

[0033] For each data enhancement strategy in the second data enhancement set, perform the following operations:

[0034] According to the strength parameter of the data enhancement strategy, data enhancement is performed on each sample image in the training set;

[0035] Obtaining a feature description extracted by the pre-trained model for each of the sample images enhanced by the data enhancement strategy;

[0036] Obtaining a second distance between the center of the feature distribution of the training set and the feature description of each of the sample images enhanced by the data enhancement strategy;

[0037] According to the plurality of second distances and the distance threshold, determine whether the data enhancement strategy is the second target data enhancement strategy.

[0038] In one embodiment, determining whether the data enhancement strategy is the second target data enhancement strategy according to the plurality of second distances and the distance threshold comprises:

[0039] Determine the number of second distances whose absolute values ​​of differences from the distance threshold are smaller than a preset difference;

[0040] When the ratio of the determined number of the second distances to the total number of the second distances exceeds a second preset ratio, determining that the data enhancement strategy is the second target data enhancement strategy;

[0041] When the ratio of the determined number of the second distances to the total number of the second distances does not exceed the second preset ratio, it is determined that the data enhancement strategy is not the second target data enhancement strategy.

[0042] In one embodiment, the method further comprises:

[0043] Fine-tuning the pre-trained model according to the enhanced training set and the label data of the training set;

[0044] Based on the fine-tuned pre-trained model, the steps of obtaining the feature distribution, searching for the target enhancement operation, searching for the target data enhancement strategy, and fine-tuning the pre-trained model are repeated until the training end condition is met.

[0045] In one embodiment, the pre-trained model is: a multi-task learning model pre-trained by the training set; the multi-task learning model includes a feature extraction network, a single classification head and a task head corresponding to the target task; the feature extraction network is respectively connected to the task head and the single classification head;

[0046] Wherein, the feature extraction network is used to extract features from each sample image in the training set;

[0047] The single classification head is used to extract the feature description of each sample image from the feature extraction result output by the feature extraction network; the loss function of the single classifier is the first loss function.

[0048] According to a second aspect of an embodiment of the present disclosure, a data enhancement strategy search device is provided, the device comprising:

[0049] A first acquisition module is used to acquire an enhancement operation set including at least two types of enhancement operations; wherein different types of enhancement operations correspond to different execution probability ranges;

[0050] A second acquisition module is used to obtain the feature distribution of the training set to be enhanced;

[0051] A first search module, configured to search for a target enhancement operation according to a feature distribution of the training set and a strength parameter of each enhancement operation in the enhancement operation set;

[0052] The second search module is used to search for a target data enhancement strategy for enhancing the training set based on the feature distribution of the training set, a preset length, and the strength parameters of each target enhancement operation; wherein the preset length is used to randomly combine different target enhancement operations in the same type of enhancement operations into a data enhancement strategy.

[0053] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data enhancement strategy search method described in any one of the first aspects is implemented.

[0054] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the data enhancement strategy search method described in any one of the first aspects is implemented.

[0055] The disclosed embodiments provide a data enhancement strategy search method and device, which searches for a target enhancement operation according to the feature distribution of a training set to be enhanced and the strength parameter of each enhancement operation in an enhancement operation set. Since the enhancement operation set includes at least two types of enhancement operations, each enhancement operation in different types of enhancement operations can participate in the search process of the target enhancement operation, thereby ensuring the diversity of the enhancement operations. Moreover, since the target enhancement operation is determined in combination with the feature distribution of the training set and the strength parameter of each enhancement operation, the search effect for the target enhancement operation can be improved.

[0056] In addition, by searching for target data enhancement strategies for enhancing the training set based on the feature distribution of the training set, the preset length, and the strength parameters of each target enhancement operation, since the target data enhancement is searched from the data enhancement strategies for random combinations of different target enhancement operations in the same type of enhancement operations based on the preset length, the search range does not include data enhancement strategies formed by target enhancement operations belonging to different types of enhancement operations. In this way, in the process of searching for data enhancement strategies, the search space of data enhancement strategies can be effectively reduced, thereby improving the search efficiency of data enhancement strategies.

[0057] Moreover, the entire search process only needs to determine the execution probability of the enhancement operation and the preset length of the enhancement operation combination. Since fewer parameters need to be set, the search process of the data enhancement strategy is greatly simplified and the computational complexity is reduced.

[0058] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0060] Figure 1 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0061] Figure 2 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0062] Figure 3 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0063] Figure 4 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0064] Figure 5 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0065] Figure 6 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0066] Figure 7 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0067] Figure 8 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0068] Fig. 9 A schematic diagram of a process flow of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown;

[0069] Fig.10a A schematic diagram of the structure of a multi-task learning model provided by an embodiment of the present disclosure is shown;

[0070] Fig.10b A schematic diagram showing the feature distribution of a training set provided by an embodiment of the present disclosure is shown;

[0071] Fig.10c A schematic diagram showing the feature distribution of the training set before and after enhancement provided by an embodiment of the present disclosure;

[0072] Fig.11 A schematic diagram of the structure of a data enhancement strategy search device provided by an embodiment of the present disclosure is shown;

[0073] Fig.12A schematic structural diagram of an electronic device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0074] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the embodiments of the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the embodiments of the present disclosure as detailed in the appended claims.

[0075] The terms used in the disclosed embodiments are only for the purpose of describing specific embodiments and are not intended to limit the disclosed embodiments. The singular forms of "a", "said" and "the" used in the disclosed embodiments and the appended claims are also intended to include plural forms unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.

[0076] It should be understood that although the terms first, second, third, etc. may be used to describe various information in the disclosed embodiments, these information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the disclosed embodiments, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0077] It can be understood that the description of various embodiments in the present disclosure focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.

[0078] In related technologies, data enhancement is mostly based on human prior knowledge. For example, when training a visual model, data enhancement operations are artificially added based on human prior knowledge. However, artificially adding data enhancement operations requires cumbersome adjustment of the parameters of the data enhancement operations. The adjustable parameters are reflected in the intensity of data enhancement, the probability of data enhancement operations being performed, and the combination of data enhancement operations. Generally speaking, people need to manually adjust the above parameters based on prior knowledge. Since the search space for these parameters is large, it is generally impossible to obtain the optimal set of parameters.

[0079] In addition, some automatic data augmentation algorithms that use models to search for data augmentation strategies, such as Autoaugment (automatic augmentation, AA), FastAA (fast automatic augmentation), FasterAA (faster automatic augmentation), and Randaugment (random augmentation), take a long time and require training and searching at the same time. Even the plug-and-play Randaugment algorithm requires manual adjustment of two hyperparameters: N (the number of categories for selecting data augmentation methods) and M (the intensity of image distortion for each data augmentation method), and the parameter values ​​need to be set based on the validation set, which results in high algorithm complexity.

[0080] The disclosed embodiment provides a data enhancement strategy search method, which can be applied to other scenarios that require data enhancement, such as image data enhancement, text data enhancement, or voice data enhancement. For example, when the method is applied to the scenario of image data enhancement, the target data enhancement strategy for image data enhancement is obtained by searching, and the target data enhancement strategy can be used to perform data enhancement on the existing image training data to obtain a large amount of image training data required for the training model, thereby improving the performance and effect of the model training.

[0081] Figure 1 A flow chart of a data enhancement strategy search method provided by an embodiment of the present disclosure is shown. Figure 1 As shown, the method may include the steps of:

[0082] 101: Obtain an enhancement operation set including at least two types of enhancement operations; wherein different types of enhancement operations correspond to different execution probability ranges;

[0083] 102: Obtain feature distribution of the training set to be enhanced;

[0084] 103: searching for a target enhancement operation according to the feature distribution of the training set and the strength parameter of each enhancement operation in the enhancement operation set;

[0085] 104: Searching for a target data enhancement strategy for enhancing the training set according to the feature distribution of the training set, a preset length, and strength parameters of each target enhancement operation; wherein the preset length is used to randomly combine different target enhancement operations in the same type of enhancement operations into a data enhancement strategy.

[0086] The data enhancement strategy search method provided in the embodiments of the present disclosure may be executed by an electronic device, which includes a terminal and / or a server. The server may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server.

[0087] In step 101, a plurality of enhancement operations corresponding to the data format of the training set to be enhanced are divided into at least two types of enhancement operations to form an enhancement operation set including at least two types of enhancement operations.

[0088] In the present disclosure, the training set to be enhanced may refer to an original training set that needs to be enhanced using an enhancement operation to obtain new data.

[0089] In some examples, when the data format of the training set to be enhanced is an image format, the enhancement operation is used to perform data enhancement on a sample image in the training set to be enhanced. For example, the sample image may be a document image.

[0090] For example, for the sample images in the image training set to be enhanced, the enhancement operation may include but is not limited to: at least two of scale transformation, tilt, flip, crop, rotate, deform, zoom, translate, pixel change, contrast change and brightness change. Flipping includes horizontal flipping and vertical flipping.

[0091] In some examples, when the data format of the training set to be enhanced is a speech format, the enhancement operation is used to perform data enhancement on the sample speech in the training set to be enhanced. For example, for the sample speech in the speech training set to be enhanced, the enhancement operation may include but is not limited to: at least two of volume enhancement, speed enhancement, and pitch enhancement.

[0092] In some examples, when the data format of the training set to be enhanced is a text format, the enhancement operation is used to perform data enhancement on the sample text in the training set to be enhanced. For example, for the sample text in the text training set to be enhanced, the enhancement operation may include but is not limited to: at least two of random replacement, random insertion, random exchange and random deletion.

[0093] In the present disclosure, each enhancement operation has attribute parameter values ​​such as execution probability, strength (or "execution degree"), etc. The execution probability of an enhancement operation indicates the possibility of executing the enhancement operation, and the probability value ranges from 0 to 1.

[0094] For example, taking an enhancement operation of an image as an example, the probability and strength of executing a certain enhancement operation are 0.8 and 20 degrees respectively, and the enhancement operation is rotation, which means that the probability of executing the rotation is 0.8 and the rotation angle is 20 degrees.

[0095] In some examples, the enhancement operation and its corresponding attribute parameter value may be stored in a triple form, for example, the triple form is: (enhancement operation, probability value, strength value).

[0096] In some examples, each enhancement operation may have a pre-statistical probability of execution and a corresponding intensity parameter.

[0097] For example, the execution probability and strength parameter of each enhancement operation may be obtained by using attribute statistics of the occurrence of each sample in the test set.

[0098] Taking the image enhancement operation as an example, the execution probability and corresponding intensity parameters of each sample in the test set with attributes including but not limited to: scale transformation, tilt, flip, cropping, rotation, deformation, translation, pixel change, contrast change and brightness change can be counted.

[0099] In the present disclosure, when the acquired enhancement operation set includes two types of enhancement operations, the two types of enhancement operations include: first type of enhancement operations and second type of enhancement operations, wherein the execution probability of each enhancement operation in the first type of enhancement operations is higher than the execution probability of all enhancement operations in the second type of enhancement operations.

[0100] Here, the first type of enhancement operation may also be referred to as a “strong data enhancement operation”; the second type of enhancement operation may also be referred to as a “weak data enhancement operation.” Exemplarily, the first type of enhancement operation may include conventional enhancement operations; the second type of enhancement operation may include unconventional enhancement operations.

[0101] In the present disclosure, when the acquired enhancement operation set includes three types of enhancement operations, the three types of enhancement operations include: first type of enhancement operations, second type of enhancement operations and third type of enhancement operations. The execution probability of each enhancement operation in the first type of enhancement operations is higher than the execution probability of all enhancement operations in the second type of enhancement operations, and the execution probability of each enhancement operation in the second type of enhancement operations is higher than the execution probability of all enhancement operations in the third type of enhancement operations.

[0102] In some examples, for multiple enhancement operations used to enhance the training set, the multiple enhancement operations can be divided into two or more categories of enhancement operations according to the execution probability of each enhancement operation. For example, using M-1 preset probabilities as the division criteria, multiple enhancement operations are divided into M categories of enhancement operations, and the execution probability ranges corresponding to different categories of enhancement operations are different. Wherein, M is a positive integer greater than 1. For example, the value of M is greater than or equal to 3, the execution probability of each enhancement operation in the m-1th category of enhancement operations is greater than or equal to the m-1th preset probability, the execution probability of each enhancement operation in the m-1th category of enhancement operations is less than the m-1th preset probability, and the execution probability of each enhancement operation in the m-1th category of enhancement operations is less than or equal to the m-1th preset probability, and the execution probability of each enhancement operation in the m-1th category of enhancement operations is less than or equal to the m-1th preset probability, wherein the m-1th preset probability is greater than the mth preset probability, m is a positive integer greater than 1, and m is less than or equal to M.

[0103] In some examples, in the above step 101, taking the example that the enhancement operation set includes two types of enhancement operations, a preset probability can be used as a classification standard, and the enhancement operations with execution probability greater than or equal to the preset probability can be classified into the first type of enhancement operations, and the enhancement operations with execution probability less than the preset probability can be classified into the second type of enhancement operations; the preset probability can be a value preset according to actual application needs. For example, the preset probability is set to 0.5 or 0.6.

[0104] In other examples, in the above step 101, taking the example that the enhancement operation set includes two categories of enhancement operations, multiple enhancement operations can be sorted according to the execution probability, and the enhancement operations ranked in the first N positions can be classified into the first category of enhancement operations, and the enhancement operations ranked after the Nth position can be classified into the second category of enhancement operations; the value of N can be pre-set according to the total number of enhancement operations, for example, the ratio of the value of N to the total number of enhancement operations is greater than or equal to a preset value, for example, the preset value is 50% or 60%.

[0105] In this embodiment, by obtaining an enhancement operation set including at least two categories of enhancement operations, after subsequently searching for the target enhancement operation from the enhancement operation set, a data enhancement strategy can be formed for different target enhancement operations in the same category of enhancement operations. Compared with directly forming a large number of data enhancement strategies for different target enhancement operations in the enhancement operation set, since there is no need to form data enhancement strategies for different target enhancement operations between different categories of enhancement operations, the search scope of the data enhancement strategy can be narrowed, thereby effectively reducing the search space of the data enhancement strategy, improving the search efficiency of the data enhancement strategy, and also greatly simplifying the search process of the data enhancement strategy.

[0106] In the above step 102, the feature distribution of the training set to be enhanced may be extracted through a neural network model.

[0107] In some examples, when the training set includes multiple sample images, the sample images may be document images or images containing natural scene text. A document image refers to a document in image format, and the document may be a plain text document or a non-plain text document. For example, the non-plain text document is a document with a table. The text content in an image containing natural scene text refers to text content existing in any natural scene, such as scene texts such as road signs, billboards, shopping mall signs, and product packaging. It is understandable that, based on different deep learning tasks, the sample image may also be an image containing a target object, such as a vehicle image, a person image, an animal image, or a plant image.

[0108] In some examples, the training set is used to train a neural network model. For example, the neural network model may be a multi-task learning model.

[0109] In the present disclosure, the feature distribution of a training set may be used to describe the spatial position distribution of the feature description of each training sample in the training set.

[0110] In some examples, the feature distribution of the training set may be obtained based on the feature description of each training sample in the sample set. For example, the feature distribution of the training set may be obtained based on the feature description of a specified dimension among multiple dimensions of each training sample in the sample set.

[0111] Here, feature descriptions of multiple dimensions of the training samples can be obtained through a neural network model, for example, by extracting feature descriptions of multiple dimensions of each sample image in the training set through a feature extraction network in a multi-task learning model.

[0112] It should be noted that the embodiment of the present disclosure does not specifically limit the execution order of step 102 and step 101 , and step 102 may be executed after step 101 , or before step 101 , or simultaneously with step 101 .

[0113] In the above step 103, the target enhancement operation may be searched for in an equal probability manner for all enhancement operations according to the feature distribution of the training set and the strength parameter of each enhancement operation in the enhancement operation set.

[0114] Exemplarily, for each enhancement operation, data enhancement can be performed on each sample in the training set according to the strength parameters of the enhancement operation, and a feature description of each sample after the enhancement operation is performed is obtained. Based on whether the feature description of each sample after the enhancement operation is performed conforms to the feature distribution of the training set, it is determined whether the enhancement operation is a target enhancement operation.

[0115] Here, the feature description of each sample in the training set after the target enhancement operation is performed can conform to the feature distribution of the training set before the enhancement.

[0116] Here, the feature description of each sample after the target enhancement operation is performed conforms to the feature distribution of the training set before enhancement, which may include: the spatial position distribution of the feature description of each sample after the target enhancement operation is at the edge of the feature distribution of the training set before enhancement.

[0117] In some examples, in the above step 103, for each enhancement operation in the enhancement operation set, data enhancement can be performed on each sample in the training set according to the strength parameter of the enhancement operation, and a feature description of each sample after performing the enhancement operation is obtained. Based on whether the feature description of each sample after performing the enhancement operation conforms to the feature distribution of the training set, it is determined whether the enhancement operation is a target enhancement operation.

[0118] For example, if the feature description of each sample after performing the enhancement operation conforms to the feature distribution of the training set before enhancement, the enhancement operation is determined to be the target enhancement operation; or, if the feature description of each sample after performing the enhancement operation does not conform to the feature distribution of the training set before enhancement, the enhancement operation is determined not to be the target enhancement operation.

[0119] Here, the feature description of each sample after performing the enhancement operation conforms to the feature distribution of the training set before enhancement, which may include: the position distribution of the feature description of each sample after performing the enhancement operation is at the edge of the feature distribution of the training set before enhancement.

[0120] The feature description of each sample after the enhancement operation is performed does not conform to the feature distribution of the training set, which may include: the position of the feature description of each sample after the enhancement operation is performed is not distributed at the edge of the feature distribution of the training set.

[0121] In some examples, in the above step 104, multiple data enhancement strategies can be generated according to the preset length and the strength parameters of each target enhancement operation, and the target data enhancement strategy for enhancing the training set can be searched among the multiple data enhancement strategies according to the feature distribution of the training set.

[0122] Here, multiple data enhancement strategies may include: multiple first data enhancement strategies and multiple second data enhancement strategies, wherein the multiple first data enhancement strategies include: data enhancement strategies obtained by randomly combining different target enhancement operations in the first category of enhancement operations with the preset length; and the multiple second data enhancement strategies include: data enhancement strategies obtained by randomly combining different target enhancement operations in the second category of enhancement operations with the preset length.

[0123] In some examples, the multiple data augmentation strategies also include a data augmentation strategy formed by a single target augmentation operation.

[0124] The preset length is used as the combined length of the target enhancement operations in the same type of enhancement operations. The preset length can be a value preset according to actual application needs. For example, the value of the preset length is 2 or 3.

[0125] For example, the preset length may be used to combine different target enhancement operations in the first type of enhancement operations into a data enhancement strategy; the preset length may be used to combine different target enhancement operations in the second type of enhancement operations into a data enhancement strategy.

[0126] Exemplarily, the target enhancement operations belonging to a certain category of enhancement operations (for example, the first category of enhancement operations) include: enhancement operation 1, enhancement operation 2, and enhancement operation 3. If the value of the preset length is set to 2, the different data enhancement strategies obtained by randomly combining the above enhancement operations according to the preset length may include: enhancement operation 1 and enhancement operation 2, enhancement operation 2 and enhancement operation 1, enhancement operation 1 and enhancement operation 3, enhancement operation 3 and enhancement operation 1, enhancement operation 2 and enhancement operation 3, and enhancement operation 3 and enhancement operation 2.

[0127] In the present disclosure, when any data enhancement strategy includes different enhancement operations that are executed in sequence, the execution order of the different enhancement operations is determined by the combination method of the data enhancement strategy.

[0128] For example, if the combination of a data enhancement strategy is: tilt and deformation, when executing the data enhancement strategy, the sample is first enhanced with the tilted intensity parameter, and then the tilted sample is enhanced with the deformed intensity parameter. If the combination of a data enhancement strategy is: deformation and tilt, when executing the data enhancement strategy, the sample is first enhanced with the deformed intensity parameter, and then the tilted intensity parameter is enhanced with the deformed sample.

[0129] In the above step 104, based on the feature distribution of the training set and the strength parameters of each data enhancement strategy, a target data enhancement strategy can be searched for so that the feature description of each sample in the training set after data enhancement matches the feature distribution of the training set before enhancement.

[0130] Exemplarily, for each data enhancement strategy, data enhancement can be performed on each sample in the training set according to the strength parameters of the data enhancement strategy, and a feature description of each sample after executing the data enhancement strategy is obtained. In addition, based on whether the feature description of each sample after executing the data enhancement strategy is consistent with the feature distribution of the training set before enhancement, it is determined whether the data enhancement strategy is the target data enhancement strategy.

[0131] For example, if the feature description of each sample after executing the data enhancement strategy is consistent with the feature distribution of the training set before enhancement, the data enhancement strategy is determined to be the target data enhancement strategy; if the feature description of each sample after executing the data enhancement strategy does not conform to the feature distribution of the training set before enhancement, the data enhancement strategy is determined not to be the target data enhancement strategy.

[0132] Here, the feature description of each sample after executing the data enhancement strategy conforms to the feature distribution of the training set, which may include: the position distribution of the feature description of each sample after executing the data enhancement strategy is at the edge of the feature distribution of the training set.

[0133] The feature description of each sample after executing the data enhancement strategy does not conform to the feature distribution of the training set, which may include: the position of the feature description of each sample after executing the data enhancement strategy is not distributed at the edge of the feature distribution of the training set.

[0134] In this embodiment, the target data enhancement strategy is used to perform data enhancement on the training set to be enhanced.

[0135] In one embodiment, the method may further include:

[0136] The training set is enhanced according to the target data enhancement strategy, and the model is retrained based on the training set enhanced by the target data enhancement strategy.

[0137] In some examples, enhancing the training set according to a target data augmentation strategy may include:

[0138] At least part of the training samples in the training set are enhanced according to the target data enhancement strategy. For example, part of the training samples randomly sampled from the training set are enhanced according to the target data enhancement strategy.

[0139] In some examples, retraining the model based on the training set enhanced by the target data enhancement strategy may include:

[0140] For each training sample enhanced by the target data enhancement strategy, the training sample is input into the neural network model trained in the training set for prediction to obtain the predicted label corresponding to the training sample; the loss value of this training is determined according to the predicted label corresponding to the training sample and the true label, and the parameters of the neural network model are retrained according to the loss value until the training end conditions are met.

[0141] For example, the training end condition may include: stopping training when the number of training times reaches a preset number of training times, or stopping training when the difference between the loss value of the model trained this time and the loss value of the model trained last time is less than a preset threshold.

[0142] In this embodiment, by enhancing the training set according to the target data enhancement strategy, and retraining the model according to the training set enhanced by the target data enhancement strategy, the amount of training data used for model training can be expanded, and the effect and generalization ability of neural network model training can be improved.

[0143] The disclosed embodiment provides a data enhancement strategy search method, which searches for a target enhancement operation according to the feature distribution of a training set to be enhanced and the strength parameter of each enhancement operation in an enhancement operation set. Since the enhancement operation set includes at least two types of enhancement operations, each enhancement operation in different types of enhancement operations can participate in the search process of the target enhancement operation, thereby ensuring the diversity of the enhancement operations. Moreover, since the target enhancement operation is determined in combination with the feature distribution of the training set and the strength parameter of each enhancement operation, the search effect for the target enhancement operation can be improved.

[0144] In addition, by searching for the target data enhancement strategy for enhancing the training set according to the characteristic distribution of the training set, the preset length, and the strength parameters of each target enhancement operation, since the target data enhancement strategy is searched from the data enhancement strategy of the random combination of different target enhancement operations in the same type of enhancement operation based on the preset length, the search range does not include the data enhancement strategy formed by the target enhancement operations of different types of enhancement operations, so in the data enhancement strategy search process, the data enhancement strategy search space is effectively reduced, which can improve the search efficiency of the data enhancement strategy. Moreover, the entire search process only needs to determine the execution probability of the enhancement operation and the preset length for the enhancement operation combination. Since fewer parameters need to be set, the search process of the data enhancement strategy is greatly simplified, and the computational complexity is reduced.

[0145] Exemplarily, when the data strategy search method provided in this embodiment is applied to image data enhancement, by obtaining an image enhancement operation set including at least two types of enhancement operations, and obtaining the feature distribution of the image training set to be enhanced, and searching for the target enhancement operation according to the feature distribution of the image training set and the strength parameters of each enhancement operation in the image enhancement operation set, searching for the target data enhancement strategy for enhancing the image training set according to the feature distribution of the image training set, the preset length, and the strength parameters of each target enhancement operation, it can greatly simplify the search process of the data enhancement strategy and reduce the computational complexity while improving the search effect of the data enhancement strategy. Based on this, using the target data enhancement strategy to perform image data enhancement on the image training set can help improve the image data enhancement effect, quickly obtain a large amount of image training data required for the training model, and thus improve the performance and effect of model training.

[0146] In one embodiment, Figure 2 As shown, in the above step 101, obtaining an enhancement operation set including at least two types of enhancement operations may include the steps of:

[0147] 201: Determine multiple enhancement operations and execution probability and strength parameters of each enhancement operation according to the difference in sample attributes between the training set and the unlabeled test set;

[0148] 202: Determine at least two types of enhancement operations that form an enhancement operation set according to a comparison result between the execution probability of each enhancement operation and a preset probability.

[0149] In some examples, the test samples in the test set and the training samples in the training set can both be images, and the sample attributes can be image attributes. The image attributes may include: geometric attributes and non-geometric attributes. For example, geometric attributes may include: image orientation, image shape and / or image size and other attributes. Non-geometric attributes may include: visual attributes such as color, texture, transparency, contrast, exposure and / or clarity and other attributes.

[0150] In some examples, the number of samples in the training set may be a preset number, and the number of samples in the test set may be greater than N times the number of samples in the training set. For example, N is 10 or other values.

[0151] In some examples, in the above step 201, for each training sample in the training set, according to the difference in sample attributes between the training sample and the multiple test samples in the test set, the enhancement operation required for enhancing the sample attribute value of the training sample to the sample attribute value of each of the test samples, as well as the execution probability and intensity parameter of the enhancement operation, can be determined. For example, a preset interface can be provided to receive the enhancement operation input by the user based on prior knowledge for the difference, as well as the execution probability and intensity parameter of the enhancement operation.

[0152] In some examples, in the above step 202, enhancement operations with execution probabilities greater than or equal to a preset probability may be classified into a first category of enhancement operations, and enhancement operations with execution probabilities less than the preset probability may be classified into a second category of enhancement operations.

[0153] In this embodiment, since the test set is usually real data, multiple enhancement operations and the execution probability and strength parameters of each enhancement operation are determined based on the differences in sample attributes between the test set and the training set. The enhancement operation set formed in this way can include enhancement operations that can enhance the training set to be close to the real data, thereby improving the availability of the enhancement operations; in addition, by comparing the execution probability of each enhancement operation with the preset probability, the enhancement operations are classified into at least two categories of enhancement operations. In this way, when it is subsequently determined that the target enhancement operation belongs to the second category of enhancement operations, the target enhancement operation can be directly combined with other target enhancement operations in the second category of enhancement operations to form a target data enhancement strategy for data enhancement on the training set, thereby greatly simplifying the search process of the data enhancement strategy and reducing the computational complexity.

[0154] In one embodiment, Figure 3 As shown, based on Figure 1, the step of obtaining the feature distribution of the training set to be enhanced may include the following steps:

[0155] 301: Obtain feature distribution of the training set according to a first loss function of a pre-trained model and feature descriptions extracted by the pre-trained model for each sample image in the training set; wherein the first loss function includes: a distance from a feature description outside a spherical surface of a sphere with a feature center of the feature descriptions of all sample images in the training set as the center of a sphere and a distance threshold as the radius to the spherical surface.

[0156] In some examples, the pre-trained model may be a neural network model pre-trained using the training set.

[0157] In some examples, the loss function may be a loss function based on Euclidean space, which may constrain the distance between feature points so that the learned features have as short an intra-class interval as possible.

[0158] In step 301, the feature distribution of the training set can be obtained by constraining the distance between the feature description of all sample images and the center of the feature distribution of the training set according to the distance contained in the loss function of the pre-training model.

[0159] The distance threshold is used to limit the distance between the feature descriptions of all sample images in the training set and the center of the feature distribution of the training set (ie, the feature center of the training set).

[0160] As an example, the distance threshold may be set to 1 or other suitable value.

[0161] As an example, the center of the feature distribution of the training set can be the average of the feature descriptions of all sample images.

[0162] As an example, the distance between the feature description of the sample image and the center of the feature distribution of the training set may be a norm distance, and the norm distance may be a 1-norm distance or a 2-norm distance, etc. For example, the 1-norm distance may be a Manhattan distance, and the 2-norm distance may be a Euclidean distance.

[0163] In this embodiment, the feature distribution of the training set is obtained by describing the features extracted from each sample image in the training set based on the loss function of the pre-training model and the feature extraction of the pre-training model. Since the distance threshold contained in the loss function of the pre-training model can constrain the distance between the sample features and the sample feature centers, the pre-training model can learn the differences between samples and improve the convergence of the model.

[0164] In one embodiment, the pre-trained model can be a multi-task learning model pre-trained via a training set; the multi-task learning model includes: a feature extraction network, a one-class head (one-class head) and a task head corresponding to the target task; the feature extraction network is respectively connected to the one-class head and the task head, wherein the feature extraction network is used to extract features from each sample image in the training set; the one-class head is used to extract feature descriptions of each sample image from the feature extraction results output by the feature extraction network; the loss function of the one-class head is the first loss function.

[0165] Here, the multi-task learning paradigm of the multi-task learning model allows the model to achieve the target task on the one hand, and on the other hand, it can also model the data through single-class learning.

[0166] In some examples, when the sample images in the training set are document images or images containing natural scene text, the multiple tasks of the multi-task learning model may include visual tasks related to text images.

[0167] For example, the target task among the multiple tasks of the multi-task learning model may include: at least one of the tasks related to text images, such as text detection, stroke detection, text erasure, and non-text element detection.

[0168] In some examples, a feature extraction network in a multi-task learning model is used to extract features from each sample image in a training set to obtain a multi-layer feature map of each sample image.

[0169] The feature extraction network may also be referred to as a backbone network, which can extract multiple layers of feature maps from an input image. Different network layers of the backbone network output feature maps of different resolutions, wherein the network layers of the backbone network can be reasonably set according to actual conditions.

[0170] For example, the backbone network may be a MobileNet, ResNet, Inception or Xception network. It is understandable that the structure of the backbone network is not specifically limited in the embodiments of the present disclosure, as long as the backbone network can extract multi-layer feature maps of the image.

[0171] Different target tasks of the multi-task learning model correspond to different task heads. For example, target detection corresponds to a target detection head, and text detection corresponds to a text detection head.

[0172] The loss function of the single classifier is a first loss function, that is, the loss function of the single classifier includes: the feature center of the feature description of all sample images in the training set is used as the center of the sphere, and the distance from the feature description outside the spherical surface to the spherical surface is used as the radius.

[0173] In this embodiment, since the distance threshold included in the loss function of the single classifier in the multi-task learning model can constrain the distance between the sample feature and the sample feature center, the feature distribution of the training set determined based on the distance threshold can be used to search for data enhancement strategies, so that the searched data enhancement strategies are suitable for the training of the multi-task learning model, so as to reduce the sample offset in the enhanced training set, thereby improving the generalization ability of the final model.

[0174] In one embodiment, Figure 4 As shown, in the above step 103, searching for the target enhancement operation according to the feature distribution of the training set and the strength parameter of each enhancement operation in the enhancement operation set may include the following steps:

[0175] 401: adjusting the execution probabilities of all enhancement operations in the enhancement operation set to the same probability;

[0176] 402: For each enhancement operation in the enhancement operation set, execute steps 402a to 402d.

[0177] 402a: Perform data enhancement on each sample image in the training set according to the strength parameter of the enhancement operation;

[0178] 402b: Obtain feature descriptions extracted by the pre-trained model for each sample image enhanced by the enhancement operation;

[0179] 402c: Obtain a first distance between the center of the feature distribution of the training set and the feature description of each sample image enhanced by the enhancement operation;

[0180] 402d: Determine whether the enhancement operation is a target enhancement operation according to the multiple first distances and the distance threshold.

[0181] In the above step 401 , the execution probabilities of all enhancement operations in the enhancement operation set may be adjusted to a same probability value between 0.5 and 1. For example, the probability value may be 0.5 or 1.

[0182] In the above step 402, for each enhancement operation in the enhancement operation set, after using the enhancement operation to perform data enhancement on each sample image in the training set, each enhanced sample image is input into the pre-training model for feature extraction to obtain a feature description of each enhanced sample image, and a first distance between the feature description of each enhanced sample image and the center of the feature distribution of the training set before enhancement (i.e., the feature center of the training set) is determined, and based on multiple first distances and a distance threshold, it is determined whether the enhancement operation is a target enhancement operation.

[0183] In some examples, the first distance may be a norm distance, and the norm distance may be a 1-norm distance or a 2-norm distance, etc. For example, the 1-norm distance may be a Manhattan distance, and the 2-norm distance may be a Euclidean distance.

[0184] In some examples, in step 402d, if there are no first distances greater than a preset number of times greater than N times the distance threshold among the plurality of first distances, it is determined that the enhancement operation is a target enhancement operation, and if there are first distances greater than a preset number of times greater than N times the distance threshold among the plurality of first distances, it is determined that the enhancement operation is not a target enhancement operation. For example, N may be a positive number greater than 1.5, and the preset number is greater than 1 / 3 of the total number of the first distances.

[0185] In this embodiment, when determining whether a certain enhancement operation is a target enhancement operation in the enhancement operation set, the first distance between the center of the feature distribution of the training set and the feature description of each sample image enhanced by the enhancement operation is obtained, and based on the first distance and the distance threshold in the pre-training model, it is determined whether the enhancement operation is the target enhancement operation. In this way, by using the distance threshold in the pre-training model to search and control the target enhancement operation, the feature description of each sample image in the training set enhanced by the target enhancement operation can be made to match the feature distribution of the training set before enhancement, and the sample offset of the enhanced training set can be suppressed, which is beneficial to improving the performance of the model trained with the enhanced training set.

[0186] In one embodiment, Figure 5 As shown, in the above step 402d, determining whether the enhancement operation is a target enhancement operation according to the multiple first distances and the distance thresholds may include the following steps:

[0187] 501: Determine the number of first distances whose absolute values ​​of differences from the distance threshold are smaller than a preset difference;

[0188] 502: When the ratio of the number of the determined first distances to the total number of the first distances exceeds a first preset ratio, determining the enhancement operation as a target enhancement operation;

[0189] 503: When the ratio of the determined number of first distances to the total number of first distances does not exceed a first preset ratio, determine that the enhancement operation is not a target enhancement operation.

[0190] In some examples, the preset difference value can be set according to actual application needs. For example, the preset difference value is 2 / 10 or 3 / 10 of the distance threshold.

[0191] In some examples, the first preset ratio can be set according to actual application needs. For example, the first preset ratio is a value greater than or equal to 7 / 10.

[0192] In the disclosed embodiment, by determining the number of first distances whose absolute value of the difference with the distance threshold is less than a preset difference, and searching for the target enhancement operation in combination with the first preset ratio, the position distribution of the feature description of each sample image in the training set enhanced by the target enhancement operation is made to be at the edge of the feature distribution of the training set before enhancement, while suppressing the sample shift in the enhanced training set and expanding the spatial distribution of the sample features, thereby further improving the performance of the model trained with the enhanced training set.

[0193] In one embodiment, the target enhancement operation includes: a first target enhancement operation belonging to a first category of enhancement operations and a second target enhancement operation belonging to a second category of enhancement operations; wherein the lower limit value of the execution probability range corresponding to the first category of enhancement operations is greater than the upper limit value of the execution probability range corresponding to the second category of enhancement operations.

[0194] In one embodiment, Figure 6 As shown, in the above step 104, searching for a target data enhancement strategy for enhancing the training set according to the feature distribution of the training set, the preset length, and the strength parameters of each target enhancement operation includes at least one of the following steps:

[0195] 601: Determine each data enhancement strategy in the first data enhancement set as a first target data enhancement strategy; wherein the first data enhancement set includes: data enhancement strategies obtained by performing a random combination of different first target enhancement operations of a preset length according to strength parameters of each first target enhancement operation;

[0196] 602: Searching for a second target data enhancement strategy in a second data enhancement set according to the feature distribution of the training set; wherein the second data enhancement set includes: a data enhancement strategy obtained by randomly combining different second target enhancement operations of a preset length according to the strength parameters of each second target enhancement operation.

[0197] Here, the first data enhancement set may include: at least one data enhancement strategy formed by different first target enhancement operations belonging to the first category of enhancement operations. The second data enhancement set may include: at least one data enhancement strategy formed by different second target enhancement operations belonging to the second category of enhancement operations.

[0198] In some examples, the target data enhancement strategy may include: at least one first target data enhancement strategy obtained by random combination of different first target enhancement operations of a preset length, and / or, at least one second target data enhancement strategy obtained by random combination of different second target enhancement operations of a preset length.

[0199] In other examples, the target data enhancement strategy may include: a third target data enhancement strategy formed by combining the first target data enhancement strategy and the second target data enhancement strategy.

[0200] In some examples, the first target data enhancement strategy may also include: a data enhancement strategy formed by a single target enhancement operation belonging to the first category of enhancement operations, and / or a data enhancement strategy formed by a single target enhancement operation belonging to the second category of enhancement operations.

[0201] In this embodiment, since the execution probability of the first type of enhancement operation is higher than that of the second type of enhancement operation, that is, compared with the second type of enhancement operation, the first type of enhancement operation is a more common enhancement operation, so after determining the target enhancement operation belonging to the first type of enhancement operation, the data enhancement strategy obtained by randomly combining different target enhancement operations in the first type of enhancement operation can be directly determined as the first target data enhancement strategy, thereby greatly simplifying the data enhancement strategy search process. And for the data enhancement strategy randomly combined with different target enhancement operations in the second type of enhancement operation, the second target data enhancement strategy is searched from the data enhancement strategy according to the feature distribution of the training set, so that in the search process of the second target data enhancement strategy, the interference of inappropriate enhancement operations can be effectively eliminated, the search space of the second target data enhancement strategy is reduced, and the search efficiency of the data enhancement strategy can be improved.

[0202] In one embodiment, Figure 7 As shown, in the above step 602, searching for the second target data enhancement strategy in the second data enhancement set according to the feature distribution of the training set may include the steps of:

[0203] 701: For each data enhancement strategy in the second data enhancement set, execute steps 701a to 701d.

[0204] 701a: Perform data enhancement on each sample image in the training set according to the strength parameter of the data enhancement strategy;

[0205] 701b: Obtain feature descriptions extracted by the pre-trained model for each sample image enhanced by the data enhancement strategy;

[0206] 701c: Obtain a second distance between the center of the feature distribution of the training set and the feature description of each sample image enhanced by the data enhancement strategy;

[0207] 701d: Determine whether the data enhancement strategy is the second target data enhancement strategy based on multiple second distances and distance thresholds.

[0208] In this embodiment, the strength parameter of any data enhancement strategy in the second data enhancement set includes: the strength parameter of the target enhancement operation combined into the data enhancement strategy.

[0209] In the above step 701, for each second data enhancement strategy, after performing data enhancement on each sample image in the training set using the second data enhancement strategy, each enhanced sample image is input into the pre-training model for feature extraction to obtain a feature description of each enhanced sample image, and a second distance between the feature description of each enhanced sample image and the center of the feature distribution of the training set before enhancement (i.e., the feature center of the training set) is determined, and based on multiple second distances and distance thresholds, it is determined whether the second data enhancement strategy is the second target data enhancement strategy.

[0210] In some examples, the second distance may be a norm distance, and the norm distance may be a 1-norm distance or a 2-norm distance, etc. For example, the 1-norm distance may be a Manhattan distance, and the 2-norm distance may be a Euclidean distance.

[0211] In some examples, in step 701d, if there are no second distances in the plurality of second distances that are greater than N times the distance threshold that exceed a preset number, it is determined that the enhancement operation is a target enhancement operation, and if there are second distances in the plurality of second distances that are greater than N times the distance threshold that exceed a preset number, it is determined that the enhancement operation is not a target enhancement operation. For example, N may be a positive number greater than 1.5, and the preset number is greater than 1 / 3 of the total number of the second distances.

[0212] In this embodiment, when determining whether a data enhancement strategy is the second target data enhancement strategy in the second data enhancement set, the second distance between the center of the feature distribution of the training set and the feature description of each sample image enhanced by the data enhancement strategy is obtained, and based on the second distance and the distance threshold in the pre-training model, it is determined whether the data enhancement strategy is the second target data enhancement strategy. In this way, the search control of the second target data enhancement strategy is performed using the distance threshold in the pre-training model, so that the feature description of each sample image in the training set enhanced by the second target data enhancement strategy can be consistent with the feature distribution of the training set before enhancement, and the sample offset of the enhanced training set can be suppressed, which is beneficial to improving the performance of the model trained with the enhanced training set.

[0213] In one embodiment, Figure 8 As shown, in the above step 702d, determining whether the data enhancement strategy is the second target data enhancement strategy according to multiple second distances and distance thresholds may include the following steps:

[0214] 801: Determine the number of second distances whose absolute values ​​of differences from the distance threshold are smaller than a preset difference;

[0215] 802: When the ratio of the number of determined second distances to the total number of second distances exceeds a second preset ratio, determining the data enhancement strategy to be a second target data enhancement strategy;

[0216] 803: When the ratio of the determined number of second distances to the total number of second distances does not exceed a second preset ratio, determine that the data enhancement strategy is not the second target data enhancement strategy.

[0217] In some examples, the preset difference value can be set according to actual application needs. For example, the preset difference value is 2 / 10 or 3 / 10 of the distance threshold.

[0218] In some examples, the second preset ratio can be set according to actual application needs. Here, the second preset ratio can be the same as or different from the first preset ratio. For example, the second preset ratio is a value greater than or equal to 7 / 10.

[0219] In the disclosed embodiment, by determining the number of second distances whose absolute value of the difference with the distance threshold is less than a preset difference, and searching for a second target data enhancement strategy in combination with a second preset ratio, the position distribution of the feature description of each sample image in the training set enhanced by the second target data enhancement strategy is made to be at the edge of the feature distribution of the training set before enhancement, while suppressing the sample shift in the enhanced training set and expanding the spatial distribution of the sample features, thereby further improving the performance of the model trained using the enhanced training set.

[0220] In one embodiment, Fig. 9 As shown, based on Figure 3 , the method may further comprise the steps of:

[0221] 105: Fine-tune the pre-trained model according to the training set enhanced by the target data enhancement strategy and the label data of the training set;

[0222] 106: Based on the fine-tuned pre-trained model, repeat steps 102 to 105 until the training end condition is met.

[0223] In this embodiment, for any one of the sample images in the training set, the true label corresponding to the sample image after being enhanced by the target data enhancement strategy is the same as the true label corresponding to the sample image before the enhancement.

[0224] In step 105, each sample image enhanced by the target data enhancement strategy may be used as a model input, and the label corresponding to each sample image may be used as a training label to fine-tune the parameters of the pre-trained model.

[0225] Here, the target data enhancement strategy may include: at least one first target data enhancement strategy obtained by random combination of different first target enhancement operations of preset length, and / or, at least one second target data enhancement strategy obtained by random combination of different second target enhancement operations of preset length.

[0226] In other examples, the target data enhancement strategy may include: a third target data enhancement strategy formed by combining the first target data enhancement strategy and the second target data enhancement strategy.

[0227] In some examples, fine-tuning the pre-trained model based on the enhanced training set and the label data of the training set may include:

[0228] For each sample image enhanced by the target data enhancement strategy, the sample image is input into the feature extraction network in the pre-trained model for feature extraction, and the extracted multi-layer feature map is predicted by the detection network corresponding to the target task in the pre-trained model to obtain the predicted label corresponding to the sample image;

[0229] The loss value of this training iteration is determined according to the predicted label and the true label corresponding to the sample image, and the parameters of the pre-trained model are slightly adjusted according to the loss value.

[0230] In step 106, based on the fine-tuned pre-trained model, steps 102 to 105 may be repeatedly performed until the training is stopped when the number of training times reaches a preset number of training times, thereby obtaining a trained pre-trained model.

[0231] In this embodiment, the pre-trained model is fine-tuned according to the enhanced training set and the label data of the training set, and the target data enhancement strategy is searched based on the fine-tuned pre-trained model. Then, the pre-trained model is iteratively trained on the training set enhanced by the target data enhancement strategy, which can further improve the generalization ability of the model.

[0232] It should be noted that the method provided in the embodiments of the present disclosure may be executed alone or together with some methods in the embodiments of the present disclosure or some methods in related technologies.

[0233] In order to further explain any embodiments of the present disclosure, some specific embodiments are provided below.

[0234] The disclosed embodiment provides a data enhancement strategy search method, which can simplify the parameter search process of the data enhancement algorithm. The method can be applied to image (e.g., document image) data enhancement, and expand the training data set through data enhancement, thereby improving the generalization ability of the deep learning model.

[0235] The data enhancement strategy search method provided by the embodiment of the present disclosure includes the following steps:

[0236] 1. Classification of data enhancement operations;

[0237] 2. Visual task learning and data modeling under the multi-task paradigm;

[0238] 3. Data enhancement strategy search;

[0239] 4. Model retraining.

[0240] In step 1 above, data augmentation operations are classified based on prior knowledge of the dataset.

[0241] In this step, the difference between the test set data and the training set data collected in advance can be compared, and the data enhancement operation can be classified using the difference.

[0242] 1.1: Determine data enhancement operations: For commonly occurring data attributes, it is necessary to determine the enhancement operations that implement the data attributes. Such operations include but are not limited to: scale transformation, tilt, flip, blur, brightness change, etc., and set the probability and intensity of execution for data enhancement.

[0243] 1.2: Classification of data augmentation operations: For common data attributes in the test set, the corresponding data augmentation operations are called strong data augmentation operations. For example, scale change and flipping are usually set as strong operations. Conversely, uncommon ones are called weak data augmentation operations, such as tilting.

[0244] In step 2 above, the multi-task paradigm enables the model to perform the target task while also modeling the distribution of the data.

[0245] In this step, the backbone network in the original visual task can be recorded as B, and the task-related head can be recorded as H. The loss function corresponding to the original visual task is recorded as L1. Based on the backbone network and the task-related head, a one-class head is added to the network for one-class learning.

[0246] like Fig.10a As shown, the backbone network is connected to the task-related head and the single classification head respectively.

[0247] In the learning part of the single classification head, a loss function (i.e., the first loss function in the aforementioned embodiment) can be set so that the distance between the feature descriptions corresponding to all samples in the training set and the feature centers of all samples is less than a distance threshold (e.g., the distance threshold can be 1). The distance can be set to a norm distance, such as 1-norm distance, 2-norm distance, etc. The feature center can be the average value of the feature descriptions of all samples, but the present embodiment is not limited thereto.

[0248] Reference Fig.10b As shown, Fig.10b The left figure in shows an initial feature distribution of a training set, where the average value of the feature descriptions corresponding to all samples in the training set is different from the initial feature center of the training set.

[0249] Fig.10b The right figure in the figure shows the feature distribution of a training set under the constraint of the loss function of a one-class classification head, where the average value of the feature descriptions corresponding to all samples in the training set coincides with the feature center of the training set, and the distance between the feature descriptions corresponding to all samples in the training set and the feature center of the training set is less than the distance threshold.

[0250] In the above step 3, based on the categories in step 1 and the data distribution after modeling in step 2, a data enhancement strategy search is performed to search for a data enhancement strategy that meets the data distribution.

[0251] The execution probabilities of all enhancement operations are unified to the same probability P, and a strategy search is performed. The benchmark for judging an acceptable combination of data enhancement strength and data enhancement operations is that after data enhancement is applied to the data set, the features of the data are distributed on the edge of the data, that is, the distance between the feature description of the data and the feature center is near a distance threshold (the distance threshold can be, for example, 1).

[0252] In some examples, step 3 above may include the steps of:

[0253] 3.1: Determine the intensity of the data augmentation operation.

[0254] For any data augmentation operation, when the execution probability is uniformly P, a data augmentation operation with an intensity of M is performed on each sample in the training set, and then sent to the backbone network B to obtain a one-class feature description through the one-class classification head.

[0255] The above calculation is performed for each sample in the data set to obtain the feature description of each sample. If the distance between these feature descriptions and the feature center is near the distance threshold, the current data enhancement operation intensity is accepted, otherwise it is discarded.

[0256] 3.2: Determine how to combine data augmentation operations.

[0257] For strong data augmentation operations, it is only necessary to search for the strength of the data augmentation operations, and there is no need to search for the combination of data augmentation operations.

[0258] The weak data enhancement operations obtained above and their corresponding accepted strengths are randomly combined with a length of n, and then the current data enhancement operation is applied to the entire data set. After obtaining the feature description, if the distance between these feature descriptions and the feature center is near the distance threshold, then the current data enhancement operation combination is accepted, otherwise it is discarded.

[0259] Reference Fig.10c , shows a feature distribution of an enhanced training set, wherein the black solid circle represents the position distribution of the feature description of the sample image in the enhanced training set, and the gray solid circle represents the position distribution of the feature description of the training set before enhancement. If the difference between the distance between the feature description of the sample image in the enhanced training set and the feature center of the training set before enhancement and the above distance threshold is less than a preset value, then the current enhancement operation combination is accepted, otherwise it is discarded.

[0260] In the above step 4, the model is retrained again based on the data enhancement strategy obtained through the search.

[0261] Exemplarily, the model is used to process pure vision tasks, all pure vision tasks are, for example, tasks related to document images, wherein tasks related to document images may include: text detection, stroke detection, text erasure, non-text element detection, and the like.

[0262] In some examples, step 4 above may include:

[0263] 4.1: Based on the strong data augmentation operation combination of length n and the weak data augmentation operation combination of length n in step 3.2, fine-tune the model obtained in step 2 and update the one-class learning branch at the same time

[0264] 4.2: For the fine-tuned model, repeat steps 2, 3, and 4.1 for a preset number of iterations (for example, 2 or 3 rounds) to obtain the trained model.

[0265] The technical solution provided by the present disclosure has at least the following beneficial effects:

[0266] 1) The entire algorithm is highly interpretable and can be understood as improving the generalization ability of the model step by step until it reaches the upper limit of the generalization ability of the model on the current data.

[0267] 2) High efficiency. Based on the training-search-fine-tuning model, only inference code is used in the search phase, which can be achieved with a larger batch size.

[0268] 3) There are few parameters to set. You only need to determine the probability of execution and the data enhancement combination method, and classify the data enhancement operations based on prior knowledge.

[0269] Fig.11 A schematic diagram of the structure of a data enhancement strategy search device provided by an embodiment of the present disclosure is shown. Fig.11 As shown, the data enhancement strategy search device 100 may include:

[0270] A first acquisition module 110 is used to acquire an enhancement operation set including at least two types of enhancement operations; wherein different types of enhancement operations correspond to different execution probability ranges;

[0271] A second acquisition module 120, for feature distribution of a training set to be enhanced;

[0272] A first search module 130, configured to search for a target enhancement operation according to the feature distribution of the training set and the strength parameter of each enhancement operation in the enhancement operation set;

[0273] The second search module 140 is used to search for a target data enhancement strategy for enhancing the training set based on the feature distribution of the training set, a preset length, and the strength parameters of each target enhancement operation; wherein the preset length is used to randomly combine different target enhancement operations in the same type of enhancement operations into a data enhancement strategy.

[0274] In one embodiment, the first acquisition module 110 is used to:

[0275] Determining a plurality of enhancement operations and an execution probability and a strength parameter of each enhancement operation according to differences in sample attributes between the training set and the unlabeled test set;

[0276] At least two types of enhancement operations forming the enhancement operation set are determined according to a comparison result between the execution probability of each enhancement operation and a preset probability.

[0277] In one embodiment, the second acquisition module 120 is used to:

[0278] The feature distribution of the training set is obtained according to the first loss function of the pre-trained model and the feature description extracted by the pre-trained model for each sample image in the training set; wherein the first loss function includes: the distance from the feature description outside the spherical surface of the sphere to the spherical surface with the feature center of the feature description of all sample images in the training set as the center of the sphere and the distance threshold as the radius.

[0279] In one embodiment, the first search module 130 includes:

[0280] An adjusting unit, configured to adjust the execution probabilities of all enhanced operations in the enhanced operation set to the same probability;

[0281] The first search unit is configured to perform the following operations for each enhancement operation in the enhancement operation set:

[0282] According to the intensity parameter of the enhancement operation, data enhancement is performed on each sample image in the training set respectively;

[0283] Obtaining a feature description extracted by the pre-trained model for each of the sample images enhanced by the enhancement operation;

[0284] Obtaining a first distance between a center of a feature distribution of the training set and a feature description of each of the sample images enhanced by the enhancement operation;

[0285] It is determined whether the enhancement operation is the target enhancement operation according to the plurality of first distances and the distance threshold.

[0286] In one embodiment, the first search unit is used to:

[0287] Determine the number of first distances whose absolute values ​​of differences from the distance threshold are smaller than a preset difference;

[0288] When the ratio of the determined number of the first distances to the total number of the first distances exceeds a first preset ratio, determining the enhancement operation as the target enhancement operation;

[0289] When the ratio of the determined number of the first distances to the total number of the first distances does not exceed the first preset ratio, it is determined that the enhancement operation is not the target enhancement operation.

[0290] In one embodiment, the target enhancement operation includes: a first target enhancement operation belonging to a first category of enhancement operations and a second target enhancement operation belonging to a second category of enhancement operations; wherein the lower limit value of the execution probability range corresponding to the first category of enhancement operations is greater than the upper limit value of the execution probability range corresponding to the second category of enhancement operations.

[0291] In one embodiment, the second search module 140 includes:

[0292] A determination unit, configured to determine each data enhancement strategy in the first data enhancement strategy set as a first target data enhancement strategy; wherein the first data enhancement set includes: data enhancement strategies obtained by performing a random combination of the preset length on different first target enhancement operations according to the strength parameters of each first target enhancement operation;

[0293] A second search unit is used to search for a second target data enhancement strategy in a second data enhancement set according to the feature distribution of the training set; wherein the second data enhancement set includes: a data enhancement strategy obtained by randomly combining different second target enhancement operations of the preset length according to the strength parameters of each second target enhancement operation.

[0294] In one embodiment, the second search unit is configured to perform the following operations for each data enhancement strategy in the second data enhancement set:

[0295] According to the strength parameter of the data enhancement strategy, data enhancement is performed on each sample image in the training set;

[0296] Obtaining a feature description extracted by the pre-trained model for each of the sample images enhanced by the data enhancement strategy;

[0297] Obtaining a second distance between the center of the feature distribution of the training set and the feature description of each of the sample images enhanced by the data enhancement strategy;

[0298] According to the plurality of second distances and the distance threshold, determine whether the data enhancement strategy is the second target data enhancement strategy.

[0299] In one embodiment, the second search unit is used to:

[0300] Determine the number of second distances whose absolute values ​​of differences from the distance threshold are smaller than a preset difference;

[0301] When the ratio of the determined number of the second distances to the total number of the second distances exceeds a second preset ratio, determining that the data enhancement strategy is the second target data enhancement strategy;

[0302] When the ratio of the determined number of the second distances to the total number of the second distances does not exceed the second preset ratio, it is determined that the data enhancement strategy is not the second target data enhancement strategy.

[0303] In one embodiment, the apparatus further comprises:

[0304] A fine-tuning module, used to fine-tune the pre-trained model according to the training set enhanced by the target data enhancement strategy and the label data of the training set;

[0305] An iteration module is used to repeat, based on the fine-tuned pre-trained model, the steps of the second acquisition module executing the step of acquiring the feature distribution, the first search module executing the step of searching the target enhancement operation, the second search module executing the step of searching the target data enhancement strategy, and the fine-tuning module executing the step of fine-tuning the pre-trained model until the training end condition is reached.

[0306] In one embodiment, the pre-trained model is: a multi-task learning model pre-trained by the training set; the multi-task learning model includes: a feature extraction network, a single classification head and a task head corresponding to the target task; the feature extraction network is respectively connected to the task head and the single classification head;

[0307] Wherein, the feature extraction network is used to extract features from each sample image in the training set;

[0308] The single classification head is used to extract the feature description of each sample image from the feature extraction result output by the feature extraction network; the loss function of the single classification head is the first loss function.

[0309] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0310] An embodiment of the present disclosure also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the data enhancement strategy search method provided by any embodiment of the present disclosure is implemented.

[0311] The memory may include various types of storage media, which are non-transitory computer storage media that can continue to memorize information stored thereon after the communication device loses power.

[0312] The processor may be connected to the memory via a bus or the like, and is used to read an executable program stored in the memory, for example, to implement the data enhancement strategy search method provided in any embodiment of the present disclosure.

[0313] An embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium stores an executable program, wherein when the executable program is executed by a processor, the data enhancement strategy search method provided by any embodiment of the present disclosure is implemented.

[0314] like Fig.12 FIG. 2 is a diagram showing a structure of an electronic device 300 according to an exemplary embodiment. For example, the electronic device 300 may be provided as a server. Fig.12 The electronic device 300 includes a processing component 310, which further includes one or more processors, and a memory resource represented by a memory 320 for storing instructions executable by the processing component 310, such as an application. The application stored in the memory 320 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 310 is configured to execute instructions to perform the data enhancement strategy search method in the above-mentioned embodiment.

[0315] The electronic device 300 may further include a power supply component 330 configured to perform power management of the electronic device 300, a wired or wireless network interface 340 configured to connect the electronic device 300 to a network, and an input / output (I / O) interface 350. The electronic device 300 may operate based on an operating system stored in the memory 320, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™ or the like.

[0316] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.

[0317] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A data enhancement strategy search method, characterized in that: The method comprises: Acquire an enhancement operation set including at least two types of enhancement operations; wherein different types of enhancement operations correspond to different execution probability ranges; Obtaining the feature distribution of the training set to be enhanced; Searching for a target enhancement operation according to the feature distribution of the training set and the strength parameter of each enhancement operation in the enhancement operation set; According to the feature distribution of the training set, the preset length and the strength parameters of each target enhancement operation, a target data enhancement strategy for enhancing the training set is searched; wherein the preset length is used to randomly combine different target enhancement operations in the same type of enhancement operations into a data enhancement strategy.

2. The method according to claim 1, characterized in that: The obtaining includes an enhancement operation set containing at least two types of enhancement operations, including: Determining a plurality of enhancement operations and an execution probability and a strength parameter of each enhancement operation according to differences in sample attributes between the training set and the unlabeled test set; At least two types of enhancement operations forming the enhancement operation set are determined according to a comparison result between the execution probability of each enhancement operation and a preset probability.

3. The method according to claim 1, characterized in that: The obtaining of the feature distribution of the training set to be enhanced includes: Acquire a feature distribution of the training set according to a first loss function of the pre-training model and a feature description extracted by the pre-training model from each sample image in the training set; The first loss function includes: taking the feature center of the feature description of all sample images in the training set as the center of a sphere and taking the distance threshold as the radius, and the distance from the feature description outside the spherical surface to the spherical surface.

4. The method according to claim 3, characterized in that The step of searching for a target enhancement operation according to the feature distribution of the training set and the strength parameter of each enhancement operation in the enhancement operation set includes: Adjusting the execution probabilities of all enhancement operations in the enhancement operation set to the same probability; For each enhancement operation in the enhancement operation set, perform the following operations: According to the intensity parameter of the enhancement operation, data enhancement is performed on each sample image in the training set respectively; Obtaining a feature description extracted by the pre-trained model for each of the sample images enhanced by the enhancement operation; Obtaining a first distance between a center of a feature distribution of the training set and a feature description of each of the sample images enhanced by the enhancement operation; It is determined whether the enhancement operation is the target enhancement operation according to the plurality of first distances and the distance threshold.

5. The method according to claim 4, characterized in that The determining, according to the plurality of first distances and the distance threshold, whether the enhancement operation is the target enhancement operation comprises: Determine the number of first distances whose absolute values ​​of differences from the distance threshold are smaller than a preset difference; When the ratio of the determined number of the first distances to the total number of the first distances exceeds a first preset ratio, determining the enhancement operation as the target enhancement operation; When the ratio of the determined number of the first distances to the total number of the first distances does not exceed the first preset ratio, it is determined that the enhancement operation is not the target enhancement operation.

6. The method according to claim 3, characterized in that The target enhancement operation includes: a first target enhancement operation belonging to a first type of enhancement operation and a second target enhancement operation belonging to a second type of enhancement operation; wherein a lower limit value of an execution probability range corresponding to the first type of enhancement operation is greater than an upper limit value of an execution probability range corresponding to the second type of enhancement operation; The step of searching for a target data enhancement strategy for enhancing the training set according to the feature distribution of the training set, the preset length, and the strength parameters of each target enhancement operation comprises: Determine each data enhancement strategy in the first data enhancement set as a first target data enhancement strategy; wherein the first data enhancement set includes: data enhancement strategies obtained by performing a random combination of the preset length on different first target enhancement operations according to the strength parameters of each first target enhancement operation; and / or, According to the feature distribution of the training set, a second target data enhancement strategy is searched in the second data enhancement set; wherein the second data enhancement set includes: a data enhancement strategy obtained by randomly combining different second target enhancement operations of the preset length according to the strength parameters of each second target enhancement operation.

7. The method according to claim 6, characterized in that The step of searching for a second target data enhancement strategy in a second data enhancement set according to the feature distribution of the training set includes: For each data enhancement strategy in the second data enhancement set, perform the following operations: According to the strength parameter of the data enhancement strategy, data enhancement is performed on each sample image in the training set; Obtaining a feature description extracted by the pre-trained model for each of the sample images enhanced by the data enhancement strategy; Obtaining a second distance between the center of the feature distribution of the training set and the feature description of each of the sample images enhanced by the data enhancement strategy; According to the plurality of second distances and the distance threshold, determine whether the data enhancement strategy is the second target data enhancement strategy.

8. The method according to claim 7, characterized in that The determining, according to the plurality of second distances and the distance threshold, whether the data enhancement strategy is the second target data enhancement strategy comprises: Determine the number of second distances whose absolute values ​​of differences from the distance threshold are smaller than a preset difference; When the ratio of the determined number of the second distances to the total number of the second distances exceeds a second preset ratio, determining that the data enhancement strategy is the second target data enhancement strategy; When the ratio of the determined number of the second distances to the total number of the second distances does not exceed the second preset ratio, it is determined that the data enhancement strategy is not the second target data enhancement strategy.

9. The method according to claim 3, characterized in that: The method further comprises: Fine-tuning the pre-trained model according to the training set enhanced by the target data enhancement strategy and the label data of the training set; Based on the fine-tuned pre-trained model, the steps of obtaining the feature distribution, searching for the target enhancement operation, searching for the target data enhancement strategy, and fine-tuning the pre-trained model are repeated until the training end condition is met.

10. A data enhancement strategy search device, characterized in that: The device comprises: A first acquisition module is used to acquire an enhancement operation set including at least two types of enhancement operations; wherein different types of enhancement operations correspond to different execution probability ranges; A second acquisition module is used to obtain the feature distribution of the training set to be enhanced; A first search module, configured to search for a target enhancement operation according to a feature distribution of the training set and a strength parameter of each enhancement operation in the enhancement operation set; The second search module is used to search for a target data enhancement strategy for enhancing the training set based on the feature distribution of the training set, a preset length, and the strength parameters of each target enhancement operation; wherein the preset length is used to randomly combine different target enhancement operations in the same type of enhancement operations into a data enhancement strategy.

Images