A data augmentation method, application and device for small sample oracle bone inscription recognition

Abstract

The present invention provides a data augmentation method, application and device for small sample oracle bone inscription recognition, characterized in that it includes the following steps: serializing large sample data and small sample data to obtain serialized large sample data and serialized small sample data; erasing the serialized large sample data to obtain erased large sample data and complete large sample data, encoding the erased large sample data to obtain encoded erased large sample data; inputting the encoded erased large sample data into a feature extraction network to obtain deep features; using a prediction network to predict the erased large sample data to obtain predicted large sample data; training and updating a deep learning network to obtain an augmenter; randomly erasing the serialized small sample data to obtain erased small sample data, and encoding the erased small sample data to obtain encoded erased small sample data; inputting the encoded erased small sample data into an augmenter to predict to obtain predicted small sample data, and using the predicted small sample data as an augmented sample.

Description

Technical Field

[0001] The present invention belongs to the technical field of data recognition, and in particular relates to a data augmentation method, application and device for small sample oracle bone inscription recognition. Background Art

[0002] Oracle bone script, also known as "oracle bone inscriptions", "Yinxu characters" or "tortoise shell and animal bone script", is the oldest known Chinese writing. Like other ancient character recognition, oracle bone recognition also faces the problem of insufficient and unbalanced data. The recognition of oracle bone inscriptions with extremely limited sample numbers naturally belongs to a small sample learning task, and previous oracle bone classification research has focused on the recognition of categories with larger sample numbers. Due to its long history, the interpretation of oracle bone inscriptions is very difficult. Only about 2,000 of the more than 4,500 characters that have been discovered have been successfully interpreted. Therefore, for the oracle bone recognition task, it is not realistic to obtain a large amount of labeled data, which cannot meet the standard small sample recognition task's demand for a large amount of labeled source data.

[0003] In addition, the large number of oracle bone inscriptions that have not yet been deciphered also have their value for oracle bone inscription recognition. The oracle bone inscription recognition task can be assisted by collecting unlabeled oracle bone inscriptions and other ancient Chinese characters. However, unlabeled source data cannot meet the training requirements of meta-learning models and metric-based models commonly used in small sample learning, and can only rely on self-supervised learning and data augmentation. The traditional geometric augmentation method is designed for bitmap images. This method performs similar global transformations on all samples. Considering that the stroke order and general shape of Chinese characters are very important for human literacy, the geometric augmentation based on the bitmap image format does not consider the overall and local shape information of Chinese characters, that is, it does not take into account the uniqueness and diversity of Chinese character samples, so that the final trained model for oracle bone inscription recognition cannot have high accuracy. Summary of the invention

[0004] In order to solve the above problems, a method, application and device for data augmentation is provided, which generates a large number of high-quality augmented samples similar to the original image and diverse in the case of only a large amount of unlabeled source data and a small number of labeled samples through self-supervised training and erasure-prediction, so as to be applied to the task of oracle bone inscription recognition. The present invention adopts the following technical solutions:

[0005] The present invention provides a data augmentation method for small sample oracle bone inscription recognition, which uses large sample data containing Chinese characters and oracle bone inscriptions without annotations and small sample data of oracle bone inscriptions with annotations to perform data augmentation to obtain augmented samples. The method is characterized in that it includes the following steps: step S1, serializing large sample data in bitmap format and small sample data using a predetermined serialization algorithm to obtain large sample data in point sequence format composed of coordinate data and state data and small sample data in point sequence format, and setting the large sample data in point sequence format as serialized large sample data, and setting the small sample data in point sequence format as serialized small sample data; step S2, erasing points in the serialized large sample data according to a predetermined erasing ratio, setting the serialized large sample data of the erased points as erased large sample data, setting the serialized large sample data of the unerased points as complete large sample data, and encoding the erased large sample data through a predetermined encoding network to obtain encoded erased large sample data; step S3, performing feature extraction on the encoded erased large sample data using a predetermined feature extraction network The deep features are extracted by the feature extraction; step S4, based on the deep features and using the predetermined prediction network, the erased large sample data is predicted to obtain the predicted large sample data; step S5, based on the complete large sample data and the predicted large sample data, a loss value is calculated by a predetermined loss function, and based on the loss value, a deep learning network composed of an encoding network, a feature extraction network and a prediction network is trained and updated until the deep learning network converges to obtain a trained deep learning network as an augmentor; step S6, a plurality of random erasing ratios are generated according to a predetermined erasing ratio generation method, the points of the serialized small sample data are erased according to the random erasing ratio, the serialized small sample data of the erased points are set as the erased small sample data, the serialized small sample data of the non-erased points are set as the complete small sample data, and the erased small sample data is encoded through the encoding network to obtain the encoded erased small sample data; step S7, the encoded erased small sample data is input into the augmentor to predict the erased small sample data to obtain the predicted small sample data, and the predicted small sample data is used as the augmented sample.

[0006] A data augmentation method for small-sample oracle bone inscription recognition provided by the present invention may further have the following technical features. In step S2-1, the points included in each sample in the serialized large-sample data are divided into three categories of serialized large-sample sub-data according to the status data; in step S2-2, the points in the serialized large-sample sub-data are erased in sequence according to the erasure ratio, and finally the serialized large-sample data of the erased points is set as the erased large-sample data, and the serialized large-sample data of the non-erased points is set as the complete large-sample data; in step S2-3, the coordinate data in the erased large-sample data is embedded and encoded using an encoding network, and the status data is encoded using a predetermined encoding method, so as to obtain the encoded erased large-sample data, where the erasure ratio is 15%.

[0007] A data augmentation method for small-sample oracle bone inscription recognition provided by the present invention may further have the following technical features. In step S3-1, the encoded erased large-sample data is converted into a retrieval vector, a key vector, and a value vector using a feature extraction network; in step S3-2, the retrieval vector, the key vector, and the value vector are processed using the feature extraction network to obtain the attention-weighted feature as the attention feature; in step S3-3, the attention feature is processed using the feature extraction network to obtain the deep feature with the same dimension as the input feature, where the feature extraction network is a deep neural network based on bidirectional encoding representation and self-attention mechanism.

[0008] A data augmentation method for small-sample oracle bone inscription recognition provided by the present invention may further have the following technical features. In step S6, it includes the following sub-steps: in step S6-1, the points included in each sample in the serialized small-sample data are divided into three categories of serialized small-sample sub-data according to the status data; in step S6-2, multiple random erasure ratios are generated using the erasure ratio generation method, and the points in each category of serialized small-sample sub-data are erased according to the randomly selected random erasure ratio, and the serialized small-sample data of the erased points is set as the erased small-sample data, and the serialized small-sample data of the non-erased points is set as the complete small-sample data; in step S6-3, the coordinate data in the erased small-sample data is embedded and encoded using an encoding network, and the status data in the erased small-sample data is encoded using a predetermined encoding method, so as to obtain the encoded erased small-sample data.

[0009] A data augmentation method for small-sample oracle bone inscription recognition provided by the present invention may further have the following technical features. The erasure ratio generation method is to uniformly sample from the interval [a, b], where 0 < a, b ≤ 1, with an interval of k to obtain different erasure ratios

[0010] A data augmentation method for small sample oracle bone inscription recognition provided by the present invention may also have such technical features, wherein the encoding network is a neural network composed of three fully connected layers, and the encoding method is a dictionary encoding method.

[0011] A data augmentation method for small sample oracle bone inscription recognition provided by the present invention may also have such a technical feature, wherein in step S5, the loss function for the complete large sample data and the coordinate data in the predicted large sample data is the L1 norm loss function, and in step S5, the loss function for the complete large sample data and the state data in the predicted large sample data is the cross entropy loss function.

[0012] A data augmentation method for small sample oracle bone inscription recognition provided by the present invention may also have such a technical feature, wherein in step S4, the prediction network is a neural network composed of four fully connected layers.

[0013] The present invention also provides an application of a data augmentation method for small sample oracle bone inscription recognition in oracle bone inscription recognition tasks, characterized in that the augmented samples and the complete small sample data are input as training data into a predetermined convolutional neural network classifier for training to obtain a trained oracle bone inscription recognition classifier for the oracle bone inscription recognition task.

[0014] The present invention also provides a data augmentation device for small sample oracle bone inscription recognition, which uses unlabeled large sample data containing Chinese characters and oracle bone inscriptions and labeled small sample data of oracle bone inscriptions to perform data augmentation to obtain augmented samples, and is characterized in that it includes: a serialization unit, which uses a predetermined serialization algorithm to serialize the large sample data in bitmap format and the small sample data to obtain the large sample data in point sequence format composed of coordinate data and status data and the small sample data in point sequence format, and sets the large sample data in point sequence format as serialized large sample data, and sets the small sample data in point sequence format as serialized small sample data; a large sample encoding unit, which erases points in the serialized large sample data according to a predetermined erasure ratio, sets the serialized large sample data of the erased points as erased large sample data, sets the serialized large sample data of the unerased points as complete large sample data, and encodes the erased large sample data through a predetermined encoding network to obtain encoded erased large sample data; a feature extraction unit, which uses a predetermined feature extraction network to perform feature extraction on the encoded erased large sample data A deep feature is obtained; a prediction unit predicts the erased large sample data based on the deep feature and using a predetermined prediction network to obtain predicted large sample data; an augmentor output unit calculates a loss value based on the complete large sample data and the predicted large sample data through a predetermined loss function, and trains and updates a deep learning network composed of a coding network, a feature extraction network and a prediction network based on the loss value until the deep learning network converges to obtain a trained deep learning network as an augmentor; a small sample encoding unit generates multiple random erasure ratios according to a predetermined erasure ratio generation method, erases the points of the serialized small sample data according to the random erasure ratio, sets the serialized small sample data of the erased points as erased small sample data, sets the serialized small sample data of the non-erased points as complete small sample data, and encodes the erased small sample data through the encoding network to obtain encoded erased small sample data; and an augmented sample unit inputs the encoded erased small sample data into the augmentor to predict the erased small sample data to obtain predicted small sample data, and uses the predicted small sample data as augmented samples.

[0015] Function and Effect of the Invention

[0016] According to a data augmentation method for small sample oracle bone inscription recognition of the present invention, since the large sample data and the small sample data in the bitmap format are serialized by a serialization algorithm, the large sample data in the dot sequence format and the small sample data in the dot sequence format are obtained, so the stroke order and the general shape of the oracle bone inscription itself are taken into account and the oracle bone inscription is more appropriately described in the form of a dot sequence, which is convenient for learning from the self-supervised learning model in the field of natural language to achieve full utilization of unlabeled large sample data. At the same time, since the feature extraction network is used to extract the features of the large sample data erased after encoding to obtain deep features, the basic shape information and semantic information of Chinese characters or oracle bone inscriptions can be captured, and the disadvantage of requiring a large amount of labeled source data in traditional small sample learning is avoided, thereby ensuring the model's efficiency while reducing the data collection cost. In addition, since the augmenter is a trained deep learning network, and the deep learning network is composed of an encoding network, a feature extraction network, and a prediction network, and is continuously updated based on the loss value composed of the complete large sample data and the predicted large sample data, the unlabeled serialized large sample data is fully utilized and the demand for supervision data is reduced. In addition, since the points of the serialized small sample data are erased according to a random erasing ratio, the serialized small sample data of the erased points are set as erased small sample data, and the erased small sample data is encoded through the encoding network to obtain the encoded erased small sample data, and then the encoded erased small sample data is input into the augmenter, so as to predict the erased small sample data to obtain predicted small sample data, and the predicted small sample data is used as the augmented sample, so as to generate a large number of high-quality augmented samples that are similar to and diverse from the original image, so that the model applied to small sample oracle bone recognition has better robustness, higher accuracy, and stronger generalization ability, and significantly improves the results of small sample oracle bone recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 A flow chart of a data augmentation method for small sample oracle bone inscription recognition according to an embodiment of the present invention;

[0018] Figure 2 This is a flowchart of the sub-steps of step S2 of an embodiment of the present invention;

[0019] Figure 3 This is a flowchart of sub-steps of step S3 of an embodiment of the present invention;

[0020] Figure 4 A flowchart of sub-steps of step S6 of an embodiment of the present invention;

[0021] Figure 5 A structural block diagram of a data augmentation device for small sample oracle bone inscription recognition according to an embodiment of the present invention;

[0022] Figure 6 A schematic diagram of the working process of the large sample encoding unit, the feature extraction unit and the prediction unit according to an embodiment of the present invention; and

[0023] Figure 7 It is a schematic diagram of the working process of the small sample encoding unit and the augmented sample unit according to an embodiment of the present invention. DETAILED DESCRIPTION

[0024] In order to make the technical means, creative features, objectives and effects achieved by the present invention easy to understand, the following is a detailed description of the data augmentation method, application and device for small sample oracle bone inscription recognition of the present invention in combination with embodiments and drawings.

[0025] <Example>

[0026] In this embodiment, the large sample data refers to a large amount of unlabeled image data containing Chinese characters and oracle bone inscriptions, and the small sample data refers to a small sample of labeled oracle bone inscriptions image data.

[0027] Figure 1 The present invention is a flowchart of a data augmentation method for small sample oracle bone inscription recognition according to an embodiment of the present invention.

[0028] like Figure 1 As shown, a data augmentation method for small sample oracle bone inscription recognition includes the following steps:

[0029] Step S1, serialize the large sample data and the small sample data in the bitmap format using a predetermined serialization algorithm to obtain the large sample data in the point sequence format composed of coordinate data and status data and the small sample data in the point sequence format, and set the large sample data in the point sequence format as serialized large sample data, and set the small sample data in the point sequence format as serialized small sample data.

[0030] Specifically, the large sample data and the small sample data in the bitmap format are serialized using a serialization algorithm, thereby obtaining a series of strokes (s 1 ,s 2 ,…s n ), each point has a corresponding coordinate displacement and state vector, for example, point s 2 The specific form is (Δx, Δy, p 1 ,p 2 ,p 3 ), where (Δx, Δy) represents point s 2 Relative to point s 1 The coordinate displacement, (p 1 ,p 2 ,p 3 ) is the state vector of the point state represented by one-hot encoding, p 1 ,p2 ,p 3 They respectively indicate the beginning of a stroke, the end of a stroke, and the end of a Chinese character or oracle bone script.

[0031] The large sample data and small sample data in bitmap format refer to image data composed of pixels. The coordinate data refers to data composed of coordinate displacements of each point, and the state data refers to data composed of state vectors of each point.

[0032] Step S2, erase the points in the serialized large sample data according to a predetermined erasure ratio, set the serialized large sample data of the erased points as the erased large sample data, set the serialized large sample data of the unerased points as the complete large sample data, and encode the erased large sample data through a predetermined encoding network to obtain the encoded erased large sample data.

[0033] Figure 2 Flow chart of sub-steps of step S2 of an embodiment of the present invention.

[0034] like Figure 2 As shown, step S2 includes the following sub-steps:

[0035] Step S2-1: divide the points contained in each sample in the serialized large sample data into three types of serialized large sample sub-data according to the state data, and the number of the three types of serialized large sample sub-data is recorded as n 1 ,n 2 ,n 3 .

[0036] Specifically, all points contained in each sample (i.e., each image) in the serialized large sample data are divided into three categories of serialized large sample sub-data according to the state data, that is, according to whether the point represents the beginning of a stroke, the end of a stroke, the end of a Chinese character or the end of oracle bone script, all points in all samples in the serialized large sample data are divided into the corresponding three categories of serialized large sample sub-data.

[0037] Step S2-2, erase points in the serialized large sample sub-data in turn according to the erasure ratio, and finally set the serialized large sample data of the erased points as the erased large sample data, and set the serialized large sample data of the unerased points as the complete large sample data.

[0038] Specifically, the erasure ratio is 15%. In each type of serialized large sample sub-data, 15% of the points are selected as erased data, and the remaining points are used as complete data, and finally the erased large sample data with an erasure ratio of 15% and the complete large sample data are obtained.

[0039] Step S2-3, using a coding network to embed the coordinate data in the erased large sample data, and using a predetermined coding method to encode the state data, thereby obtaining the encoded erased large sample data.

[0040] Among them, the encoding network used to erase the coordinate data in the large sample data is a neural network composed of three fully connected layers, specifically, the three fully connected layers are fully connected layers with the number of neurons being (64, 128, 128).

[0041] In addition, the encoding method used for erasing state data in large sample data is a dictionary encoding method, specifically, a 128-dimensional vector is used to encode each state.

[0042] Step S3, using a predetermined feature extraction network to extract features from the encoded and erased large sample data to obtain deep features.

[0043] Figure 3 Flow chart of sub-steps of step S3 of an embodiment of the present invention.

[0044] like Figure 3 As shown, step S3 includes the following sub-steps:

[0045] Step S3-1, using a feature extraction network to convert the encoded and erased large sample data into a retrieval vector, a key vector and a value vector.

[0046] Among them, the feature extraction network includes a residual network and a fully connected layer.

[0047] Specifically, the fully connected layer in the feature extraction network is used to erase each point (s 1 ,s 2 ,…s n ) is converted into three types of data: Query, Key, and Value, which are specifically expressed as (q 1 ,q 2 ,…q n )、(k 1 ,k 2 ,…k n ) and (v 1 ,v 2 ,…v n ).

[0048] Step S3-2, using the feature extraction network to process the retrieval vector, key vector and value vector to obtain the attention-weighted features as the attention features.

[0049] Among them, the attention weight is Where N is the dimension of the key vector and the attention feature is

[0050] Step S3-3, use the feature extraction network to process the attention feature to obtain a deep feature with the same dimension as the input feature.

[0051] Specifically, the residual network and the fully connected layer in the feature extraction network are used to process the attention features to obtain output features with the same dimension as the input features as deep features.

[0052] Among them, the feature extraction network is a deep neural network based on bidirectional encoding representation and self-attention mechanism.

[0053] Step S4, predicting the erased large sample data based on the deep features and using a predetermined prediction network to obtain predicted large sample data.

[0054] Among them, the prediction network is a neural network composed of four fully connected layers, specifically a fully connected layer with the number of neurons being (128, 128, 64, 5).

[0055] Specifically, the prediction network predicts the erased points in the erased large sample data, thereby obtaining the predicted coordinate displacement and state vector of the erased points to form the predicted large sample data. The predicted points can be expressed as

[0056] Step S5, based on the complete large sample data and the predicted large sample data, a loss value is calculated through a predetermined loss function, and based on the loss value, a deep learning network composed of an encoding network, a feature extraction network, and a prediction network is trained and updated until the deep learning network converges to obtain a trained deep learning network and use it as an augmentor.

[0057] Among them, the loss function for the complete large sample data and the coordinate data in the predicted large sample data is the L1 norm loss function; the loss function for the complete large sample data and the state data in the predicted large sample data is the cross entropy loss function.

[0058] In this embodiment, the training update of the deep learning network is performed based on the loss value obtained by the L1 norm loss function and the loss value obtained by the cross entropy loss function.

[0059] Step S6, generate multiple random erasure ratios according to a predetermined erasure ratio generation method, erase the points of the serialized small sample data according to the random erasure ratios, set the serialized small sample data of the erased points as the erased small sample data, set the serialized small sample data of the non-erased points as the complete small sample data, and encode the erased small sample data through the encoding network to obtain the encoded erased small sample data.

[0060] Figure 4It is a flowchart of the sub - steps of step S6 in the embodiment of the present invention.

[0061] As Figure 4 shown, step S6 includes the following sub - steps:

[0062] Step S6 - 1: Divide the points included in each sample in the serialized small - sample data into three categories of serialized small - sample sub - data according to the status data, and the quantity is denoted as n 4 ,n 5 ,n 6 .

[0063] Specifically, divide all the points included in each sample (i.e., each image) in the serialized small - sample data into three categories of serialized small - sample sub - data according to the status data. That is, according to which state among the states of the start of a stroke, the end of a stroke, and the end of oracle bone inscriptions the point represents, all the points in each sample in the serialized small - sample data are divided into three categories of serialized small - sample sub - data.

[0064] Step S6 - 2: Use the erasure ratio generation method to generate multiple random erasure ratios, erase the points in each category of serialized small - sample sub - data according to the randomly selected random erasure ratios, set the serialized small - sample data of the erased points as the erased small - sample data, and set the serialized small - sample data of the non - erased points as the complete small - sample data.

[0065] Among them, the erasure ratio generation method is to uniformly sample from the interval [a, b], where 0 < a, b ≤ 1, with an interval of k to obtain different erasure ratios as multiple random erasure ratios.

[0066] In this embodiment, for the points in each category of serialized small - sample sub - data in all images, erase them according to the randomly selected random erasure ratio m i and finally obtain samples with different erasure ratios to form the erased small - sample data.

[0067] Step S6 - 3: Use the encoding network to perform embedding encoding on the coordinate data in the erased small - sample data, and use the predetermined encoding method to encode the status data in the erased small - sample data, so as to obtain the encoded erased small - sample data.

[0068] Among them, the encoding network used for the coordinate data is a neural network composed of three fully - connected layers. Specifically, the fully - connected layer is a fully - connected layer with the number of neurons (64, 128, 128).

[0069] In addition, the encoding method used for the status data is the dictionary encoding method. Specifically, each status is encoded using a 128 - dimensional vector.

[0070] Step S7, inputting the encoded erased small sample data into an augmentor to predict the erased small sample data to obtain predicted small sample data, and using the predicted small sample data as augmented samples.

[0071] In this embodiment, the augmenter predicts the erased points in the encoded erased small sample data, thereby obtaining the predicted coordinate displacement and state vector of the erased points to form predicted small sample data, and uses the predicted small sample data as augmented samples.

[0072] Specifically, the feature extraction network trained in the augmentor is used to extract features from the encoded erased small sample data to obtain the corresponding deep features, and then the deep features are input into the trained prediction network to predict the coordinate data and state data of the erased part to obtain predicted small sample data, and finally the predicted small sample data is used as the augmented sample.

[0073] In addition, the above-mentioned data augmentation method for small sample oracle bone recognition is applied to the oracle bone recognition task by inputting the augmented sample and the complete small sample data as training data into a predetermined convolutional neural network classifier for training to obtain a trained oracle bone recognition classifier for the oracle bone recognition task.

[0074] Figure 5 The present invention is a structural block diagram of a data augmentation device for small sample oracle bone inscription recognition according to an embodiment of the present invention.

[0075] The data augmentation method for small sample oracle bone inscription recognition can be applied in a computer to form a data augmentation device 1, which includes a serialization unit 11, a large sample encoding unit 12, a feature extraction unit 13, a prediction unit 14, an augmentation output unit 15, a small sample encoding unit 16 and an augmented sample unit 17 (such as Figure 5 shown).

[0076] The serialization unit 11 uses a predetermined serialization algorithm to serialize the large sample data and the small sample data in the bitmap format to obtain the large sample data in the point sequence format and the small sample data in the point sequence format composed of coordinate data and status data, and sets the large sample data in the point sequence format as serialized large sample data, and sets the small sample data in the point sequence format as serialized small sample data.

[0077] Figure 6 The present invention is a flowchart of the large sample encoding unit, feature extraction unit and prediction unit in accordance with an embodiment of the present invention.

[0078] The large sample encoding unit 12 erases points in the serialized large sample data according to a predetermined erasure ratio, sets the serialized large sample data of the erased points as erased large sample data, sets the serialized large sample data of the unerased points as complete large sample data, and encodes the erased large sample data through a predetermined encoding network to obtain encoded erased large sample data.

[0079] like Figure 6 As shown, the large sample encoding unit 12 erases the points on each image in the serialized large sample data according to the erasing ratio m=15%, thereby forming erased large sample data. At this time, each point of the erased large sample data is represented by (s 1 ,s 2 ,-,…,s n-2 ,-,s n ), the erased large sample data is encoded to obtain the encoded erased large sample data, and each point of the encoded erased large sample data is represented as (E 1 ,E 2 ,E M ,…,E n-2 ,E M ,E n ).

[0080] The feature extraction unit 13 uses a predetermined feature extraction network to extract features from the encoded and erased large sample data to obtain deep features.

[0081] like Figure 6 As shown, the feature extraction unit 13 uses a bidirectional encoding representation and a self-attention mechanism (i.e. Figure 6 The feature extraction network of the self-attention layer in the encoding removes the large sample data to obtain the deep feature (O 1 ,O 2 ,O 3 ,…,O n-2 ,O n-1 ,O n ).

[0082] The prediction unit 14 predicts the erased large sample data based on the depth feature and using a predetermined prediction network to obtain predicted large sample data.

[0083] like Figure 6 As shown, the prediction unit 14 calculates the depth feature ( 1 ,O 2 ,O 3 ,…,O n-2 ,O n-1 ,O n ) predicts the erased point (S 3 ,S n-1 ), thereby obtaining predicted large sample data.

[0084] The augmenter output unit 15 obtains a loss value based on the complete large sample data and the predicted large sample data by using a predetermined loss function, and trains and updates the deep learning network composed of the encoding network, the feature extraction network and the prediction network based on the loss value until the deep learning network converges to obtain a trained deep learning network as an augmenter.

[0085] like Figure 6 As shown, the deep learning network is updated based on the complete large sample data and the loss constructed based on the predicted large sample data.

[0086] Figure 7 It is a schematic diagram of the working process of the small sample encoding unit and the augmented sample unit according to an embodiment of the present invention.

[0087] The small sample encoding unit 16 generates multiple random erasure ratios according to a predetermined erasure ratio generation method, erases the points of the serialized small sample data according to the random erasure ratios, sets the serialized small sample data of the erased points as erased small sample data, sets the serialized small sample data of the non-erased points as complete small sample data, and encodes the erased small sample data through the encoding network to obtain the encoded erased small sample data.

[0088] like Figure 7 As shown, the small sample encoding unit 16 erases the points in the serialized small sample data according to the random erasure ratios m=10%, m=25% and m=50% generated by the random erasure ratio generation method. Each point of the erased small sample data is represented by (s 1 ,s 2 ,-,…,s n-2 ,-,s n ), the erased small sample data is encoded to obtain the encoded erased small sample data, and each point of the encoded erased small sample data is represented as (E 1 ,E 2 ,E M ,…,E n-2 ,E M ,E n ).

[0089] The augmented sample unit 17 inputs the encoded erased small sample data into the augmentor to predict the erased small sample data to obtain predicted small sample data, and uses the predicted small sample data as the augmented sample.

[0090] like Figure 7 As shown, the augmented sample unit 17 uses the trained bidirectional encoding representation and self-attention mechanism (i.e. Figure 7 The feature extraction network of the self-attention layer in the erased small sample data extracts features to obtain deep features (O 1,O 2 ,O 3 ,…,O n-2 ,O n-1 ,O n ), and input the trained prediction network in the augmenter to predict the erased points (S 3 ,s n-1 ), thereby obtaining predicted small sample data, and using the predicted small sample data as augmented samples.

[0091] Example Function and Effect

[0092] According to a data augmentation method, application and device for small sample oracle bone inscription recognition provided by this embodiment, since the large sample data and small sample data in bitmap format are serialized by a serialization algorithm, thereby obtaining large sample data in dot sequence format and small sample data in dot sequence format, the stroke order and general shape of the oracle bone inscription itself are taken into account and the oracle bone inscription is more appropriately described in the form of a dot sequence, which is convenient for drawing on the self-supervised learning model in the field of natural language to achieve full utilization of unlabeled large sample data. At the same time, since the feature extraction network is used to extract the features of the large sample data erased after encoding to obtain deep features, the basic shape information and semantic information of Chinese characters or oracle bone inscriptions can be captured, and the disadvantage of requiring a large amount of labeled source data in traditional small sample learning is avoided, thereby ensuring the model's efficiency while reducing the data collection cost. In addition, since the augmenter is a trained deep learning network, and the deep learning network is composed of an encoding network, a feature extraction network and a prediction network, and is continuously updated based on the loss value composed of the complete large sample data and the predicted large sample data, the unlabeled serialized large sample data is fully utilized and the demand for supervision data is reduced. In addition, since the points of the serialized small sample data are erased according to a random erasing ratio, the serialized small sample data of the erased points are set as erased small sample data, and the erased small sample data is encoded through the encoding network to obtain the encoded erased small sample data, and then the encoded erased small sample data is input into the augmenter, so as to predict the erased small sample data to obtain predicted small sample data, and the predicted small sample data is used as the augmented sample, so as to generate a large number of high-quality augmented samples that are similar to and diverse from the original image, so that the model applied to small sample oracle bone recognition has better robustness, higher accuracy, and stronger generalization ability, and significantly improves the results of small sample oracle bone recognition.

[0093] In addition, in the embodiment, since in step S2-2 the points in the serialized large sample data are divided into three categories of serialized large sample sub-data according to the state data and the points of each category of serialized large sample sub-data are erased according to the erasure ratio to obtain erased large sample data and complete large sample data, the points in each state are erased according to the erasure ratio to make the erased large sample data and the complete large sample data more evenly distributed.

[0094] In addition, in the embodiment, since the feature extraction network is a deep neural network based on bidirectional encoding representation and self-attention mechanism, it is able to capture the basic shape information and semantic information of Chinese characters or oracle bone inscriptions.

[0095] In addition, in the embodiment, since the points of each type of serialized small sample sub-data are erased according to a randomly selected random erasure ratio in step S6-2, the ratio of each type of serialized small sample sub-data being erased is random, which increases the randomness of the object predicted by the augmenter in the subsequent step S7.

[0096] The above embodiments are only used to illustrate specific implementation modes of the present invention, and the present invention is not limited to the description scope of the above embodiments.

[0097] In the embodiment, a data augmentation method for small sample oracle bone script recognition is applied to the oracle bone script recognition task. In other schemes of the present invention, it can also be applied to data augmentation tasks of other Chinese characters, various texts and hand-drawn sketches, thereby generating a large number of high-quality augmented samples that are similar to and diverse from the original images.

[0098] In the embodiment, the erasure ratio in step S2 is set to 15%. In other schemes of the present invention, the erasure ratio may also be set to 10%, 20%, or other erasure ratios that do not affect the quality of the predicted large sample data.

Claims

1. A data augmentation method for small sample oracle bone inscription recognition, which uses large sample data containing Chinese characters and oracle bone inscriptions without annotations and small sample data containing oracle bone inscriptions with annotations to perform data augmentation to obtain augmented samples, characterized in that: The steps include: Step S1, serializing the large sample data and the small sample data in bitmap format using a predetermined serialization algorithm to obtain large sample data in point sequence format and small sample data in point sequence format composed of coordinate data and state data, and setting the large sample data in point sequence format as serialized large sample data, and setting the small sample data in point sequence format as serialized small sample data; Step S2, erasing points in the serialized large sample data according to a predetermined erasure ratio, setting the serialized large sample data of the erased points as the erased large sample data, setting the serialized large sample data of the unerased points as the complete large sample data, and encoding the erased large sample data through a predetermined encoding network to obtain encoded erased large sample data; Step S3, using a predetermined feature extraction network to extract features from the encoded and erased large sample data to obtain deep features, includes the following sub-steps: Step S3-1, using the feature extraction network to convert the encoded and erased large sample data into a search vector, a key vector and a value vector, Step S3-2, using the feature extraction network to process the search vector, the key vector and the value vector to obtain attention-weighted features as attention features, Step S3-3, using the feature extraction network to process the attention feature to obtain the deep feature with the same dimension as the input feature, wherein the feature extraction network is a deep neural network based on bidirectional encoding representation and self-attention mechanism; Step S4, predicting the erased large sample data based on the deep features and using a predetermined prediction network to obtain predicted large sample data; Step S5, calculating a loss value through a predetermined loss function based on the complete large sample data and the predicted large sample data, and training and updating a deep learning network composed of the encoding network, the feature extraction network and the prediction network based on the loss value until the deep learning network converges to obtain a trained deep learning network as an augmentor; Step S6, generating multiple random erasure ratios according to a predetermined erasure ratio generation method, erasing the points of the serialized small sample data according to the random erasure ratios, setting the serialized small sample data of the erased points as the erased small sample data, setting the serialized small sample data of the unerased points as the complete small sample data, and encoding the erased small sample data through the encoding network to obtain the encoded erased small sample data; Step S7, inputting the encoded erased small sample data into the augmentor to predict the erased small sample data to obtain predicted small sample data, and using the predicted small sample data as the augmented sample.

2. The data augmentation method for small sample oracle bone inscription recognition according to claim 1, characterized in that: in, The step S2 includes the following sub-steps: Step S2-1, dividing the points contained in each sample in the serialized large sample data into three types of serialized large sample sub-data according to the state data; Step S2-2, sequentially erasing the points in the serialized large sample sub-data according to the erasure ratio, and finally setting the serialized large sample data of the erased points as the erased large sample data, and setting the serialized large sample data of the unerased points as the complete large sample data; Step S2-3, using the coding network to embed the coordinate data in the erased large sample data, and using a predetermined coding method to encode the state data, thereby obtaining the encoded erased large sample data. The erasure ratio is 15%.

3. The data augmentation method for small sample oracle bone inscription recognition according to claim 1, characterized in that: in, The step S6 includes the following sub-steps: Step S6-1, dividing the points contained in each sample in the serialized small sample data into three types of serialized small sample sub-data according to the state data; Step S6-2, using the erasure ratio generation method to generate a plurality of random erasure ratios, erasing the points of each type of the serialized small sample sub-data according to the randomly selected random erasure ratio, setting the serialized small sample data of the erased points as the erased small sample data, and setting the serialized small sample data of the unerased points as the complete small sample data; Step S6-3, using the coding network to embed the coordinate data in the erased small sample data, and using a predetermined coding method to encode the state data in the erased small sample data, so as to obtain the encoded erased small sample data.

4. The data augmentation method for small sample oracle bone inscription recognition according to claim 3 is characterized by: in, The erasure ratio generation method is to uniformly sample different erasure ratios from the interval [a, b], where 0 < a, b ≤ 1, at intervals of k as multiple random erasure ratios. ​ 5. The data augmentation method for small sample oracle bone inscription recognition according to claim 2 or 3, characterized in that: in, The encoding network is a neural network consisting of three fully connected layers. The encoding method is a dictionary encoding method.

6. The data augmentation method for small sample oracle bone inscription recognition according to claim 1, characterized in that: in, In step S5, the loss function for the coordinate data in the complete large sample data and the predicted large sample data is an L1 norm loss function. In the step S5, the loss function for the complete large sample data and the state data in the predicted large sample data is a cross entropy loss function.

7. The data augmentation method for small sample oracle bone inscription recognition according to claim 1, characterized in that: in, In step S4, the prediction network is a neural network consisting of four fully connected layers.

8. A method for applying the data augmentation method for small sample oracle bone inscription recognition as claimed in any one of claims 1 to 7 to an oracle bone inscription recognition task, characterized in that: The augmented samples and the complete small sample data are used as training data to input into the predetermined convolutional neural network classifier for training, thereby obtaining a trained oracle bone inscription recognition classifier for the oracle bone inscription recognition task. The augmented sample is obtained by the data augmentation method for small sample oracle bone inscription recognition described in any one of claims 1 to 7.

9. A data augmentation device for small sample oracle bone inscription recognition, which uses large sample data containing Chinese characters and oracle bone inscriptions without annotations and small sample data containing oracle bone inscriptions with annotations to perform data augmentation to obtain augmented samples, characterized in that: include: A serialization unit, which serializes the large sample data and the small sample data in the bitmap format using a predetermined serialization algorithm to obtain large sample data in a point sequence format and small sample data in a point sequence format composed of coordinate data and state data, and sets the large sample data in the point sequence format as serialized large sample data, and sets the small sample data in the point sequence format as serialized small sample data; A large sample encoding unit, which erases points in the serialized large sample data according to a predetermined erasure ratio, sets the serialized large sample data of the erased points as the erased large sample data, sets the serialized large sample data of the unerased points as the complete large sample data, and encodes the erased large sample data through a predetermined encoding network to obtain encoded erased large sample data; The feature extraction unit uses a predetermined feature extraction network to extract features from the encoded and erased large sample data to obtain deep features, including the following sub-steps: step S3-1, using the feature extraction network to convert the encoded and erased large sample data into a retrieval vector, a key vector and a value vector, step S3-2, using the feature extraction network to process the retrieval vector, the key vector and the value vector to obtain attention-weighted features as attention features, step S3-3, using the feature extraction network to process the attention features to obtain the deep features with the same dimension as the input features, wherein the feature extraction network is a deep neural network based on bidirectional coding representation and self-attention mechanism; A prediction unit, which predicts the erased large sample data based on the deep features and using a predetermined prediction network to obtain predicted large sample data; An augmentor output unit calculates a loss value based on the complete large sample data and the predicted large sample data through a predetermined loss function, and trains and updates a deep learning network composed of the encoding network, the feature extraction network, and the prediction network based on the loss value until the deep learning network converges to obtain a trained deep learning network as an augmentor; The small sample encoding unit generates a plurality of random erasure ratios according to a predetermined erasure ratio generation method, erases the points of the serialized small sample data according to the random erasure ratios, sets the serialized small sample data of the erased points as erased small sample data, sets the serialized small sample data of the unerased points as complete small sample data, and encodes the erased small sample data through the encoding network to obtain encoded erased small sample data; and The augmented sample unit inputs the encoded erased small sample data into the augmentor to predict the erased small sample data to obtain predicted small sample data, and uses the predicted small sample data as the augmented sample.

Images