A semantically enhanced zero-shot oracle bone character recognition method

Abstract

The present invention discloses a semantically enhanced zero-shot oracle-bone script character recognition method, comprising: constructing learnable prompt information for oracle-bone script images according to domains and character categories; introducing a contrastive language-image pre-training model, and sampling data by a balanced sampling strategy to optimize the learnable prompt information, so as to mine semantic information from oracle-bone script rubbings and character mold images; removing domain prompt information to obtain domain-independent prompt information, and then encoding domain-independent character semantics through a semantic encoder; supervising and training a visual encoder through an intra-class shared feature learning branch and an inter-class difference feature learning branch; and using the trained visual encoder to achieve zero-shot oracle-bone script character recognition.

Description

Technical Field

[0001] The invention belongs to the technical field of image processing and artificial intelligence, and in particular relates to a semantically enhanced zero-sample oracle bone character recognition method. Background Art

[0002] Oracle bone inscriptions are a valuable heritage of ancient culture and have irreplaceable and important value in studying ancient history, language and writing development, and social life. Oracle bone inscriptions not only carry rich historical information, but are also the key to studying ancient Chinese and its evolution, and exploring ancient social structure and culture. However, due to the particularity of oracle bone writing and its long history, its recognition is extremely challenging and can usually only be performed manually by experts with rich experience. This recognition process that relies on expert experience is cumbersome and often requires fine character identification and a large amount of data review, resulting in low efficiency. With the development of digital technology and artificial intelligence, the research on automatic recognition of oracle bone inscriptions has become an emerging trend. Through some intelligent algorithms, efficient and accurate recognition of oracle bone images can be achieved, which is of great significance for accelerating the research process of oracle bone inscriptions and promoting the inheritance and development of traditional culture. It can not only provide researchers with powerful tools to speed up the collation and research of oracle bone inscriptions, but also enable the general public to more conveniently access and understand this ancient cultural heritage. Therefore, studying an automatic recognition method for oracle bone inscriptions is not only a need for technological innovation, but also an important need for cultural inheritance and academic research.

[0003] In the research history of oracle bone inscription automatic recognition, the methods can be roughly divided into two categories: traditional methods and deep learning-based methods. Traditional oracle bone inscription recognition technology mainly relies on manually designed features, such as Fourier descriptors based on curvature histograms, undirected graphs, and block histograms to assist recognition. The core of these methods lies in fine feature extraction and complex system integration, including denoising, stroke or component extraction, feature extraction, and post-processing. Although these methods have made some progress in the early research on oracle bone inscription recognition, they generally have some significant limitations, such as high dependence on domain expertise, high complexity of manually constructed integrated systems, and weak model generalization ability. These factors have jointly limited their improvement in oracle bone inscription recognition performance. With the rise and development of deep learning technology, the research focus in the field of oracle bone inscription recognition has begun to shift to using this technology for recognition. Early deep learning methods significantly improved the performance of oracle bone inscription recognition by introducing convolutional neural networks. Furthermore, researchers have constructed a public oracle bone inscription rubbing character dataset and used a variety of mainstream deep learning models on this dataset for verification. Subsequent research work mainly aims to solve the problem of class imbalance in oracle bone script datasets. This is because some oracle bone script characters have a large sample size, while others are relatively rare, which leads to overfitting of frequently occurring categories during model training and difficulty in accurately identifying those rare categories. To meet this challenge, researchers have adopted a variety of strategies including data enhancement, complex loss function design, and generative adversarial networks (GANs). Data enhancement uses techniques such as Mixup to generate minority class samples to improve the recognition ability of rare categories; complex loss functions, such as triple loss, train models by sampling difficult triples to better learn the features of minority classes; and the introduction of generative adversarial networks (GANs) provides another solution to the problem of class imbalance. By generating minority class samples, GANs can increase the amount of data for rare categories, thereby assisting the model in learning more balanced feature representations. These methods have effectively improved the performance of automatic oracle bone script recognition.

[0004] Although these deep learning-based methods have made significant progress in oracle bone recognition, most of them are based on the so-called closed set assumption, which means that the system can only recognize those categories that have been seen during the training phase. In short, if the model is not exposed to certain specific oracle bone characters during learning, then in practical applications, it will not be able to recognize these newly appeared characters. This assumption has encountered obvious challenges in practical applications, because with the deepening of archaeological excavations, new oracle bone materials and unknown character categories in them are constantly discovered, which limits the scope of application of this method. Faced with this challenge, some researchers began to explore zero-shot oracle bone recognition methods to achieve the recognition of oracle bone characters that the model has never seen. It uses visual matching to match the similarity of the rubbing image to be recognized with the font image library of known categories, and takes the category of the most similar font image as the classification result. However, due to the large intra-class variance of oracle bone inscriptions, there are diverse glyphs in the same category of oracle bone characters, which affects the visual matching between rubbings and fonts, resulting in the low accuracy of zero-shot recognition of oracle bone characters. Summary of the invention

[0005] In view of this, it is necessary to provide a semantically enhanced zero-shot oracle bone character recognition method for the above-mentioned technical problems. The method, based on visual matching, first uses the visual and semantic alignment characteristics of the contrastive visual-language pre-training model CLIP to mine domain-independent character semantic features from oracle bone rubbings and character template images, and then uses semantics to guide the model to extract visual features shared by the same type of characters in different glyphs, ignoring the differences in local visual details, thereby alleviating the problem of glyph changes affecting visual matching and improving recognition performance.

[0006] The present invention discloses a semantically enhanced zero-sample oracle bone character recognition method, comprising the following steps:

[0007] Step 1, construct learnable prompt information for oracle bone inscription images based on domain and character category;

[0008] Step 2: introduce a contrastive language-image pre-training model, sample data through a balanced sampling strategy, and optimize the learnable prompt information to mine semantic information from oracle bone rubbings and character mold images;

[0009] Step 3, removing the domain prompt information to obtain domain-independent prompt information, and then encoding it through a semantic encoder to obtain domain-independent character semantics;

[0010] Step 4: supervise the visual encoder training through the intra-class shared feature learning branch and the inter-class difference feature learning branch;

[0011] Step 5: Use the trained visual encoder to achieve zero-shot oracle character recognition.

[0012] Specifically, the learnable prompt information is in the form of "A photo of character”.

[0013] where d and c represent the domain and character category of the oracle bone image, respectively, d∈{p,s}, p and s represent the domain of the rubbing and the font, respectively, and c∈C b ,C b represents the set of character categories visible during training. M and N represent the number of tokens in the domain hint information and character category hint information, respectively. (m∈{1,...,M}) and (n∈{1,...,N}) represents a token, which is a vector of the same dimension as the word vector in CLIP.

[0014] Each oracle bone image in the training set needs to construct its learnable prompt information according to its domain and character category. The specific construction method is as follows:

[0015] According to the domain category d of the image, we set learnable domain hint information for the oracle image

[0016] Similarly, according to the character category c, we set learnable character category hint information for the oracle bone image

[0017] Then, using the commonly used prompt template "A photo of_", fill the learnable domain prompt information and character category prompt information into the prompt template, and add the category information "character", so as to obtain the final learnable prompt information "A photo of character”.

[0018] Specifically, the contrastive language-image pre-training model introduced is CLIP, which constructs a shared multimodal feature space through image and text input, and uses the contrastive learning mechanism to shorten the distance between correctly matched image-text pairs in the feature space, while pushing away the distance between mismatched image-text pairs in the feature space, thus having the ability to align vision and semantics in the multimodal feature space. With this ability, the oracle bone rubbings and character image data are first sampled using a balanced sampling strategy, and then the learnable prompt information is optimized.

[0019] Specifically, the balanced sampling strategy is as follows:

[0020] Randomly sample a subset of character categories from the training character categories;

[0021] Uniformly sample each category in the character category subset so that the number of samples for each category is equal;

[0022] At the same time, uniform sampling is also performed at the level of the two domains, rubbings and character molds, so that the number of samples in the two domains is also equal;

[0023] Through the above steps, balanced sampling at the domain level and character category level can be achieved.

[0024] Specifically, the specific process of the optimization is as follows:

[0025] Use the pre-trained CLIP visual encoder to encode the oracle bone rubbings and character images to obtain visual features Where d and c represent the domain and character category of the oracle bone image, respectively;

[0026] At the same time, the pre-trained CLIP semantic encoder is used to encode the learnable prompt information to obtain semantic features Where d and c represent the domain and character category corresponding to the prompt information respectively;

[0027] Similar to the training process of CLIP, the contrastive learning loss function L is used con , taking the visual features and semantic features of the same domain and the same character category as positive sample pairs, shortening their distance in the feature space, and taking the visual features and semantic features of different domains or different character categories as negative sample pairs, pushing their distance in the feature space, thereby making the learnable hint information semantic features Visual features of the corresponding oracle bone images Align and optimize the semantics of learnable prompt information;

[0028] The loss function is as follows:

[0029]

[0030] Among them, L i is the contrast loss function that aligns visual features to the semantic space, L t It is a contrast loss function that aligns semantic features to the visual space. The formulas for the two are as follows:

[0031]

[0032]

[0033] Where B is the batch size, and the character categories corresponding to the i-th visual feature, the t-th semantic feature, and the j-th and q-th visual or semantic features in the batch are y i ,yt ,y j ,y q ∈C b Indicates the character category, and the corresponding domains are d i ,d t ,d j ,d q ∈{p,s}, the positive sample index sets corresponding to the i-th visual feature and the t-th semantic feature in the batch are P(i) and P(t) respectively, τ represents the temperature coefficient, and <·,·> represents the cosine similarity.

[0034] Specifically, the domain prompt information is removed in step 3 to obtain the domain-independent prompt information, which is obtained by removing the domain prompt information in the learnable prompt information after the learnable prompt information is optimized in step 2. Remove, get the domain-independent prompt "A photo of character”.

[0035] Specifically, the semantic encoder is a pre-trained CLIP semantic encoder, which is used to generate the domain-independent prompt information "A photo of The domain-independent character semantic features obtained by encoding "character" are recorded as T c .

[0036] Specifically, in step 4, the visual encoder is the visual encoder of CLIP, but it is no longer pre-trained, but the parameters are re-initialized.

[0037] Specifically, in the intra-class shared feature learning branch, the rubbing image and the character image are first encoded using a visual encoder to obtain visual features: and Then, contrastive learning loss is used, with the visual features of the rubbings or font images as anchors, the semantic features of the same character category as positive samples, and the semantic features of the remaining categories as negative samples, to shorten the distance to the positive samples and increase the distance to the negative samples.

[0038] Among them, the semantic feature is the domain-independent character semantics T obtained in step 3 c , and in the training process of step 4, the domain-independent semantic features are fixed and no parameters need to be updated to provide stable semantics.

[0039] Preferably, the contrastive learning loss can be implemented in engineering by first storing the semantic features of all character categories during training, and then using the cross entropy loss L ce It can be implemented cleverly, and the formula is as follows:

[0040]

[0041] Among them, B represents the batch size, q is the index of the positive sample, and d i ∈{p,s} represents the domain of the i-th sample in the batch, y i ,y j ,y q Respectively represent the character categories corresponding to the i, j, q visual or semantic features in the batch, C b Represents the set of character categories visible during training.

[0042] Specifically, in the inter-class difference feature learning branch, oracle bone rubbing images and character mold images are sampled as training samples through a random sampling strategy, and after encoding these images through a visual encoder, the N-Pair loss function L is used to np To supervise.

[0043] The random sampling strategy refers to randomly sampling n character categories in each batch, and sampling a rubbing image and a character mold image for each category.

[0044] The N-Pair loss function L np , the formula is as follows:

[0045]

[0046] Where n is the number of randomly sampled character categories in each batch, and y i ,y j represents the character category of the i,jth sample in the batch, p and s represent the rubbing and font domain respectively.

[0047] Specifically, the zero-sample oracle bone character recognition is achieved by the visual encoder through visual matching after the training in step 4, and the steps are as follows:

[0048] Given the glyph images of all oracle bone characters that need to be recognized, a glyph image library is formed;

[0049] Whenever there is a new category that needs to be identified, the font image of the new category is also put into the font image library;

[0050] Encode all the fonts in the font image library through the visual encoder to obtain their visual features;

[0051] Encode the rubbing image to be identified through a visual encoder to obtain its visual features;

[0052] Calculate the Euclidean distance between the visual features of the rubbing image to be identified and the visual features of all the fonts in the font image library;

[0053] The character image with the smallest Euclidean distance is found, and the character category of the character image is used as the classification result of the rubbing image to be recognized, thereby realizing zero-sample oracle bone character recognition.

[0054] Compared with the prior art, the method of the present invention has the beneficial effect that it introduces the character semantics of oracle bone script, guides the model to focus on the key features shared within the class in the intra-class shared feature learning branch, aligns the visual features of different glyphs in each character category to the semantics of the character category, and guides the model to focus on the key features shared within the class, thereby alleviating the problem of large intra-class variance in oracle bone script and improving the recognition accuracy. At the same time, the inter-class difference feature learning branch is used to further enhance the model's discrimination of different character categories, thereby better completing zero-sample oracle bone script character recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 A schematic flow chart of an implementation method of the present invention is shown; DETAILED DESCRIPTION

[0056] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] For the purpose of reference and clarity, the technical terms, abbreviations or acronyms used below are summarized and explained as follows:

[0058] Rubbing images: obtained by rubbing on tortoise shells, animal bones and other text carriers, they are the first-hand material for oracle bone research and are also the objects of oracle bone character recognition.

[0059] Character mold image: obtained by experts copying the rubbing image, and at the same time repairing the incomplete character shape, it is an important tool to assist in the study of oracle bone inscriptions.

[0060] Mixup: A data augmentation method that generates new data by performing a linear combination between two different samples and their labels.

[0061] GAN (Generative Adversarial Network): Generative adversarial network, a neural network commonly used to generate data, which consists of a generator network and a discriminator network, and the two networks compete with each other during training.

[0062] CLIP (Contrastive Language-Image Pre-training): Contrastive vision-language pre-training model is a multimodal pre-training technology that uses contrastive learning of large amounts of text and image data to understand and associate these two different forms of information.

[0063] Prompt information: It is a preset text or question used to guide or stimulate the pre-trained model to generate specific outputs. Through carefully designed prompt information, the model can more effectively understand and respond to the user's intentions on specific tasks, thereby improving the accuracy and relevance of task execution.

[0064] Contrastive Learning: Contrastive learning loss is a self-supervised learning method that learns an effective representation of data by maximizing the similarity between pairs of positive samples while minimizing the similarity between pairs of negative samples.

[0065] N-Pair loss: is a loss function used for metric learning in deep learning, which aims to improve the distinction between positive and negative samples in the feature space by considering one positive sample pair and multiple negative sample pairs at the same time.

[0066] Figure 1 A flowchart of an embodiment of the present invention is shown. A semantically enhanced zero-sample oracle bone character recognition method comprises the following steps:

[0067] Step 1, construct learnable prompt information for oracle bone inscription images based on domain and character category;

[0068] Step 2: introduce a contrastive language-image pre-training model, sample data through a balanced sampling strategy, and optimize the learnable prompt information to mine semantic information from oracle bone rubbings and character mold images;

[0069] Step 3, removing the domain prompt information to obtain domain-independent prompt information, and then encoding it through a semantic encoder to obtain domain-independent character semantics;

[0070] Step 4: supervise the visual encoder training through the intra-class shared feature learning branch and the inter-class difference feature learning branch;

[0071] Step 5: Use the trained visual encoder to achieve zero-shot oracle character recognition.

[0072] Specifically, this embodiment adopts the following steps to implement the inventive method.

[0073] 1. Collect oracle bone rubbings and character model data to be used as the training set of the model. Their category space needs to be the same, and C b is the set of all character categories in the category space.

[0074] 2. Construct learnable hints for oracle bone images based on domains and character categories.

[0075] (1) It is necessary to construct a graph of the form “A photo of character" can be learned.

[0076] (2) We use d and c to represent the domain of oracle bone images and character categories, d∈{p,s}, p and s represent the domain of rubbings and fonts, and c∈C b For an image with domain d and character category c, the specific construction method of its learnable hint information is as follows: According to the domain category d of the image, we set the learnable domain hint information for the oracle bone image Similarly, according to the character category c, we set learnable character category hint information for the oracle bone image Where M and N represent the number of tokens in the domain prompt information and character category prompt information, respectively. (m∈{1,...,M}) and (n∈{1,...,N}) represents a token, which is a vector of the same dimension as the word vector in CLIP.

[0077] (3) Then, using the commonly used prompt template “A photo of_”, fill in the learnable domain prompt information and character category prompt information into the prompt template, and add the category information “character” to obtain the final learnable prompt information “A photo of character”.

[0078] 3. Introduce a contrastive language-image pre-training model, sample data through a balanced sampling strategy, and optimize the learnable prompt information to extract semantic information from oracle bone rubbings and character mold images.

[0079] (1) Specifically, the contrastive language-image pre-training model introduced is CLIP, which constructs a shared multimodal feature space through image and text input, and uses the contrastive learning mechanism to shorten the distance between correctly matched image-text pairs in the feature space, while pushing away the distance between mismatched image-text pairs in the feature space. Therefore, it has the ability to align vision and semantics in the multimodal feature space. With this ability, the learnable prompt information is optimized.

[0080] (2) A balanced sampling strategy is used to sample the oracle bone script rubbings and font image data. First, a character category subset is randomly sampled from the training character categories. Then, each category in the character category subset is uniformly sampled so that the number of samples in each category is equal. Finally, uniform sampling is performed at the level of both the rubbings and fonts so that the number of samples in the two domains is also equal. Through the above steps, balanced sampling can be achieved at the domain level and the character category level.

[0081] (3) After sampling a batch of data using the balanced sampling strategy, the specific process of the optimization is as follows: First, the oracle bone rubbings and character mold images are encoded using the pre-trained CLIP visual encoder to obtain visual features Where d and c represent the domain and character category of the oracle bone image, respectively. At the same time, the pre-trained CLIP semantic encoder is used to encode the learnable prompt information corresponding to these oracle bone images to obtain the semantic features Where d and c represent the domain and character category of the prompt information respectively. Then, similar to the training process of CLIP, the contrastive learning loss function L con , taking the visual features and semantic features of the same domain and the same character category as positive sample pairs, shortening their distance in the feature space, and taking the visual features and semantic features of different domains or different character categories as negative sample pairs, pushing their distance in the feature space, thereby making the learnable hint information semantic features Visual features of the corresponding oracle bone images Align and optimize the semantics of learnable prompt information;

[0082] The loss function is as follows:

[0083]

[0084] Among them, L i is the contrast loss function that aligns visual features to the semantic space, L t It is a contrast loss function that aligns semantic features to the visual space. The formulas for the two are as follows:

[0085]

[0086]

[0087] Where B is the batch size, and the character categories corresponding to the i-th visual feature, the t-th semantic feature, and the j-th and q-th visual or semantic features in the batch are y i ,y t ,y j ,y q ∈C bIndicates the character category, and the corresponding domains are d i ,d t ,d j ,d q ∈{p,s}, the positive sample index sets corresponding to the i-th visual feature and the t-th semantic feature in the batch are P(i) and P(t) respectively, τ represents the temperature coefficient, and <·,·> represents the cosine similarity.

[0088] (4) Repeat the sampling and optimization steps using the balanced sampling strategy until the loss function L con No longer falling.

[0089] 4. Remove the domain prompt information to obtain domain-independent prompt information, and then encode it through the semantic encoder to obtain the domain-independent character semantics.

[0090] (1) After optimizing the learnable prompt information, replace the domain prompt information in the learnable prompt information Remove, get the domain-independent prompt "A photo of character”.

[0091] (2) Using the pre-trained CLIP semantic encoder, we can train the domain-independent prompt “A photo of The domain-independent character semantic features obtained by encoding "character" are denoted as T c .

[0092] 5. Supervise the visual encoder training through the intra-class shared feature learning branch and the inter-class difference feature learning branch.

[0093] (1) Using a random sampling strategy, we randomly sample n character categories from the batch and sample a rubbing image and a character template image for each category.

[0094] (2) The sampled rubbing image and character image are encoded using a visual encoder, and the visual features obtained are and The visual encoder is then divided into two branches for intra-class shared feature learning and inter-class difference feature learning to perform supervised training.

[0095] (3) In the intra-class shared feature learning branch, contrastive learning loss is used, the visual features of the rubbings or font images are used as anchors, the semantic features of the same character category are used as positive samples, and the semantic features of the other categories are used as negative samples, which shortens the distance between the positive samples and increases the distance between the negative samples. The semantic features are the domain-independent character semantic features T obtained in the previous step. c , and during the training process, the domain-independent semantic features are fixed and no parameters need to be updated to provide stable semantics.

[0096] In engineering implementation, the contrastive learning loss first stores the semantic features of all character categories during training, and then uses the cross entropy loss L ce It can be implemented cleverly, and the formula is as follows:

[0097]

[0098] Among them, B represents the batch size, q is the index of the positive sample, and d i ∈{p,s} represents the domain of the i-th sample in the batch, y i ,y j ,y q represents the character category corresponding to the visual or semantic features of the i, j, qth sample in the batch, C b Represents the set of character categories visible during training.

[0099] (4) In the inter-class difference feature learning branch, oracle bone rubbing images and character mold images are sampled as training samples through a random sampling strategy. The random sampling strategy refers to randomly sampling n character categories in each batch, and sampling one rubbing image and character mold image for each category. After encoding these images through a visual encoder, the N-Pair loss function L is used. np To supervise, the formula is as follows:

[0100]

[0101] Where n is the number of randomly sampled character categories in each batch, and y i ,y j represents the character category of the i,jth sample in the batch, p and s represent the rubbing and font domain respectively.

[0102] (5) Use a random sampling strategy to sample multiple batches of data, and supervise the visual encoder through the intra-class shared feature learning branch and the inter-class difference feature learning branch until the cross entropy loss L ce And N-Pair loss function L np When neither decreases, stop training.

[0103] 6. Use the trained visual encoder to achieve zero-shot oracle character recognition. The steps are as follows:

[0104] Given the glyph images of all oracle bone characters that need to be recognized, a glyph image library is formed;

[0105] Whenever there is a new category that needs to be identified, the font image of the new category is also put into the font image library;

[0106] Encode all the fonts in the font image library through the visual encoder to obtain their visual features;

[0107] Encode the rubbing image to be identified through a visual encoder to obtain its visual features;

[0108] Calculate the Euclidean distance between the visual features of the rubbing image to be identified and the visual features of all the fonts in the font image library;

[0109] The character image with the smallest Euclidean distance is found, and the character category of the character image is used as the classification result of the rubbing image to be recognized, thereby realizing zero-sample oracle bone character recognition.

[0110] Specifically: (1) Given the font images of all oracle bone characters of category C that need to be recognized, a font image library is formed.

[0111] (2) Whenever there is a new category C that needs to be identified n When C n The corresponding font images are also put into the font image library.

[0112] (3) Encode all the fonts in the font image library through a visual encoder to obtain their visual features.

[0113] (4) Encode the rubbing image to be identified through a visual encoder to obtain its visual features.

[0114] (5) Calculate the Euclidean distance between the visual features of the rubbing image to be identified and the visual features of all the fonts in the font image library.

[0115] (6) Find the character image with the smallest Euclidean distance, and use the character category of the character image as the classification result of the rubbing image to be recognized, thereby achieving zero-sample oracle bone character recognition.

[0116] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0117] The technical features of the above embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0118] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0119] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A semantically enhanced zero-shot oracle bone character recognition method, characterized in that: The method comprises: Step 1, construct learnable prompt information for oracle bone inscription images based on domain and character category; Step 2: introduce a contrastive language-image pre-training model, sample data through a balanced sampling strategy, and optimize the learnable prompt information to mine semantic information from oracle bone rubbings and character mold images; Step 3, removing the domain prompt information to obtain domain-independent prompt information, and then encoding it through a semantic encoder to obtain domain-independent character semantics; Step 4: supervise the visual encoder training through the intra-class shared feature learning branch and the inter-class difference feature learning branch; Step 5, using the trained visual encoder to achieve zero-shot oracle bone character recognition; The learnable prompt information form is "A photo of character”, in and Respectively represent the domain and character category of the oracle bone image, , and Respectively represent the rubbing and font domains, and , represents the set of character categories visible during training, and Respectively represent the number of tokens in the domain prompt information and character category prompt information, and Represents a token, which is a vector of the same dimension as the word vector in CLIP. , , Each oracle bone image in the training set needs to construct its learnable hint information according to its domain and character category; The contrastive language-image pre-training model introduced is CLIP, which constructs a shared multimodal feature space through image and text input, and uses the contrastive learning mechanism to shorten the distance between correctly matched image-text pairs in the feature space, while pushing away the distance between mismatched image-text pairs in the feature space; The balanced sampling strategy is as follows: Randomly sample a subset of character categories from the training character categories; Uniformly sample each category in the character category subset so that the number of samples for each category is equal; At the same time, uniform sampling is also performed at the level of the two domains, rubbings and character molds, so that the number of samples in the two domains is also equal; Through the above steps, balanced sampling at the domain level and character category level can be achieved; The specific process of the optimization is as follows: Use the pre-trained CLIP visual encoder to encode the oracle bone rubbings and character mold images to obtain visual features; At the same time, the pre-trained CLIP semantic encoder is used to encode the learnable prompt information to obtain semantic features; Similar to the training process of CLIP, the contrastive learning loss function is used , taking the visual features and semantic features of the same domain and the same character category as positive sample pairs, shortening their distance in the feature space, taking the visual features and semantic features of different domains or different character categories as negative sample pairs, pushing their distance in the feature space, thereby aligning the semantic features of the learnable prompt information with the visual features of the corresponding oracle bone inscription image, and optimizing the semantics of the learnable prompt information; The loss function is as follows: ； in, It is a contrast loss function that aligns visual features to the semantic space. It is a contrast loss function that aligns semantic features to the visual space. The formulas for the two are as follows: ； in is the batch size, the visual features, semantic features, and and The character categories corresponding to the visual or semantic features are Indicates character categories, and the corresponding domains are , the first batch The visual features and The positive sample index sets corresponding to the semantic features are and , represents the temperature coefficient, Represents cosine similarity.

2. A semantically enhanced zero-shot oracle bone character recognition method as claimed in claim 1, characterized in that: The removal of the domain prompt information to obtain the domain-independent prompt information is to remove the domain prompt information in the learnable prompt information after optimizing the learnable prompt information in step 2. Remove, get the domain-independent prompt "A photo of character”.

3. The semantically enhanced zero-shot oracle bone character recognition method according to claim 1, characterized in that: In the intra-class shared feature learning branch, the rubbing image and the character mold image are first encoded using a visual encoder to obtain visual features, and then contrastive learning loss is used to use the visual features of the rubbing or character mold image as anchors, the semantic features of the same character category are used as positive samples, and the semantic features of the remaining categories are used as negative samples, so as to shorten the distance between the positive samples and extend the distance between the negative samples. The semantic feature is the domain-independent character semantics obtained in step 3, and in the training process of step 4, the domain-independent semantic feature is fixed and does not need to update parameters to provide stable semantics.

4. The semantically enhanced zero-shot oracle bone character recognition method according to claim 1, characterized in that: In the inter-class difference feature learning branch, oracle bone rubbing images and character mold images are sampled as training samples through a random sampling strategy. After encoding these images through a visual encoder, the N-Pair loss function is used to To supervise, The random sampling strategy mentioned above refers to random sampling in each batch character categories, and sample a rubbing image and a character mold image for each category. The N-Pair loss function , the formula is as follows: ； in, is the number of randomly sampled character categories in each batch, , Indicates the first , The character categories of samples, and Respectively represent the rubbing and font domain, Represented as the rubbing image The domain corresponding to visual features, Represented as the font image The domain corresponding to visual features, Represented as the font image The domain corresponding to visual features, T is the transposition symbol.

5. The semantically enhanced zero-shot oracle bone character recognition method according to claim 1, characterized in that: The zero-sample oracle bone character recognition is achieved by visual matching by the visual encoder after training in step 4, and the steps are as follows: Given the glyph images of all oracle bone characters that need to be recognized, a glyph image library is formed; Whenever there is a new category that needs to be identified, the font image of the new category is also put into the font image library; Encode all the fonts in the font image library through the visual encoder to obtain their visual features; Encode the rubbing image to be identified through a visual encoder to obtain its visual features; Calculate the Euclidean distance between the visual features of the rubbing image to be identified and the visual features of all the fonts in the font image library; The character image with the smallest Euclidean distance is found, and the character category of the character image is used as the classification result of the rubbing image to be recognized, thereby realizing zero-sample oracle bone character recognition.

Images