Character detection method and apparatus, computer device and storage medium

Abstract

The present disclosure relates to the technical field of image recognition, and provides a character detection method and apparatus, a computer device, and a storage medium. The character detection method of the present disclosure comprises: inputting multiple acquired open-set samples into a trained interpretable target detection network, to obtain character detection results corresponding to the open-set samples; on the basis of local features and global features of the open-set samples corresponding to the respective character detection results, clustering the open-set samples, to obtain clustering results of the open-set samples, the clustering results being used to represent categories of the open-set samples whose categories are to be verified; on the basis of the local features and the global features of the open-set samples, and local features and global features of multiple characters pre-stored in the target detection network, determining a first character feature corresponding to each category of characters among the open-set samples; and on the basis of the first character feature, providing an interpretability support for the clustering result of the open-set samples corresponding to the first character feature, the interpretability support comprising an image feature and a semantic feature corresponding to the first character feature.

Description

Character detection method, device, computer equipment and storage medium Technical Field

[0001] The present disclosure belongs to the field of image recognition technology, and in particular relates to a character detection method, apparatus, computer equipment, and storage medium. Background Art

[0002] Pictographic characters, an ancient form of linguistic expression that uses pictographic symbols to represent objects, concepts, and actions, played a vital role in ancient record-keeping, communication, and cultural heritage. Some ethnic groups, such as the Dongba script used by the Naxi, continue to use pictographic characters today. Oracle bone script, a representative type of pictographic characters, was widely used from the late Shang Dynasty to the Spring and Autumn Period. It is the earliest systematic and mature writing system in ancient China and holds a profound historical and cultural heritage. Oracle bone script not only records the lives of ancient people and social changes, but also provides invaluable data for research in fields such as archaeology, history, and linguistics. However, fewer than 5,000 oracle bone characters have been discovered, of which only between 1,500 and 2,000 can be deciphered. Even within deciphered oracle bone inscriptions, the meanings of some characters remain uncertain. Incomplete materials and the difficulty of deciphering them limit the efficiency and accuracy of traditional manual recognition methods.

[0003] Summary of the Invention

[0004] The present disclosure aims to solve at least one of the technical problems existing in the prior art and provides a character detection method, apparatus, computer device and storage medium.

[0005] In a first aspect, the technical solution adopted to solve the technical problem of the present disclosure is a character detection method, comprising:

[0006] Inputting the obtained multiple open set samples into a trained and interpretable object detection network to obtain character detection results corresponding to the open set samples; the open set samples are pre-processed pictographic character images of the to-be-verified category and to-be-decoded;

[0007] Clustering the open set samples according to the local features and global features of the open set samples corresponding to the respective character detection results to obtain a clustering result of the open set samples; the clustering result is used to characterize the category of the open set samples of the category to be verified;

[0008] Based on the local features and global features of the open set samples, as well as the local features and global features of multiple characters pre-stored in the target detection network, the first character features corresponding to each type of characters in the open set samples are determined, and based on the first character features, explainability support is provided for the clustering results of the corresponding open set samples; wherein the explainability support includes image features and semantic features corresponding to the first character features.

[0009] In some embodiments, clustering the open set samples according to the local features and global features of the open set samples corresponding to each of the character detection results to obtain a clustering result of the open set samples includes:

[0010] Splicing the local features and the global features of the open set samples to obtain initial features;

[0011] Normalizing the initial features to obtain target features of the open set samples;

[0012] The open set samples are clustered according to target features of the open set samples to obtain clustering results of the open set samples.

[0013] In some embodiments, clustering the open set samples according to the target features of the open set samples to obtain the clustering results of the open set samples includes:

[0014] For each of the multiple to-be-verified categories, randomly sampling a first preset number of open set samples in the to-be-verified category, and using the remaining open set samples as test samples for zero-sample clustering;

[0015] Using the target features of the first preset number of open set samples as a priori cluster centers;

[0016] Using a preset similarity algorithm, the distance between the test sample and each cluster center under each category to be verified is obtained;

[0017] For each of the test samples, taking the categories of the second preset number of cluster centers that are closest to the test sample as candidate categories;

[0018] The voting algorithm is used to calculate the number of votes for each candidate category, and the candidate category with the highest number of votes is used as the clustering result of the test sample.

[0019] In some embodiments, after obtaining the clustering result of the test sample, the method further includes:

[0020] According to the confidence level indicated by the clustering result of each test sample, the average correct rate of the clustering results of all the test samples is determined as the clustering accuracy rate under the zero-sample setting.

[0021] In some embodiments, determining the first character feature corresponding to each type of character in the open set of samples based on the local features and global features of the open set of samples and the local features and global features of multiple characters pre-stored by the object detection network includes:

[0022] Determining a first similarity matrix group based on the local features and global features of the open set samples and the local features and global features of the plurality of characters pre-stored by the object detection network; the first similarity matrix group includes first similarity matrices corresponding to the plurality of characters pre-stored by the object detection network; the first similarity matrix represents similarity information between one of the open set samples and one of the characters;

[0023] According to the first similarity matrix group, the first similarity matrices that meet the first preset condition are screened out, and the characters corresponding to the screened out first similarity matrices are used as the first character features corresponding to each type of characters in the open set samples.

[0024] In some embodiments, screening out the first similarity matrices that meet a first preset condition based on the first similarity matrix group, and using the characters corresponding to the screened out first similarity matrices as the first character features corresponding to each type of characters in the open set of samples, includes:

[0025] For the first similarity matrix corresponding to each of the characters, determining an average similarity of the first similarity matrix according to the values ​​of each element in the first similarity matrix;

[0026] For the average similarities of the multiple first similarity matrices, a third preset number of average similarities are selected in descending order, and the characters corresponding to the selected average similarities are used as the first character features corresponding to each type of characters in the open set samples.

[0027] In some embodiments, providing interpretable support for the clustering results of the open set samples corresponding to the first character feature based on the first character feature includes:

[0028] Using the second similarity matrix between the local features and the global features of the open set samples and each of the first character features as a first activation matrix;

[0029] For any of the first activation matrices, upsampling the first activation matrix to obtain an upsampled first activation matrix; a larger value of an element in the upsampled first activation matrix indicates that the feature of the open set sample is more similar to the first character feature;

[0030] According to each of the upsampled first activation matrices, first character features similar to the open set samples are screened out, and the first character features similar to the open set samples are used to provide interpretable support for the clustering results of the corresponding open set samples.

[0031] In some embodiments, screening out first character features similar to the open set samples based on each of the upsampled first activation matrices, and using the first character features similar to the open set samples to provide interpretable support for clustering results of the corresponding open set samples includes:

[0032] Selecting a fourth preset number of first character features in descending order according to the maximum element value in each of the upsampled first activation matrices;

[0033] Based on the image features and semantic features corresponding to the fourth preset number of first character features, explainability support is provided for the clustering results of the corresponding open set samples.

[0034] In some embodiments, after providing interpretable support for the clustering result of the corresponding open set samples based on the first character feature, the method further includes:

[0035] For open set samples belonging to the same category, summing the element values ​​of P first similarity matrices in the first similarity matrix group to obtain P summation results; the first similarity matrix group corresponding to the open set samples includes P first similarity matrices; P is a positive integer;

[0036] For open set samples belonging to the same category, based on the summation results, the first similarity matrix that meets the second preset condition is screened, and the characters corresponding to the screened first similarity matrix are used as the second character features corresponding to each type of characters in the open set samples belonging to the same category, and based on the second character features, explainability support is provided for the open set samples belonging to the same category; the explainability support includes image features and semantic features corresponding to the second character features.

[0037] In some embodiments, after determining the second character feature corresponding to each character in the open set of samples belonging to the same category, the method further includes:

[0038] According to the category to which the closed set sample belongs, determining that the second character feature corresponds to the original category in the closed set sample, and determining the target closed set sample according to the original category;

[0039] Compare the open set samples of the category corresponding to the second character feature with the target closed set samples, and provide deciphering reference information for the open set samples of the category corresponding to the second character feature based on the comparison result; the deciphering reference information includes image features and semantic features of closed set samples similar to the open set samples.

[0040] In some embodiments, the step of training an interpretable target detection network includes:

[0041] Obtain closed-set character samples and open-set character samples; the closed-set character samples are pictographic character images with known labeled categories; the open-set character samples are pictographic character images with categories to be verified and to be deciphered;

[0042] Processing the closed-set character samples and the open-set character samples to obtain closed-set samples and open-set samples;

[0043] Inputting a plurality of the closed set samples and a plurality of the open set samples into an interpretable target detection network for training, thereby obtaining a trained target detection network.

[0044] In some embodiments, processing the closed-set character samples and the open-set character samples to obtain closed-set samples and open-set samples includes:

[0045] Preprocessing the closed-set character samples and the open-set character samples to determine initial closed-set samples and initial open-set samples that meet a preset size;

[0046] Randomly perturb the initial closed set samples and the initial open set samples to obtain augmented closed set samples and open set samples.

[0047] In some embodiments, inputting the plurality of closed set samples and the plurality of open set samples into an interpretable object detection network for training to obtain a trained object detection network includes:

[0048] Inputting a plurality of the closed set samples and a plurality of the open set samples into an interpretable target detection network, obtaining a second similarity matrix group corresponding to each of the closed set samples and each of the open set samples, which is output by an intermediate segment of the target detection network, and a sample prediction result finally output by the target detection network; the closed set samples or the open set samples may belong to K categories, each category pre-stored with P characters; K and P are both positive integers; the second similarity matrix group includes third similarity matrices corresponding to each of the plurality of characters, which are pre-stored; the third similarity matrix represents similarity information between one of the open set samples or one of the closed set samples and one of the characters;

[0049] For each category, the entropy corresponding to the category is determined based on the third similarity matrix of the P characters, and the entropies corresponding to the categories constitute the global grouping entropy;

[0050] For each category, determining an average similarity value based on the third similarity matrix of the P characters;

[0051] Determine the average global entropy based on the average similarity values ​​corresponding to the K categories;

[0052] Determining an entropy constraint of the object detection network based on the global group entropy and the average global entropy;

[0053] According to the classification loss and entropy constraint indicated by the sample prediction result, a loss function of the target detection network is determined, and the network update is supervised according to the loss function until the target detection network converges, thereby obtaining the trained target detection network.

[0054] In some embodiments, the network architecture of the object detection network includes an embedding coding module, a feature extraction module, a prototype module, and a prediction category module;

[0055] Inputting the plurality of closed set samples and the plurality of open set samples into an interpretable target detection network, obtaining a second similarity matrix group output by an intermediate segment of the target detection network and corresponding to each of the closed set samples and each of the open set samples, and a sample prediction result finally output by the target detection network, including:

[0056] Processing the plurality of closed-set samples and the plurality of open-set samples in sequence through the embedded coding module, the feature extraction module, and the prototype module to obtain a second similarity matrix group output by the prototype module and corresponding to each of the closed-set samples and each of the open-set samples;

[0057] Each third similarity matrix in each second similarity matrix group is input into the prediction category module to obtain the sample prediction results corresponding to each closed set sample and each open set sample respectively.

[0058] In a second aspect, an embodiment of the present disclosure further provides a character detection device, comprising a processing module, a clustering module, and a parsing module;

[0059] The processing module is configured to input the obtained multiple open set samples into a trained and interpretable object detection network to obtain character detection results corresponding to the open set samples; the open set samples are pre-processed pictographic character images of the to-be-verified category and to-be-deciphered;

[0060] The clustering module is configured to cluster the open set samples according to the local features and global features of the open set samples corresponding to each of the character detection results to obtain a clustering result of the open set samples; the clustering result is used to characterize the category of the open set samples of the category to be verified;

[0061] The parsing module is configured to determine the first character feature corresponding to each type of character in the open set samples based on the local features and global features of the open set samples, as well as the local features and global features of multiple characters pre-stored in the target detection network, and provide explainability support for the clustering results of the corresponding open set samples based on the first character feature; wherein the explainability support includes image features and semantic features corresponding to the first character feature.

[0062] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the character detection method described in any one of the first aspects are performed.

[0063] In a fourth aspect, an embodiment of the present disclosure further provides a computer non-volatile readable storage medium having a computer program stored thereon, which executes the steps of the character detection method as described in any one of the first aspects when the computer program is executed by a processor. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] FIG1 is a flow chart of a character detection method provided by an embodiment of the present disclosure;

[0065] FIG2 is a system architecture diagram of a character detection method provided by an embodiment of the present disclosure;

[0066] FIG3 is a specific flow chart of the character detection method provided by an embodiment of the present disclosure;

[0067] FIG4 is a schematic diagram of a specific flow chart of a character detection method provided by an embodiment of the present disclosure;

[0068] FIG5 is a schematic diagram of a character detection device provided by an embodiment of the present disclosure;

[0069] FIG6 is a schematic structural diagram of a computer device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0070] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0071] Unless otherwise defined, the technical or scientific terms used in this disclosure should have the usual meanings understood by people with ordinary skills in the field to which this disclosure belongs. The words "first", "second" and similar words used in this disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, words such as "one", "an" or "the" do not indicate a quantity limitation, but rather indicate the existence of at least one. Words such as "include" or "comprise" mean that the elements or objects appearing before the word include the elements or objects listed after the word and their equivalents, without excluding other elements or objects. Words such as "connect" or "connected" are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. "Up", "down", "left", "right" and the like are only used to indicate relative positional relationships. When the absolute position of the object being described changes, the relative positional relationship may also change accordingly.

[0072] In this disclosure, "multiple or several" refers to two or more. "And / or" describes the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0073] Deep learning models have revolutionized visual recognition tasks, even surpassing average human performance in certain tasks. However, in many practical applications, there remains a need to integrate visual recognition technology with complex scenarios. Research on computer image recognition technology is closely aligned with the study of ancient characters, and oracle bone inscriptions can be considered a special type of image. Oracle bone inscriptions with known annotations and used in model training correspond to "closed-set hieroglyphics," while those with yet-to-be-verified categories and deciphered characters correspond to "open-set hieroglyphics." However, oracle bone inscription images present unique challenges in terms of their differentiation from other images. These ancient hieroglyphics developed from the late Shang Dynasty to the Spring and Autumn Period, covering a wide range of time and geographical distribution. This results in significant variation within oracle bone inscriptions, resulting in complex and diverse morphology and structure. Furthermore, oracle bone inscription characters are scarce and exhibit a pronounced long-tail distribution, making common visual recognition models face significant challenges when processing oracle bone inscription images.

[0074] Traditional technologies have made some progress in the recognition of oracle bone inscriptions. However, these recognition methods still have some limitations. For example, traditional pictographic character recognition is usually divided into character recognition focusing on pixel matrices or character recognition focusing on vector sequence formats. Among them, the recognition of pixel matrix images mainly adopts models based on convolutional networks or twin networks. The recognition of vector sequence formats mainly adopts BERT technology (mainly applying the bidirectional training of the popular attention model Transformer to language modeling) to remodel pictographic characters to achieve the purpose of data amplification.

[0075] However, most previous character recognition methods either simplified the task settings of oracle bone inscriptions, ignoring task pain points such as small data volume or data diversity, or did not fully utilize the differences between oracle bone inscription image data and natural image data in terms of methods, such as extreme sparsity, lack of texture and color, and the expression of complex semantic information by a limited combination of simple structures. Therefore, they have no practical significance.

[0076] More importantly, existing deep learning methods for oracle bone script recognition often lack strong interpretability. While some methods have achieved high accuracy, their internal decision-making processes are often difficult to explain and understand. This is a significant issue for experts such as archaeologists and professional calligraphers, who need to verify and interpret recognition results. Models lacking interpretability make it difficult for experts to provide a robust reference, limiting the practical application of oracle bone script recognition technology.

[0077] In view of this, embodiments of the present disclosure provide a character detection method, specifically, under the condition of extremely sparse labeled examples, object-shaped character detection is realized, and the interpretability of the detection results is ensured.

[0078] FIG. 1 is a flowchart of a character detection method provided by an embodiment of the present disclosure. As shown in FIG. 1, the character detection method includes steps S11 to S14.

[0079] S11. Input a plurality of open-set samples obtained into a trained and interpretable object detection network to obtain character detection results corresponding to the open-set samples.

[0080] In this step, the open-set samples are preprocessed object-shaped character images of the categories to be verified and to be deciphered. Object-shaped characters include, but are not limited to, oracle bone inscriptions, Dongba characters, etc. Usually, there are only extremely sparse labeled examples, such as known labeled and extremely sparse closed-set samples that can participate in model training. Among them, the categories to which the open-set samples belong are the categories to be verified, which are predicted and labeled according to actual experience and need to be verified and calibrated subsequently.

[0081] The training samples for training the object detection network model include open sets and closed sets. The open set includes open-set samples, and the closed set includes closed-set samples. The interpretable object detection network is a model with a local attention mechanism, which can capture the global and local features of object-shaped characters, can better understand the decision-making basis of the model, and make the recognition process more transparent and supervised. At the same time, the present disclosure also focuses on interpretability to provide a clear explanation for each decision in the process of object-shaped character recognition.

[0082] In this step, the character detection results mainly include the prediction results of the open-set samples. For example, when inputting an oracle bone inscription image of "person", the probability of outputting the modern Chinese character result of "person" is a%, and the probability of outputting the modern Chinese character result of "enter" is b%, a + b = 100.

[0083] Exemplarily, the object detection network is composed of an embedding encoding module, a feature extraction module, a prototype module, and a prediction category module. For a plurality of open-set samples, they are sequentially processed by the embedding encoding module, the feature extraction module, and the prototype module. Through the feature extraction module, the local and global features of the object-shaped characters in the open-set samples after feature extraction can be obtained. Through the prototype module, a first similarity matrix group can be obtained, and through the prediction category module, the prediction results (i.e., probabilities) of the open-set samples can be obtained.

[0084] S12. Cluster the open-set samples according to the local and global features of the open-set samples corresponding to each character detection result to obtain the clustering result of the open-set samples.

[0085] One open-set sample corresponds to one character detection result. Multiple open-set samples are input into the object detection network to obtain multiple character detection results. The clustering result is used to characterize the category of the open-set samples of the category to be verified.

[0086] Specifically, some open-set samples can be randomly selected from multiple open-set samples as test samples; in the zero-shot setting, according to the local and global features of the open-set samples corresponding to each character detection result, KNN clustering is performed on the test samples to obtain the clustering result.

[0087] The clustering result can characterize the predicted category of the pictographic characters corresponding to the open-set samples. Personnel in related fields can classify or label the pictographic characters of the category to be verified in the open-set samples according to the predicted category, so as to define them as accurate categories, or it can also be used for model training to expand the closed-set sample size. Or, the predicted category can also provide interpretability support for the deciphering of personnel in related fields.

[0088] S13. Determine the first character feature corresponding to each category of characters in the open-set sample according to the local and global features of the open-set sample, and the local and global features of multiple characters pre-stored in the object detection network.

[0089] The open-set samples may belong to K categories, such as K modern Chinese character texts. P prototypes are stored in the object detection network for each category in the object detection network. The so-called "prototype" is a term in deep learning and is a highly abstract feature used to represent a set of learnable parameters in space. In the present disclosure, it refers to the feature parameters of the learnable pictographic characters in the oracle bone inscription images, and can also be understood as the manifestation of the local and global features of the characters. Therefore, for the convenience of understanding and distinction, the following text of the present disclosure uses the expression of the first prototype to replace the expression of "the local and global features of multiple characters pre-stored in the object detection network".

[0090] Specifically, take out the first prototype stored in the object detection network, compare it with the local and global features of the open-set sample, and screen out the eligible first prototype as the first character feature corresponding to each category of characters in the open-set sample. Among them, the first character feature corresponding to a category of characters refers to the same character feature possessed by a category of characters. For example, the feature of a category of characters represented by "犭" is the animal feature; the first character feature corresponding to each category of characters in the open-set sample refers to which first character features the pictographic characters in the open-set sample possess.

[0091] S14. Provide interpretability support for the clustering result of the open-set sample corresponding to the first character feature.

[0092] Among them, the interpretability support includes the image feature and semantic feature corresponding to the first character feature.

[0093] Since the essence of the first character feature is a prototype, in order to distinguish it from the first prototype and for ease of understanding, the first character feature will be denoted as the second prototype in the following description of the present disclosure.

[0094] One open-set sample corresponds to multiple second prototypes Here, p represents the second prototype, M represents the number of second prototypes, and i represents the i-th one. In this way, using the relevant information of multiple second prototypes can provide interpretability support for the clustering results of the corresponding open-set samples. The relevant information of the second prototype [[ID=​​​​​​​​​​​​​​The present disclosure provides an interpretable pictographic character understanding method based on extremely sparse annotated samples. This method utilizes a trained, interpretable target detection network to identify and classify open-set pictographic character samples, and uses the first prototypes stored in the network to provide interpretable support for clustering results. Compared to traditional methods, the present disclosure utilizes an interpretable target detection network, making the pictographic character recognition process more transparent and supervised. By introducing global and local first prototypes, the model can explain each decision, providing a clear basis for the recognition of each character. The interpretability of this model with a local attention mechanism not only conforms to the structural composition of pictographic characters and the visual laws of human understanding of pictographic characters, helping to improve recognition accuracy, but also provides a reliable interpretive reference for experts in fields such as archaeologists and professional calligraphers, enabling them to better understand the recognition results, thereby improving the reliability and practicality of the technology. Furthermore, the present disclosure enables interpretable recognition and deciphering of novel pictographic characters. This provides new possibilities for archaeologists and researchers, enabling them to better understand and interpret the meaning and cultural value behind unknown pictographic characters, expanding the breadth and depth of research.

[0098] In some embodiments, the screening process of the open set of samples specifically includes S121 to S123.

[0099] S121. Concatenate the local features and global features of the open set samples to obtain initial features.

[0100] Specifically, given the local features and global features, the Concatenate algorithm is used to concatenate the two to obtain the initial feature F0.

[0101] S122. Normalize the initial features to obtain target features of the open set samples.

[0102] Specifically, a normalization algorithm is used to normalize the initial features to obtain the target feature F1 of the open set samples.

[0103] S123. Clustering the open set samples according to the target features of the open set samples to obtain a clustering result of the open set samples.

[0104] Specifically, we first need to determine the a priori cluster centers. Then, we use the KNN clustering method to cluster the open-set samples based on the a priori cluster centers and the target features of the open-set samples, obtaining the clustering results for the open-set samples. For example, we can randomly select open-set samples and use their target features as the a priori cluster centers. Alternatively, we can define our own a priori cluster centers.

[0105] Exemplarily, the KNN clustering process specifically includes steps S123 - 1 to S123 - 5 .

[0106] S123 - 1 . For each of the multiple categories to be verified, randomly sample a first preset number of open set samples from the category to be verified, and use the remaining open set samples as test samples for zero-sample clustering.

[0107] The first preset number can be set according to actual conditions and experience, and is not specifically limited in the embodiment of the present disclosure. The present disclosure selects 5 as the first preset number.

[0108] For example, there are m categories to be verified. For any of these categories, a first preset number of open set samples are randomly sampled, and their target features are used as cluster centers. For example, for one category to be verified, 5 open set samples are randomly selected; for m categories to be verified, 5×m open set samples need to be selected and 5×m cluster centers need to be determined.

[0109] S123-2. Use the target features of the first preset number of open set samples as a priori cluster centers.

[0110] S123-3. Using a preset similarity algorithm, obtain the distance between the test sample and each cluster center under each category to be verified.

[0111] In this step, the preset similarity algorithm may be a cosine similarity algorithm, which is used to measure the distance between the test sample and the cluster center. The closer the distance, the greater the possibility that the test sample belongs to the category of the open set of samples corresponding to the cluster center.

[0112] This step can obtain the distance between the test sample and 5×m cluster centers in total.

[0113] S123-4. For each test sample, the categories of the second preset number of cluster centers that are closest to the test sample are selected as candidate categories.

[0114] The second preset number can be set according to actual conditions and experience, and is not specifically limited in the embodiment of the present disclosure. The present disclosure selects the second preset number to be 5.

[0115] For the 5×m cluster centers, 5 cluster centers are selected in order from small to large, and the categories of the 5 cluster centers are used as candidate categories of the test samples.

[0116] S123-5. Calculate the number of votes for each candidate category using a voting algorithm; and use the candidate category with the highest number of votes as the clustering result of the test sample.

[0117] Among them, the voting algorithm is a machine learning classification method.

[0118] In some embodiments, the clustering result is not only used to characterize the category of the open set samples of the category to be verified, but also used to indicate the confidence of the current clustering result, that is, the accuracy of the predicted category.

[0119] After obtaining the clustering results for the test samples, the average accuracy of the clustering results for all test samples can be determined based on the confidence level indicated by the clustering results for each test sample, which serves as the clustering accuracy under the zero-shot setting. The output clustering accuracy can then be used to determine the accuracy of the model in predicting open-set pictographic characters.

[0120] In some embodiments, the process of determining the second prototype of the open set samples specifically includes S131 to S132.

[0121] S131. Determine a first similarity matrix group based on the local features and global features of the open set samples and the local features and global features of multiple characters pre-stored by the target detection network.

[0122] The first similarity matrix group includes first similarity matrices s corresponding to multiple first prototypes pre-stored in the target detection network; the first similarity matrix represents the similarity information between an open set sample and a first prototype. It can be seen as the intersection of the between-category dimension and the within-category prototype dimension.

[0123] An open set sample corresponds to a first similarity matrix group, and a first similarity matrix group includes first similarity matrices between the open set sample and multiple first prototypes; a first similarity matrix is ​​composed of multiple first similarities between the open set sample and a first prototype.

[0124] The prototype module in the target detection network can calculate the first similarity matrix between the open set samples and each first prototype based on the local features and global features of the open set samples and each first prototype through a special similarity algorithm, thereby obtaining the first similarity matrix group corresponding to the open set samples.

[0125] S132: Filter out the first similarity matrices that meet a first preset condition according to the first similarity matrix group, and use the characters corresponding to the filtered out first similarity matrices as first character features corresponding to each type of characters in the open set samples.

[0126] In one possible implementation, the first preset condition may be a judgment condition for selecting a plurality of highest average similarities. Specifically, for each first prototype corresponding to a first similarity matrix, the average similarity of the first similarity matrix is ​​determined based on the values ​​of each element in the first similarity matrix; for the average similarities of multiple first similarity matrices, a third preset number of average similarities are selected in descending order, and the first prototypes corresponding to the selected average similarities are used as the second prototypes of the open set samples.

[0127] Exemplarily, the mean of the values ​​of each element in each first similarity matrix is ​​calculated and recorded as the average similarity of the first similarity matrix. For the average similarities of multiple first similarity matrices, based on the size of the average similarities, starting from the maximum average similarity, in descending order, a third preset number of average similarities are searched in sequence. The third preset number can be set based on actual application and experience, and is not specifically limited in the embodiment of the present disclosure. The first prototype corresponding to the selected average similarity is also the first prototype corresponding to the first similarity matrix corresponding to the selected average similarity.

[0128] In another possible implementation, the first preset condition may also be a judgment condition for selecting an average similarity greater than a preset value. Specifically, for the first similarity matrix corresponding to each first prototype, the average similarity of the first similarity matrix is ​​determined based on the values ​​of each element in the first similarity matrix; for the average similarities of multiple first similarity matrices, the average similarity greater than the preset value is selected based on the size of the average similarity, and the first prototype corresponding to the selected average similarity is used as the second prototype of the open set sample. Here, the preset value can be set according to actual application and experience, and is not specifically limited in the embodiment of the present disclosure.

[0129] In some embodiments, for step S14, interpretability support is provided for the clustering results of the corresponding open set samples based on each second prototype, which specifically includes S141 to S143.

[0130] S141. Using the second similarity matrix between the local features and global features of the open set samples and each second prototype as a first activation matrix.

[0131] Among them, the local features and global features of the open set samples are the features extracted by the feature extraction module (in the form of feature vectors).

[0132] Use similarity algorithm to determine the open set samples and each second prototype The second similarity matrix between them is used to obtain M second similarity matrices; the second similarity matrix is ​​used as the first activation matrix to obtain M first activation matrices, that is, the first activation matrix between the open set sample and each second prototype.

[0133] S142. For any first activation matrix, upsample the first activation matrix to obtain an upsampled first activation matrix.

[0134] Among them, the larger the value of the element in the upsampled first activation matrix, the more similar the characteristics of the open set sample are to the second prototype.

[0135] The first activation matrix is ​​mapped upsampled to the size of the original ideographic character (open set sample). The higher the value in the upsampled first activation matrix, the higher the similarity between the region corresponding to the open set sample and the corresponding second prototype, which means that the features contained in this second prototype play a more important role in clustering.

[0136] S143. According to each upsampled first activation matrix, a second prototype similar to the open set sample is screened out, and the second prototype similar to the open set sample is used to provide interpretable support for the clustering results of the corresponding open set samples.

[0137] According to each upsampled first activation matrix, the second prototype similar to the open set sample can be screened out as the second prototype similar to the open set sample M′ represents the number of second prototypes similar to the open set samples. The second prototypes similar to the open set samples are used to provide interpretable support for the clustering results of the corresponding open set samples.

[0138] In a possible implementation, the second prototypes similar to the open set samples are screened out. Specifically, a fourth preset number of second prototypes can be selected in descending order according to the maximum element value in each upsampled first activation matrix. M′ represents a fourth preset number, wherein the fourth preset number can be set according to actual conditions and experience, and is not specifically limited in the embodiment of the present disclosure.

[0139] Afterwards, the second prototype may be The corresponding image features and semantic features provide interpretable support for the clustering results of the corresponding open set samples.

[0140] In some embodiments, after providing explainability support for the corresponding open set samples based on the second prototype, it also includes S21 to S22, wherein: S21, for open set samples belonging to the same category, the element values ​​of the P first similarity matrices in the first similarity matrix group are summed respectively to obtain P summation results; the first similarity matrix group corresponding to the open set samples includes P first similarity matrices; P is a positive integer.

[0141] There are Q open-set samples belonging to the same category. Each open-set sample corresponds to a group of first similarity matrices, with a total of Q groups of first similarity matrices. Each group of first similarity matrices has P first similarity matrices, with a total of Q×P first similarity matrices. The elements in each first similarity matrix are summed, resulting in a total of Q×P summation results. Q is a positive integer.

[0142] S22. For the open-set samples belonging to the same category, according to each summation result, filter the first similarity matrices that meet the second preset condition, and use the characters corresponding to the filtered first similarity matrices as the second character features corresponding to each category of characters in the open-set samples belonging to the same category, and provide interpretability support for the open-set samples belonging to the same category based on the second character features.

[0143] Among them, the first character feature corresponding to a category of characters refers to the same character feature possessed by a category of characters. For example, the feature of a category of characters represented by "犭" is an animal feature; the first character features corresponding to each category of characters in the open-set samples refer to which first character features are possessed by the pictographic characters in the open-set samples. The second character features corresponding to each category of characters in the open-set samples belonging to the same category refer to the common features of the pictographic characters in the open-set samples belonging to the same category.

[0144] Since the second character feature refers to the prototype, in order to distinguish it from the first prototype and the second prototype and for easy understanding, the second character feature will be denoted as the third prototype in the following text of this disclosure for description. <00003~07><00~00308>Among the Q×P summation results, filter the first similarity matrices that meet the second preset condition. In one possible implementation, the second preset condition can be a judgment condition for selecting several highest summation results. Specifically, for the values indicated by each summation result among the Q×P summation results, in descending order, select several summation results in sequence, and use the first prototypes corresponding to the selected summation results as the third prototypes of the open-set samples belonging to the same category.

[0146] One category to be verified corresponds to multiple third prototypes <00~00311> Among them, p represents the third prototype, N represents the number of third prototypes, and i represents the i-th one. In this way, using the relevant information of multiple third prototypes can provide interpretability support for a group of open-set samples of the category to be verified. The relevant information of the third prototype includes but is not limited to the image features and semantic features noted by the third prototype. [[ID=~22]]

[0147] The third prototype is the first prototype with a relatively high feature similarity to the open-set samples belonging to the same category. Then, multiple third prototypes To a certain extent, it can be used as a structural reference for a class of pictographic characters of an open set of samples, and can also be considered as the main structure of pictographic characters that constitute a class of open set samples. Therefore, the third prototype is used The image features and semantic features can give a reasonable explanation for a class of pictographic characters in open set samples to a certain extent.

[0148] In some embodiments, after determining the third prototype of the open set samples belonging to the same category, multiple third prototypes can be further traced back. The original corresponding category in the closed set; and compare the structural features of the open set samples and closed set samples of the same prototype part, infer the interpretation of the open set samples, and provide a reference for deciphering novel pictographic characters.

[0149] Specifically, based on the category to which the closed set sample belongs, the original category in the closed set sample corresponding to the third prototype is determined. For example, it is known that there are K categories to which the closed set sample may belong, and each category has P first prototypes. The first prototype that is the same as the third prototype among the K×P first prototypes is determined, and the category p to which the first prototype belongs is determined, that is, the original category in the closed set sample corresponding to the third prototype. p∈K, p is a positive integer. Afterwards, the target closed set sample can be determined based on the original category. Since the category of the closed set sample is known, after determining the original category, a closed set sample of the same category is found as the target closed sample. Afterwards, the local features and global features of the open set sample of the category corresponding to the third prototype and the target closed sample are compared, and based on the comparison results, deciphering reference information is provided for the open set sample of the category corresponding to the third prototype; the deciphering reference information includes the image features and semantic features of the closed sample similar to the open set sample.

[0150] Essentially, the method compares the similarity of local and global features between open-set samples and target closed-set samples with the same category prototype. Based on the similarity comparison results, it finds target closed-set samples similar to the open-set samples. Based on the annotations of the found target closed-set samples, it queries the semantic information of the decomposition structure of the pictographic characters. Similar simple structures are likely to map to the same semantic information. By referring to the semantic information of the local decomposition structure of similar target closed samples, it infers the semantic information of similar structures in the open-set samples. Combining the semantic information of the local decomposition structure of multiple target closed samples provides a reference for deciphering novel open-set samples.

[0151] The disclosed embodiments enable interpretable identification and deciphering of novel pictographic characters, providing new possibilities for archaeologists and researchers to better understand and interpret the meaning and cultural value of unknown pictographic characters, expanding the breadth and depth of research.

[0152] For open-set sample detection, this paper utilizes a trained, interpretable object detection network to identify and classify open-set pictographic character samples, and uses the first prototype stored in the network to provide interpretable support for the clustering results. It can even use the first prototype as an intermediate link, leveraging the known semantic information of annotated closed-set pictographic characters to infer and decipher the semantics of unknown open-set pictographic characters, thus providing an interpretable deciphering reference for novel pictographic characters and providing a new approach to the classification and interpretation of unknown characters.

[0153] In some embodiments, for the interpretable object detection network that has been trained in step S11, the step of training the interpretable object detection network specifically includes steps S31 to S33.

[0154] S31. Obtain closed-set character samples and open-set character samples.

[0155] Among them, closed-set character samples are pictographic character images with known labeled categories; open-set character samples are pictographic character images with categories to be verified and to be deciphered.

[0156] Specifically, closed-set character samples can be obtained from the closed set stored in the known database, and open-set character samples can be obtained from the open set stored in the known database. However, both closed-set character samples and open-set character samples are extremely sparsely labeled samples.

[0157] S32: Process the closed-set character samples and the open-set character samples to obtain closed-set samples and open-set samples.

[0158] The processing process of this step may include pre-processing and / or augmentation processing before training.

[0159] In one possible implementation, closed-set character samples and open-set character samples are preprocessed to determine closed-set samples and open-set samples that meet a preset size. Here, the preprocessing process also involves scaling the closed-set character samples and open-set character samples so that the scaled closed-set samples and open-set samples ultimately meet the preset size.

[0160] The preset size may be set according to the actual network model and experience, and is not specifically limited in the embodiments of the present disclosure.

[0161] In another possible implementation, the closed-set character samples and the open-set character samples are preprocessed to determine initial closed-set samples and initial open-set samples that meet a preset size; the initial closed-set samples and the initial open-set samples are randomly perturbed to obtain augmented closed-set samples and open-set samples. This implementation differs from the previous implementation in that, after preprocessing, augmentation processing is further performed to randomly perturb the initial closed-set samples and the initial open-set samples. For example, using 2D-FFD, the coordinates of each control point of the FFD framework are randomly perturbed to obtain augmented closed-set samples and open-set samples.

[0162] S33. Input multiple closed-set samples and multiple open-set samples into an interpretable target detection network for training to obtain a trained target detection network.

[0163] Here, the actual training process is specifically referred to the following steps S331 to S336.

[0164] S331. Input multiple closed-set samples and multiple open-set samples into an interpretable target detection network to obtain a second similarity matrix group corresponding to each closed-set sample and each open-set sample output by the middle segment of the target detection network, as well as a sample prediction result finally output by the target detection network.

[0165] Among them, pictographic characters may belong to K categories, each category has P first prototypes pre-stored; K and P are both positive integers. The second similarity matrix It can be seen as the intersection of the between-category dimension and the within-category prototype dimension.

[0166] The second similarity matrix group includes pre-stored third similarity matrices corresponding to the first prototypes of the pictographic characters; the third similarity matrix represents the similarity information between an open set sample or a closed set sample and a first prototype.

[0167] Exemplarily, the network architecture of the object detection network includes an embedding coding module, a feature extraction module, a prototype module, and a prediction category module. This step specifically includes: processing multiple closed-set samples and multiple open-set samples sequentially through the embedding coding module, the feature extraction module, and the prototype module to obtain a second similarity matrix group corresponding to each closed-set sample and each open-set sample, output by the prototype module; and inputting each third similarity matrix in each second similarity matrix group into the prediction category module to obtain a sample prediction result corresponding to each closed-set sample and each open-set sample.

[0168] S332 . For each category, determine the entropy corresponding to the category according to the third similarity matrices of the P first prototypes. The entropies corresponding to the various categories constitute a global grouping entropy.

[0169] There are a total of P first prototypes between each category. For one category i, there are P third similarity matrices. The entropy of the P third similarity matrices is calculated and recorded as E gr_i The entropy corresponding to each category constitutes the global group entropy, recorded as

[0170] S333 : For each category, determine an average similarity value according to the third similarity matrices of the P first prototypes.

[0171] The third similarity matrices of the P first prototypes are known, that is, the element value of each third similarity matrix in the P third similarity matrices is known, and the average value of all element values ​​of the P third similarity matrices is calculated and recorded as the similarity average value of the current category.

[0172] S334. Determine the average global entropy based on the average similarities corresponding to the K categories.

[0173] Specifically, the entropy of the K similarity averages is calculated to obtain the average global entropy, which is recorded as E gl .

[0174] S335. Determine the entropy constraint of the target detection network based on the global group entropy and the average global entropy.

[0175] use and E gl Computing entropy constraints for object detection networks

[0176] S336. Determine the loss function of the target detection network based on the classification loss and entropy constraint indicated by the sample prediction results, and supervise the network update according to the loss function until the target detection network converges to obtain a trained target detection network.

[0177] Sample prediction results include the classification loss of the target detection network The classification loss and entropy constraints Add them together to get the loss function, which is Using loss function Oversee network updates.

[0178] The disclosed embodiments are based on an interpretable pictographic character understanding method using extremely sparse annotated samples. Taking into account the reality that pictographic character data is small in volume and highly diverse, the method fully utilizes the data characteristics of pictographic characters, such as extreme sparseness, lack of texture and color, and the expression of complex semantic information by a limited combination of simple structures. Combined with the local information combination rules when humans recognize pictographic characters, the method can obtain a target detection network with higher recognition accuracy by using only a small number of extremely sparse annotated samples to train the target detection network.

[0179] In some embodiments, the trained object detection network can not only recognize and analyze unknown open-set samples, but also recognize closed-set samples and provide interpretability support.

[0180] The classification and identification process of closed set samples includes step S41, and the process of providing interpretability support includes S42 to S43.

[0181] S41. Input the obtained multiple closed-set samples into a trained and interpretable object detection network to obtain character detection results corresponding to the closed-set samples.

[0182] The character detection results corresponding to the closed-set samples mainly include classification prediction results for the closed-set samples, such as the probability of the category to which the closed-set samples belong.

[0183] The closed set samples may belong to any of the K categories (assuming it is k), and there are P prototypes belonging to each category in the target detection network.

[0184] S42: Determine a fourth prototype corresponding to a closed set of samples based on the detection results of each character and a plurality of first prototypes of pre-stored pictographic characters.

[0185] Specifically, it is known that the plurality of first prototypes of the pre-stored pictographic characters include K×P. Based on the detection results of each character, when it is determined that the character detection result indicates that the closed set sample belongs to category k, the closed set samples belonging to the same target category k are determined. Based on the target category k to which the closed sample belongs, a fourth prototype corresponding to the target category k is screened out from the plurality of first prototypes. That is, the first prototype of the target category k to which the current closed set sample belongs.

[0186] One closed set sample corresponds to multiple fourth prototypes Among them, p represents the fourth prototype, k represents the category to which the closed set sample belongs, P represents the number of fourth prototypes, and i represents the i-th one.

[0187] S43. Provide explainability support for the character detection results of the corresponding closed set samples based on the fourth prototype.

[0188] Thus, using multiple fourth prototypes The relevant information can provide interpretable support for the character detection results of the corresponding closed set samples. The relevant information includes but is not limited to the image features and semantic features that the fourth prototype pays attention to.

[0189] In a possible implementation, the local features and global features of the closed set samples and the fourth similarity matrix between each fourth prototype can be used as the second activation matrix; for any second activation matrix, the second activation matrix is ​​upsampled to obtain an upsampled second activation matrix; the larger the value of the element in the upsampled second activation matrix, the more similar the characteristics of the closed set sample are to the fourth prototype; based on each upsampled second activation matrix, the fourth prototype similar to the closed set sample is screened out, and the fourth prototype similar to the closed sample is used to provide explainability support for the character detection results of the corresponding closed set sample.

[0190] Specifically, the similarity algorithm can be used to determine the closed set samples and each fourth prototype The fourth similarity matrix between them is used to obtain multiple fourth similarity matrices; the fourth similarity matrices are all used as second activation matrices to obtain multiple second activation matrices, that is, the second activation matrices between the closed set samples and each fourth prototype. Afterwards, the second activation matrix is ​​mapped and upsampled to the size of the original pictographic character (closed set sample). The higher the value in the upsampled second activation matrix, the higher the similarity between the area corresponding to the closed set sample and the corresponding fourth prototype, which means that the feature information contained in this fourth prototype plays a more important role in the classification process. In this way, the fourth prototype similar to the closed set sample can be screened out according to each upsampled second activation matrix as the fourth prototype similar to the closed sample. Here, P' represents the number of fourth prototypes similar to the closed set samples. The image features and semantic features of the fourth prototypes similar to the closed set samples are used to provide interpretable support for the character detection results of the corresponding closed set samples.

[0191] In a possible implementation, a fourth prototype similar to the closed set sample is screened out. Specifically, a fifth preset number of fourth prototypes can be selected in descending order according to the maximum element value in each upsampled second activation matrix. P' represents a fifth preset number, wherein the fifth preset number can be set according to actual conditions and experience, and is not specifically limited in the embodiment of the present disclosure.

[0192] Afterwards, based on the image features and semantic features corresponding to the fifth preset number of fourth prototypes, explainability support is provided for the character detection results of the corresponding closed set samples.

[0193] The embodiment of the present disclosure is based on a method for understanding pictographic characters with interpretability under extremely sparse annotated samples. The trained and interpretable target detection network can recognize and classify closed-set pictographic character samples, and use the first prototype stored in the network to provide interpretability support for the classification prediction results indicated by the second prototype output result. The method of the present disclosure has achieved excellent results in the recognition of closed-set pictographic characters. Compared with traditional methods, the present disclosure takes into account the internal differences and sparsity of pictographic characters and achieves higher recognition accuracy and interpretability. This provides more reliable technical support for the research and practical application of pictographic characters, and helps to promote further development and application in the field of pictographic characters; the method of providing interpretability support for pictographic characters disclosed in the present disclosure is not only applicable to oracle bone inscriptions, but also to a wide range of pictographic character fields. By emphasizing interpretability in the recognition process, the present disclosure provides new ideas and methods for the research and application of various pictographic characters, which helps to promote development and innovation in this field.

[0194] The use of an interpretable object detection network model makes the pictographic character recognition process more transparent and supervised. By introducing global and local first prototypes, the model can explain each decision, providing a clear basis for the recognition of each character. The interpretability of this model with a local attention mechanism not only conforms to the structural composition of pictographic characters and the visual laws of human understanding of pictographic characters, helping to improve recognition accuracy, but also provides a reliable explanatory reference for experts in fields such as archaeology and professional calligraphers, allowing them to better understand the recognition results, thereby improving the reliability and practicality of the technology.

[0195] The method disclosed in the present invention has achieved excellent results in the recognition of closed-set oracle bone inscriptions. Compared with traditional methods, the present invention achieves higher recognition accuracy and interpretability based on the consideration of the internal differences and sparsity of pictographic characters. This provides more reliable technical support for the research and practical application of pictographic characters, and helps to promote further development and application in the field of pictographic characters; the interpretable recognition method provided by the present invention is not only applicable to oracle bone inscriptions, but also to a wide range of pictographic character fields. By emphasizing interpretability in the recognition process, the present invention provides new ideas and methods for the research and application of various pictographic characters, which helps to promote development and innovation in this field.

[0196] FIG2 is a system architecture diagram of the character detection method provided by an embodiment of the present disclosure. As shown in FIG2 , the dotted line represents the open set sample detection data flow. The specific detection process includes the open set sample 21 being input into the target detection network 20. One path of data is processed by the target detection network 20 to output the local features and global features of the open set sample, which are used for KNN clustering 22 to obtain a clustering result 23. Another path of data is compared with the open set sample 21 and K×P first prototypes. A third preset number of second prototypes 24 are screened out from the K×P first prototypes. The third preset number of second prototypes 24 are again compared with the open set sample. From the third preset number of second prototypes, a fourth preset number of second prototypes 25 similar to the open set sample are screened out. Subsequently, the image features and semantic features 26 of the fourth preset number of second prototypes 25 are used to provide interpretability support for the clustering result 23.

[0197] Continuing with Figure 2, the solid line represents the closed-set sample detection data flow. The specific detection process includes the closed-set sample 27 being input into the target detection network 20. One path of data is classified by the target detection network 20 to obtain a character detection result, i.e., the classification result 28. For the other path of data, after determining the category k to which the closed-set sample belongs, P first prototypes corresponding to category k are directly screened out from the K×P first prototypes as the fourth prototype 29 corresponding to the closed-set sample; the P fourth prototypes 29 are then compared with the closed-set sample, and P' fourth prototypes 210 similar to the closed-set sample are screened out from the P fourth prototypes 29. Subsequently, the image features and semantic features 211 of the P' fourth prototypes 210 can be used to provide interpretability support for the classification result 28. At the same time, the image features and semantic features 211 of the P' fourth prototypes 210 can be transferred to provide a deciphering reference for the open-set sample.

[0198] The object detection network can be viewed as a classifier or an interpreter. As a classifier, it can identify and classify closed-set ideographic character samples; as an interpreter, it uses the first prototype stored in the network to provide interpretability support for classification prediction results. Similarly, as a classifier, it can also identify and cluster open-set ideographic character samples; as an interpreter, it uses the first prototype stored in the network to provide interpretability support for clustering results.

[0199] In order to more clearly understand the embodiments of the present disclosure, the character detection method provided by the present disclosure is described below with a complete example. FIG3 is a specific flow chart of the character detection method provided by the embodiment of the present disclosure, as shown in FIG3 , including steps S51 to S58.

[0200] S51. Preprocess the closed-set character samples and the open-set character samples, and use a nonlinear variation algorithm to randomly perturb the preprocessed initial closed-set samples and the initial open-set samples to obtain augmented closed-set samples and open-set samples.

[0201] The specific implementation process of this step can refer to the above steps S31 to S32.

[0202] S52. Input multiple closed-set samples and multiple open-set samples into an interpretable target detection network, and train them using a loss function with entropy constraints to obtain a trained target detection network that can be regarded as a classifier and interpreter.

[0203] The specific implementation process of this step can refer to the above step S33 (including S331 to S336).

[0204] S53, input the closed set samples into the target detection network trained in S52 and regard it as a classifier, perform classification prediction, and obtain the classification result.

[0205] The specific implementation process of this step can refer to the above step S41.

[0206] S54. Treat the target detection network trained in S52 as an interpreter, take out the first prototype stored in the prototype module, and compare it with the closed set samples to provide interpretability support for the classification results of S53.

[0207] The specific implementation process of this step can refer to the above steps S42 to S43.

[0208] S55. Input the open set samples into the target detection network trained in S52 and regard them as a classifier. Obtain the local features and global features of the open set samples output by the prototype module, and perform KNN clustering under the zero-sample setting to obtain the clustering results.

[0209] The specific implementation process of this step can refer to the above steps S11 to S12 (including S121 to S123, and S123-1 to S123-5).

[0210] S56. Treat the target detection network trained in S52 as an interpreter, take out the first prototype stored in the prototype module, and compare it with the open set samples to provide interpretability support for the clustering results of S55.

[0211] The specific implementation process of this step can refer to the above steps S13 to S14 and their specific possible various implementation methods.

[0212] S57. For each open set sample belonging to the same category, based on the first similarity matrix group output by the target detection network, find the third prototype of the open set sample belonging to the same category as the image feature of the pictographic character of the category.

[0213] The specific implementation process of this step can refer to the above steps S21 to S22.

[0214] S58, tracing back the original categories of several third prototypes in the closed set samples in S57, comparing the relevant image features of the open set samples and the closed set samples of the same prototype part, and inferring the interpretation of the open set samples.

[0215] The specific implementation process of this step can be found in the above "Tracing Multiple Third Prototypes" An embodiment of the "original corresponding category in the closed set".

[0216] Figure 4 is a specific flow chart of the character detection method provided by an embodiment of the present disclosure. As shown in Figure 4, 40 represents the trained target detection network, 41 represents closed set samples, 42 represents open set samples, 43 represents P first prototypes under K categories, 44 represents classification results, 45 represents clustering results, 46 represents P' fourth prototypes similar to the closed set samples, 47 represents a fourth preset number of second prototypes similar to the open set samples, 48 ​​represents the target closed set samples under the closed set original category corresponding to the third prototype, and 49 represents the third prototype of the open set samples belonging to the same category.

[0217] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0218] In addition, a character detection device corresponding to the character detection method is also provided in the embodiment of the present disclosure. Since the principle of solving the problem by the device in the embodiment of the present disclosure is similar to the above-mentioned character detection method in the embodiment of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0219] FIG5 is a schematic diagram of a character detection device provided by an embodiment of the present disclosure. As shown in FIG5 , the character detection device includes a processing module 51 , a clustering module 52 and a parsing module 53 .

[0220] The processing module 51 is configured to input the obtained multiple open-set samples into a trained and interpretable target detection network to obtain character detection results corresponding to the open-set samples; the open-set samples are pre-processed pictographic character images whose categories are to be verified and to be deciphered.

[0221] It should be noted that the processing module 51 in the embodiment of the present disclosure is configured to execute step S11 in the above-mentioned character detection method.

[0222] The clustering module 52 is configured to cluster the open set samples according to the local features and global features of the open set samples corresponding to the detection results of each character, and obtain clustering results of the open set samples; the clustering results are used to characterize the category of the open set samples of the category to be verified.

[0223] It should be noted that the clustering module 52 in the embodiment of the present disclosure is configured to execute step S12 in the above-mentioned character detection method.

[0224] The parsing module 53 is configured to determine the first character feature corresponding to each type of character in the open set samples based on the local features and global features of the open set samples, as well as the local features and global features of multiple characters pre-stored by the target detection network, and provide interpretability support for the clustering results of the corresponding open set samples based on the first character feature; wherein the interpretability support includes image features and semantic features corresponding to the first character feature.

[0225] It should be noted that the parsing module 53 in the embodiment of the present disclosure is configured to execute steps S13 and S14 in the above-mentioned character detection method.

[0226] The character detection device provided by the embodiment of the present disclosure utilizes a trained, interpretable target detection network to identify and classify open-set pictographic character samples, and utilizes the first prototype stored in the network to provide interpretable support for the clustering results. Compared to traditional methods, the present disclosure adopts an interpretable target detection network, making the pictographic character recognition process more transparent and supervised. By introducing global and local first prototypes, the model can explain each decision, providing a clear basis for the recognition of each character. The interpretability of this model with a local attention mechanism not only conforms to the structural composition rules of pictographic characters and the visual rules of human understanding of pictographic characters, helping to improve recognition accuracy, but also provides a reliable explanatory reference for experts in fields such as archaeologists and professional calligraphers, enabling them to better understand the recognition results, thereby improving the reliability and practicality of the technology. At the same time, the present disclosure can achieve interpretable recognition and deciphering of novel pictographic characters. This provides new possibilities for archaeologists and researchers to better understand and interpret the meaning and cultural value behind unknown pictographic characters, expanding the breadth and depth of research.

[0227] Figure 6 is a schematic diagram of the structure of a computer device provided in an embodiment of the present disclosure. As shown in Figure 6, the computer device provided in an embodiment of the present disclosure includes: one or more processors 601, memory 602, and one or more I / O interfaces 603. Memory 602 stores one or more programs, which, when executed by the one or more processors, enable the one or more processors to implement any of the character detection methods described in the above embodiments; one or more I / O interfaces 603 are connected between the processor and memory and are configured to enable information exchange between the processor and memory.

[0228] Among them, the processor 601 is a device with data processing capabilities, including but not limited to a central processing unit (CPU); the memory 602 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), and flash memory (FLASH); the I / O interface (read-write interface) 603 is connected between the processor 601 and the memory 602, and can realize information interaction between the processor 601 and the memory 602, including but not limited to a data bus (Bus), etc.

[0229] In some embodiments, the processor 601 , the memory 602 , and the I / O interface 603 are connected to each other via a bus 604 , and further connected to other components of the computing device.

[0230] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium is further provided, wherein the non-transitory computer-readable storage medium stores a computer program, wherein when the program is executed by a processor, the steps of the character detection method in any of the above embodiments are implemented.

[0231] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a machine-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication part, and / or installed from a removable medium. When the computer program is executed by a central processing unit (CPU), the above-mentioned functions defined in the system of the present disclosure are executed.

[0232] It should be noted that the computer non-transitory readable medium shown in the present disclosure can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or component, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any non-transitory computer-readable storage medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the non-transitory computer-readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wire, optical cable, RF, etc., or any suitable combination thereof.

[0233] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architectures, functions and operations of the devices, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the aforementioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical functions. It should also be noted that in some alternative implementations, the functions marked in the boxes can also occur in an order different from that marked in the accompanying drawings. For example, two connected boxes can actually represent execution in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0234] It is understood that the above embodiments are merely exemplary embodiments for illustrating the principles of the present disclosure, and the present disclosure is not limited thereto. Those skilled in the art may make various modifications and improvements without departing from the spirit and substance of the present disclosure, and such modifications and improvements are also considered to be within the scope of protection of the present disclosure.

Claims

1. A character detection method, wherein: include: Input the obtained multiple open set samples into a trained and interpretable object detection network to obtain character detection results corresponding to the open set samples; The open set samples are pre-processed pictographic character images whose categories are to be verified and decoded; Clustering the open set samples according to the local features and global features of the open set samples corresponding to the respective character detection results to obtain a clustering result of the open set samples; the clustering result is used to characterize the category of the open set samples of the category to be verified; Based on the local features and global features of the open set samples, as well as the local features and global features of multiple characters pre-stored in the target detection network, the first character features corresponding to each type of characters in the open set samples are determined, and based on the first character features, explainability support is provided for the clustering results of the corresponding open set samples; wherein the explainability support includes image features and semantic features corresponding to the first character features.

2. The character detection method according to claim 1, wherein: Clustering the open set samples according to the local features and global features of the open set samples corresponding to the respective character detection results to obtain a clustering result of the open set samples includes: Concatenating the local features and the global features of the open set samples to obtain initial features; Normalizing the initial features to obtain target features of the open set samples; The open set samples are clustered according to target features of the open set samples to obtain clustering results of the open set samples.

3. The character detection method according to claim 2, wherein: Clustering the open set samples according to the target features of the open set samples to obtain clustering results of the open set samples includes: For each of the multiple to-be-verified categories, randomly sampling a first preset number of open set samples in the to-be-verified category, and using the remaining open set samples as test samples for zero-sample clustering; Using the target features of the first preset number of open set samples as a priori cluster centers; Using a preset similarity algorithm, the distance between the test sample and each cluster center under each category to be verified is obtained; For each of the test samples, taking the categories of the second preset number of cluster centers that are closest to the test sample as candidate categories; The voting algorithm is used to calculate the number of votes for each candidate category, and the candidate category with the highest number of votes is used as the clustering result of the test sample.

4. The character detection method according to claim 3, wherein: After obtaining the clustering result of the test sample, the method further includes: According to the confidence level indicated by the clustering result of each test sample, the average correct rate of the clustering results of all the test samples is determined as the clustering accuracy rate under the zero-sample setting.

5. The character detection method according to claim 1, wherein: Determining the first character feature corresponding to each type of character in the open set of samples based on the local features and global features of the open set of samples and the local features and global features of multiple characters pre-stored by the target detection network includes: Determining a first similarity matrix group based on the local features and global features of the open set samples and the local features and global features of the plurality of characters pre-stored by the object detection network; the first similarity matrix group includes first similarity matrices corresponding to the plurality of characters pre-stored by the object detection network; the first similarity matrix represents similarity information between one of the open set samples and one of the characters; According to the first similarity matrix group, the first similarity matrices that meet the first preset condition are screened out, and the characters corresponding to the screened out first similarity matrices are used as the first character features corresponding to each type of characters in the open set samples.

6. The character detection method according to claim 5, wherein: The step of screening out the first similarity matrices that meet a first preset condition based on the first similarity matrix group, and using characters corresponding to the screened out first similarity matrices as first character features corresponding to each type of characters in the open set of samples, includes: For the first similarity matrix corresponding to each of the characters, determining an average similarity of the first similarity matrix according to the values of each element in the first similarity matrix; For the average similarities of the multiple first similarity matrices, a third preset number of average similarities are selected in order from large to small, and the characters corresponding to the selected average similarities are used as the first character features corresponding to each type of characters in the open set samples.

7. The character detection method according to claim 5 or 6, wherein: Providing interpretable support for the clustering results of the open set samples corresponding to the first character feature based on the first character feature includes: Using the second similarity matrix between the local features and the global features of the open set samples and each of the first character features as a first activation matrix; For any of the first activation matrices, upsampling the first activation matrix to obtain an upsampled first activation matrix; a larger value of an element in the upsampled first activation matrix indicates that the feature of the open set sample is more similar to the first character feature; According to each of the upsampled first activation matrices, first character features similar to the open set samples are screened out, and the first character features similar to the open set samples are used to provide interpretable support for the clustering results of the corresponding open set samples.

8. The character detection method according to claim 7, wherein: The method of screening out first character features similar to the open set samples based on each of the upsampled first activation matrices, and using the first character features similar to the open set samples to provide interpretable support for clustering results of the corresponding open set samples includes: Selecting a fourth preset number of first character features in descending order according to the maximum element value in each of the upsampled first activation matrices; Based on the image features and semantic features corresponding to the fourth preset number of first character features, explainability support is provided for the clustering results of the corresponding open set samples.

9. The character detection method according to claim 5, wherein: After providing interpretability support for the clustering result of the corresponding open set samples based on the first character feature, the method further includes: For open set samples belonging to the same category, sum the element values of the P first similarity matrices in the first similarity matrix group to obtain P summation results; The corresponding first similarity matrix group includes P first similarity matrices; P is a positive integer; For open set samples belonging to the same category, based on the summation results, the first similarity matrix that meets the second preset condition is screened, and the characters corresponding to the screened first similarity matrix are used as the second character features corresponding to each type of characters in the open set samples belonging to the same category, and based on the second character features, explainability support is provided for the open set samples belonging to the same category; the explainability support includes image features and semantic features corresponding to the second character features.

10. The character detection method according to claim 9, wherein: After determining the second character feature corresponding to each character in the open set samples belonging to the same category, the method further includes: According to the category to which the closed set sample belongs, determining that the second character feature corresponds to the original category in the closed set sample, and determining the target closed set sample according to the original category; Compare the open set samples of the category corresponding to the second character feature with the target closed set samples, and provide deciphering reference information for the open set samples of the category corresponding to the second character feature based on the comparison result; the deciphering reference information includes image features and semantic features of closed set samples similar to the open set samples.

11. The character detection method according to claim 1, wherein: It also includes steps for training an interpretable object detection network, including: Obtain closed-set character samples and open-set character samples; the closed-set character samples are pictographic character images with known labeled categories; the open-set character samples are pictographic character images with categories to be verified and to be deciphered; Processing the closed-set character samples and the open-set character samples to obtain closed-set samples and open-set samples; Inputting a plurality of the closed set samples and a plurality of the open set samples into an interpretable target detection network for training, thereby obtaining a trained target detection network.

12. The character detection method according to claim 11, wherein: The processing of the closed-set character samples and the open-set character samples to obtain closed-set samples and open-set samples includes: Preprocessing the closed-set character samples and the open-set character samples to determine initial closed-set samples and initial open-set samples that meet a preset size; The initial closed set samples and the initial open set samples are randomly perturbed to obtain the augmented Closed set samples and open set samples.

13. The character detection method according to claim 11, wherein: Inputting the plurality of closed set samples and the plurality of open set samples into an interpretable target detection network for training to obtain a trained target detection network includes: Inputting a plurality of the closed set samples and a plurality of the open set samples into an interpretable target detection network, obtaining a second similarity matrix group corresponding to each of the closed set samples and each of the open set samples, which is output by an intermediate segment of the target detection network, and a sample prediction result finally output by the target detection network; the closed set samples or the open set samples may belong to K categories, each category pre-stored with P characters; K and P are both positive integers; the second similarity matrix group includes third similarity matrices corresponding to each of the plurality of characters, which are pre-stored; the third similarity matrix represents similarity information between one of the open set samples or one of the closed set samples and one of the characters; For each category, the entropy corresponding to the category is determined based on the third similarity matrix of the P characters, and the entropies corresponding to the categories constitute the global grouping entropy; For each category, determining an average similarity value based on the third similarity matrix of the P characters; Determine the average global entropy based on the average similarity values corresponding to the K categories; Determining an entropy constraint of the object detection network based on the global group entropy and the average global entropy; According to the classification loss and entropy constraint indicated by the sample prediction result, a loss function of the target detection network is determined, and the network update is supervised according to the loss function until the target detection network converges, thereby obtaining the trained target detection network.

14. The character detection method according to claim 13, wherein: The network architecture of the target detection network includes an embedding coding module, a feature extraction module, a prototype module and a prediction category module; Inputting the plurality of closed set samples and the plurality of open set samples into an interpretable target detection network, obtaining a second similarity matrix group output by an intermediate segment of the target detection network and corresponding to each of the closed set samples and each of the open set samples, and a sample prediction result finally output by the target detection network, including: Processing the plurality of closed-set samples and the plurality of open-set samples in sequence through the embedded coding module, the feature extraction module, and the prototype module to obtain a second similarity matrix group output by the prototype module and corresponding to each of the closed-set samples and each of the open-set samples; Each third similarity matrix in each second similarity matrix group is input into the prediction category module to obtain the sample prediction results corresponding to each closed set sample and each open set sample respectively.

15. A character detection device comprising a processing module, a clustering module and a parsing module; The processing module is configured to input the obtained multiple open set samples into a trained and interpretable object detection network to obtain character detection results corresponding to the open set samples; The open set samples are pre-processed pictographic character images whose categories are to be verified and decoded; The clustering module is configured to cluster the open set samples according to the local features and global features of the open set samples corresponding to each of the character detection results to obtain a clustering result of the open set samples; the clustering result is used to characterize the category of the open set samples of the category to be verified; The parsing module is configured to determine the first character feature corresponding to each type of character in the open set samples based on the local features and global features of the open set samples, as well as the local features and global features of multiple characters pre-stored in the target detection network, and provide explainability support for the clustering results of the corresponding open set samples based on the first character feature; wherein the explainability support includes image features and semantic features corresponding to the first character feature.

16. A computer device, wherein: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the character detection method according to any one of claims 1 to 14 are performed.

17. A computer non-transitory readable storage medium, wherein: The computer non-transitory readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the character detection method according to any one of claims 1 to 14 are executed.

Images