A calligraphy and painting recognition method based on deep learning

Abstract

This application discloses a method and apparatus for calligraphy and painting recognition based on deep learning, belonging to the field of artificial intelligence and digital image processing technology. The method includes: acquiring images of calligraphy and painting works; preprocessing and enhancing restoration by introducing a generative adversarial network with an edge-preserving loss function to preserve the texture of the ink marks; extracting the character structure features of the signature, inscription, and seal areas; inputting them into a specially trained optical character recognition and semantic analysis hybrid model; during decoding, weighted fusion of visual feature vectors and contextual semantic probability vectors to output initial recognition text; combining a calligraphy dictionary database, performing semantic error correction and matching through a joint calculation formula of quantified character topological edit distance and semantic probability to obtain and output the target recognition text. This solves the problems of failure due to lack of topological repair capability for damaged pixels and inability to hit the correct characters, achieving high-precision recognition and deep knowledge interpretation of calligraphy and painting text.

Description

Technical Field

[0001] This application belongs to the field of artificial intelligence and digital image processing technology, specifically relating to a calligraphy and painting recognition method based on deep learning. Background Technology

[0002] The core technological challenge in the authentication and digital archiving of Chinese calligraphy and painting works lies in the accurate identification of inscriptions, signatures, and seals. The fundamental flaw in existing technologies lies in their underlying logic, which is based on the deformation assumption of "modern standardized fonts," making it unable to analyze the high-dimensional deformation variables present in calligraphy and painting works. Specifically, existing technologies face unavoidable constraints: due to their age, calligraphy and painting works often suffer from yellowing, damage, and mold on the paper, coupled with ink smudges, causing topological breaks or adhesions in the strokes of characters. Traditional optical character recognition technologies fail because they lack the ability to topologically repair damaged pixels. Variant characters and taboo characters in ancient running script, cursive script, and seal carving possess a high degree of freedom, and their structural forms do not conform to the standard framework of modern Chinese characters, causing technical solutions based on standard character libraries to fail to match the correct characters. Faced with the aforementioned physical damage and character heterogeneity, relying solely on a single modality of visual features inevitably leads to ambiguity in multiple candidate characters. Existing technologies lack a computational mechanism that jointly constrains the contextual grammar of ancient Chinese with visual defects. Furthermore, traditional identification systems only output discrete text symbols and lack knowledge graph mapping based on entity relationships, thus failing to provide auxiliary decision-making basis for identification personnel. Summary of the Invention

[0003] The purpose of this application is to provide a deep learning-based method for calligraphy and painting recognition, in order to solve the problems of failure and / or inability to hit the correct characters in the existing technology due to the lack of topological repair capability for damaged pixels.

[0004] The first aspect of this application provides a deep learning-based method for calligraphy and painting recognition, including: The image of the calligraphy and painting work to be identified is obtained, and the image of the calligraphy and painting work is preprocessed and enhanced to obtain the target calligraphy and painting work image; The target calligraphy and painting image is used to extract features by a pre-constructed feature extraction network to obtain the character structure features corresponding to the target image; The glyph structure features are input into the hybrid model of optical character recognition and semantic analysis to obtain the initial recognized text; The initial identified text is semantically corrected and matched with a pre-built calligraphy dictionary database to obtain the target identified text, which is then converted into a searchable text format and output.

[0005] Further, an image of the calligraphy or painting work to be identified is acquired, and the image is preprocessed and enhanced to obtain the target calligraphy or painting work image, including: Acquire images of calligraphy and painting works to be identified by staff through human-computer interaction or collect images of calligraphy and painting works to be identified from a specified data source; The images of the calligraphy and painting works are sequentially subjected to noise reduction, tilt correction and contrast stretching to obtain the pre-processed images of the calligraphy and painting works. A generative adversarial network-based image inpainting model is used to perform pixel-level enhancement and restoration on the preprocessed calligraphy and painting images to obtain the target calligraphy and painting image.

[0006] Furthermore, the method also includes: Using a deep learning-based object detection algorithm, the signature area, inscription area, and seal area are located and cropped from the target image; The seal area is segmented and binarized using ink color thresholds within a specific color space to separate red or white ink marks and obtain the target data area. Feature extraction is performed on the target data region to obtain the glyph structure features corresponding to the target image; The glyph structure features are input into the hybrid model of optical character recognition and semantic analysis to obtain the initial recognized text; The initial identified text is semantically corrected and matched with a pre-built calligraphy dictionary database to obtain the target identified text, which is then converted into a searchable text format and output.

[0007] Furthermore, the hybrid model of optical character recognition and semantic analysis further includes the following before use: Construct a specialized training dataset containing authentic works by famous calligraphers throughout history, damaged rubbings, cursive script, and blurred signatures; The basic optical character recognition sub-model is adjusted using the aforementioned specialized training dataset and combined with the contextual semantic analysis sub-model, so that the model can learn the grammar and textual logic of the context while learning the shape of individual characters. The optical character recognition and semantic analysis hybrid model adopts an encoder-decoder architecture. In each step of decoding and generating text, the current frame image feature vector extracted by the optical character recognition sub-model is weighted and fused with the context semantic probability vector calculated by the semantic analysis sub-model based on the generated preceding text, and finally outputs the initial recognized text.

[0008] Furthermore, the initial identified text is semantically corrected and matched with a pre-built calligraphy dictionary database to obtain the target identified text, including: The edit distance between each character in the initial identified text and each glyph topological feature in the pre-constructed calligraphy dictionary database is obtained, and a similarity score is obtained based on the edit distance; the calligraphy dictionary database stores glyph topological features and their corresponding standard text labels; Based on the similarity score and combined with the contextual probability distribution output by the semantic analysis sub-model, candidate replacements are performed on characters with confidence scores below a preset threshold to obtain the target identification text.

[0009] Furthermore, the similarity score is obtained based on the edit distance as follows: ; in, Indicates the characters in the initial identified text. Topological features of glyphs in a pre-built calligraphy dictionary database Similarity score between them Weighting coefficients representing the similarity of character structures. Weight coefficients representing the semantic probability of context. Indicates the characters in the initial identified text. Topological features of glyphs in a pre-built calligraphy dictionary database Edit distance between Indicates the characters in the initial identified text. dimensionality of feature vectors This represents the topological features of each character form in a pre-built calligraphy dictionary database. dimensionality of feature vectors Indicates taking and The maximum value between This indicates the glyph topological features in the current context. The semantic probability value of occurrence.

[0010] Furthermore, the method also includes: Based on the target identified text, a preset expert interpretation knowledge graph is invoked to generate an expert interpretation result corresponding to the target identified text.

[0011] Furthermore, based on the target identified text, a preset expert interpretation knowledge graph is invoked to generate an expert interpretation result corresponding to the target identified text, including: Based on the target text, key entities are extracted, including the author's name, chronological information, place name, studio name, and seal owner; Based on the key entity input, experts are dispatched to interpret the knowledge graph to perform related queries and output an interpretation report containing auxiliary criteria for judging the authenticity of calligraphy and painting, the author's life background, the historical origin of the inscriptions and literary interpretations, thus obtaining the expert interpretation results corresponding to the target identification text.

[0012] Furthermore, the method also includes: The image of the calligraphy or painting to be identified, the target identification text, and the corresponding expert interpretation results are associated and stored, and then displayed through a visualization device.

[0013] A second aspect of this application provides a deep learning-based calligraphy and painting recognition device, comprising: The data processing module is used to acquire images of calligraphy and painting works to be identified, and to preprocess and enhance the images of calligraphy and painting works to obtain the target calligraphy and painting work image; The feature extraction module is used to extract features from the target calligraphy and painting image through a pre-constructed feature extraction network to obtain the character structure features corresponding to the target image; The text recognition module is used to input the glyph structure features into the hybrid model of optical character recognition and semantic analysis to obtain the initial recognized text; The matching and error correction module is used to perform semantic error correction and matching between the initial identified text and the pre-built calligraphy dictionary database to obtain the target identified text, and to convert the target identified text into a searchable text format and output it.

[0014] The beneficial effects of this application are as follows: This application provides a deep learning-based method and apparatus for calligraphy and painting recognition. The method includes: acquiring images of calligraphy and painting works; preprocessing and enhancing restoration using a generative adversarial network with an edge-preserving loss function to retain the texture of the ink marks; extracting the character structure features of the signature, inscription, and seal areas; inputting these features into a specially trained hybrid model of optical character recognition and semantic analysis; during decoding, weighted fusion of visual feature vectors and contextual semantic probability vectors to output initial recognition text; combining a calligraphy dictionary database, performing semantic error correction and matching through a joint calculation formula of quantified character topological edit distance and semantic probability to obtain the target recognition text; finally, converting it into searchable text and calling an expert interpretation knowledge graph to output an auxiliary identification report. This solves the problems of failure due to lack of topological repair capabilities for damaged pixels and inability to hit the correct characters, achieving high-precision recognition and deep knowledge interpretation of calligraphy and painting texts. Attached Figure Description

[0015] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0016] Figure 1 A flowchart of a deep learning-based calligraphy and painting recognition method provided for this application; Figure 2 The diagram shows a structure of a deep learning-based calligraphy and painting recognition device provided in this application.

[0017] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0019] The embodiments of this application are described in detail below with reference to the accompanying drawings.

[0020] like Figure 1 As shown, this application provides a deep learning-based method for calligraphy and painting recognition, including: S101. Obtain the image of the calligraphy and painting work to be identified, and preprocess and enhance the image of the calligraphy and painting work to obtain the target calligraphy and painting work image; S102. Extract features from the target calligraphy and painting image using a pre-constructed feature extraction network to obtain the character structure features corresponding to the target image; S103. Input the character structure features into the optical character recognition and semantic analysis hybrid model to obtain the initial recognized text; S104. Perform semantic error correction and matching between the initial identified text and the pre-built calligraphy dictionary database to obtain the target identified text, and convert the target identified text into a searchable text format and output it.

[0021] In some possible embodiments, an image of the calligraphy or painting work to be identified is acquired, and the image is preprocessed and enhanced to obtain the target calligraphy or painting work image, including: Acquire images of calligraphy and painting works to be identified by staff through human-computer interaction or collect images of calligraphy and painting works to be identified from a specified data source; The images of the calligraphy and painting works are sequentially subjected to noise reduction, tilt correction and contrast stretching to obtain the pre-processed images of the calligraphy and painting works. A generative adversarial network-based image inpainting model is used to perform pixel-level enhancement and restoration on the preprocessed calligraphy and painting images to obtain the target calligraphy and painting image.

[0022] The principle behind this step is to address the pixel topology loss problem caused by physical carrier degradation. The system acquires images of the calligraphy and painting works to be identified using a high-precision scanning device. First, an adaptive median filtering algorithm is used to remove paper texture noise from the surface of Xuan paper or silk, and then Hough transform is used for tilt correction and contrast stretching.

[0023] For damaged, blurred, and ink-blemished areas, a generative adversarial network (GAN)-based image inpainting model is used for pixel-level enhancement and restoration. The core technical constraint is that conventional mean squared error loss functions can produce an over-smoothing effect during the restoration process, erasing the unique artistic texture of calligraphy and painting. Therefore, this embodiment introduces an edge-preserving loss function into the discriminator of the GAN. During training and inference, while calculating the pixel differences between the generated image and the real image, the network extracts the edge gradient maps of both using the Sobel operator. This forces the network to retain the dry brush and texture features of the ink when filling in damaged areas, ultimately outputting a target image with clear edges and continuous strokes.

[0024] Generative Adversarial Networks (GANs) typically consist of two competing neural networks: a generator (G) and a discriminator (D). The generator G takes an input image of a painting or calligraphy work with damage or holes (usually the damaged areas are marked with a binary mask). G's task is to decode and generate pixel values ​​to fill in the missing areas based on the effective pixel textures around the damaged regions. Essentially, it learns a mapping function from the damaged image space to the complete image space. The discriminator D receives two types of images: genuine, undamaged images of the painting or calligraphy, and images restored by the generator G. Its main function is to output a probability value to determine whether the input image is genuine or a forgery. During training, G attempts to generate images as realistic as possible to deceive D, while D tries to accurately identify G's forgery attempts. This zero-sum game forces the pixel distribution generated by G to infinitely approximate the pixel distribution of the genuine painting or calligraphy, thus achieving a highly realistic restoration.

[0025] In each forward propagation of the GAN, not only are the generated image and the real image input to the discriminator, but the corresponding generated edge map and real edge map are also obtained through the Sobel operator. The edge preservation loss function is defined as the mean square error between these two edge maps. This mean square error is added to the loss function of the generator G (such as by weighted summation with the original generator G's loss function or by direct addition). Through this mechanism, the Sobel operator is used as a hard-coded supervisor to ensure the physical authenticity of the calligraphy and painting restoration at the pixel topology level.

[0026] Exemplarily, assume that a partial image of a Chinese ink painting by Xu Wei in the Ming Dynasty is input. There is a wormhole with a diameter of about 15 pixels in the picture, and the hole just cuts across the first horizontal stroke of the character "tian" in the signature. If conventional restoration is adopted, this horizontal stroke will be repaired into a smooth solid-color block, losing the sense of pause and change in writing with a brush. By using the generative adversarial network introducing an edge-preserving loss function in the present invention, the network senses the edge gradient directions at both ends of the first horizontal stroke of the character "tian" through the Sobel operator. When generating pixels to fill the hole, it will generate a continuous pixel sequence with depth changes and a slightly rough edge in accordance with the gradients at both ends, so as to physically restore the target image that conforms to the physical characteristics of the writing brush.

[0027] In some possible embodiments, the method further includes: Using a deep learning-based object detection algorithm to locate and crop the signature area, the inscription area, and the seal area from the target image; Performing ink color threshold segmentation and binary processing on the seal area in a specific color space to separate the red-character imprint or the white-character imprint and obtain the target data area; Performing feature extraction on the target data area to obtain the glyph structure features corresponding to the target image; Inputting the glyph structure features into an optical character recognition and semantic analysis hybrid model to obtain an initial identification text; Performing semantic error correction and matching on the initial identification text with a pre-constructed calligraphy dictionary database to obtain a target identification text, and converting the target identification text into a retrievable text format and outputting it.

[0028] By separately identifying and analyzing the signature area, the inscription area, and the seal area, the recognition accuracy of the origin of the calligraphy and painting can be improved.

[0029] Exemplarily, the YOLO object detection algorithm can be used to locate and crop the signature area, the inscription area, and the seal area from the target image and crop these areas.

[0030] For the seal area, its physical property is that red ink of a specific wavelength is adsorbed on the paper surface, having a color space difference from the black ink and the yellowed paper. The seal area can be converted from the RGB color space to the HSV specific color space, and the hue threshold range of the red ink of the seal is set for color threshold segmentation and binary processing to completely strip the interference of the paper background color and accurately separate the red-character imprint or the white-character imprint. Subsequently, the above-cropped area is input into the ResNet feature extraction network to extract high-dimensional glyph structure features including the stroke topology structure.

[0031] Suppose the target image contains a white-text seal with the inscription "Wen Peng". Due to its age, the ink is extremely faint and has a slight color difference from the surrounding yellowish Xuan paper. We can convert it to HSV color space, locking the H (hue) channel to 0-10 degrees and the S (saturation) channel to a range greater than 0.4. We set all background pixels of the yellowish paper below this threshold to 0 (black), and set pixels matching the ink characteristics to 255 (white), thus obtaining a high-contrast binarized image of the white-text seal. This binarized image is then input into a ResNet network, which converts the seal's line direction and intersections into a 1024-dimensional glyph structure feature vector and outputs it.

[0032] In some possible embodiments, feature extraction is performed on the target calligraphy and painting image using a pre-constructed feature extraction network to obtain the character structure features corresponding to the target image, including: ResNet can be used as the feature extraction network, and after training, a pre-constructed feature extraction network can be obtained. Traditional convolutional neural networks are prone to network degradation after stacking multiple layers, meaning that the deeper the layers, the more blurred the extracted features become. ResNet (Residual Neural Network) introduces cross-layer skip connections, allowing the network to directly pass the original gradient information of shallow layers when extracting features, ensuring that deep networks can learn extremely subtle stroke differences, rather than losing them in multiple convolutions. Optionally, since the font of the signature is usually small (a few millimeters), while the font of the calligraphy and painting content and the title and colophon are larger, a single-scale convolutional kernel cannot adapt to both simultaneously. The idea of ​​Feature Pyramid Network (FPN) can be used to fuse the high-resolution, strong positional information features extracted by the shallow network with the low-resolution, strong semantic information features extracted by the deep network element-wise, ensuring that regardless of the font size, a feature map with uniform dimension and containing complete structural details can be output.

[0033] In some possible embodiments, the hybrid optical character recognition and semantic analysis model further includes, prior to use: Construct a specialized training dataset containing authentic works by famous calligraphers throughout history, damaged rubbings, cursive script, and blurred signatures; The basic optical character recognition sub-model is adjusted using the aforementioned specialized training dataset and combined with the contextual semantic analysis sub-model, so that the model can learn the grammar and textual logic of the context while learning the shape of individual characters. Among them, the optical character recognition and semantic analysis hybrid model adopts an encoder-decoder architecture. At each step of decoding and generating text, the feature vector of the current frame image extracted by the optical character recognition sub-model (such as TrOCR) is weighted and fused with the context semantic probability vector calculated by the semantic analysis sub-model (such as a large language model or other semantic analysis models) based on the previously generated text, and finally the initial identification text is output.

[0034] The embodiment of this application uses a dual-modal fusion mechanism under the encoder-decoder architecture to solve the problem of recognizing incomplete characters. Therefore, before using the model, a targeted training process can be carried out, specifically including: constructing a special training data set containing 200,000 groups of authentic works of famous artists of past dynasties, damaged rubbings, scribbled cursive scripts, and blurred signatures. The basic optical character recognition sub-model is adjusted using this special training data set to make it adapt to heterogeneous writing systems. The optical character recognition and semantic analysis hybrid model adopts an encoder-decoder architecture. At each step of the decoder generating text, it does not solely rely on visual features. Specifically, the optical character recognition sub-model extracts the feature vector of the current frame image; at the same time, the semantic analysis sub-model calculates the context semantic probability vector based on the previously generated text sequence. The feature vector of the current frame image and the context semantic probability vector are weighted and fused to generate the final predicted probability distribution. When encountering scribbled or damaged and blurred fonts, there is uncertainty in the visual feature vector, and the fusion weight of the semantic probability vector can be automatically increased according to a preset weight change strategy, and the correct character can be locked using ancient Chinese grammar and writing logic, so as to output the initial identification text sequence and the confidence score of each character.

[0035] Exemplarily, it is assumed that the postscript "清風徐來" is being recognized, but for the character "徐", due to water stains smudging, the radical "彳" is completely blurred, and only the right side "余" remains. A traditional model only looking at visual features is very likely to mis-recognize the incomplete "徐" as "馀" or "涂". When the decoder processes this incomplete position, the visual feature vector output by the optical character recognition sub-model contains the low-confidence features of "余" and the blurred strokes; at the same time, the semantic analysis sub-model extracts the previous sequence "清風" and calculates that the semantic probability of the subsequent connection of "徐來" (from "The Ode to the Red Cliff") in the context of ancient Chinese is extremely high. After weighted fusion, the semantic probability vector strongly corrects the deviation of the visual feature, forces the output of the character "徐", and gives a relatively low visual confidence score (such as 0.72) to this character in the initial identification text, marking it as a character to be corrected. It is worth noting that existing optical character recognition and semantic analysis hybrid analysis techniques can also be directly used for analysis.

[0036] In some possible embodiments, the initial identification text is semantically corrected and matched with a pre-constructed calligraphy dictionary database to obtain the target identification text, including: Obtain the edit distance between each character in the initial recognition text and each glyph topological feature in the pre-constructed calligraphy dictionary database, and obtain the similarity score according to the edit distance; the calligraphy dictionary database stores glyph topological features and their corresponding standard text labels. Based on the similarity score, combined with the context probability distribution output by the semantic analysis sub-model, replace the characters with a confidence level lower than the preset threshold with candidate items to obtain the target recognition text.

[0037] In some possible embodiments, obtaining the similarity score according to the edit distance is: ; Wherein, represents the character in the initial recognition text and the glyph topological feature in the pre-constructed calligraphy dictionary database the similarity score between them, represents the weight coefficient of the glyph structure similarity, represents the weight coefficient of the context semantic probability, represents the character in the initial recognition text and the glyph topological feature in the pre-constructed calligraphy dictionary database the edit distance between them, represents the character in the initial recognition text the feature vector dimension of, represents each glyph topological feature in the pre-constructed calligraphy dictionary database the feature vector dimension of, represents taking and the maximum value between, represents that in the current context, the semantic probability value of the glyph topological feature appearing.

[0038] For example, for the signature year, due to paper damage, the optical character recognition and semantic analysis hybrid model misrecognizes the character "Xin" with low confidence as the candidate character X = Xing, with a confidence of 0.60, triggering the error correction mechanism. Match the standard character Y = "Xin" from the calligraphy dictionary database. Assume that the topological edit distance can be obtained by calculating the difference in the stroke skeleton diagrams of "Xing" and "Xin": (that is, 102 basic pixel operations are required to transform the skeleton). Calculate the normalized score of the glyph structure part: ; The semantic analysis model combines the context "歲在○亥" for inference: Since there is only "Xin Hai" and no "Xing Hai" in the stem-branch chronology in ancient Chinese, the output semantic probability value is: . The final similarity score is calculated as: Since the final similarity score of 0.932 far exceeds the self-matching score of the original candidate character "Xing", the system executes the replacement, outputs the correct "Xinhai", and completes the precise error correction based on the quantitative data.

[0039] In some possible embodiments, the method further includes: According to the target identification text, call a preset expert interpretation knowledge graph to generate an expert interpretation result corresponding to the target identification text.

[0040] In some possible embodiments, according to the target identification text, calling a preset expert interpretation knowledge graph to generate an expert interpretation result corresponding to the target identification text includes: Extract key entities based on the target identification text, where the key entities include author name, chronology information, place name, studio name, and seal owner; Based on the input of the key entities, schedule an associated query in the expert interpretation knowledge graph, output an interpretation report including auxiliary judgment basis for the authenticity of calligraphy and painting, author's life background, historical origin of inscriptions, and literary interpretation, and obtain an expert interpretation result corresponding to the target identification text.

[0041] Map discrete text symbols to a structured knowledge system to complete the closed-loop from symbol identification to knowledge reasoning. Convert traditional Chinese characters or variant Chinese characters in the target identification text into a standard retrievable text format through a preset mapping table and output them, and store them in a relational database to support subsequent full-text retrieval. Further, according to the retrievable text, call a preset expert interpretation knowledge graph. Through named entity recognition technology, extract key entities based on the target identification text. Input the key entities into the expert interpretation knowledge graph for an associated query, and output an interpretation report including auxiliary judgment basis for the authenticity of calligraphy and painting, author's life background, historical origin of inscriptions, and literary interpretation.

[0042] Exemplarily, assume that the finally output retrievable text is "Jiajing Renyin Wen Zhengming Shu". Extract the key entities through named entity recognition: chronology information "Jiajing Renyin" (corresponding to 1542 AD), author name "Wen Zhengming", and action "Shu". Input "Wen Zhengming" and "Jiajing Renyin" into the expert interpretation knowledge graph. Through associated query, the knowledge graph finds that Wen Zhengming was born in 1470 and died in 1559. In 1542, Wen Zhengming was 72 years old, belonging to the late stage when his calligraphy style reached its peak. At the same time, compare the seal entity "Zhengming" (assumed to be a white seal) extracted from the current image, and it is retrieved in the knowledge graph that this white seal was only started to be used by Wen Zhengming after he was 70 years old. Since the inscription chronology and the seal start time highly coincide, output the interpretation report: "The cross-verification of the chronology and the seal usage period passes, conforming to the late writing characteristics of Wen Zhengming, and the probability of auxiliary judgment as a genuine work is relatively high". If there is a contradiction between the inscription chronology and the seal start time, output the opposite auxiliary judgment basis.

[0043] In some possible embodiments, the method further includes: The image of the calligraphy or painting to be identified, the target identification text, and the corresponding expert interpretation results are associated and stored, and then displayed through a visualization device.

[0044] The beneficial effects of this invention include: by introducing a generative adversarial network with an edge-preserving loss function, a technical path for pixel-level completion is established without destroying the textural features of brushstrokes and dry brushstrokes, thus solving the fundamental problem of distortion in the extraction of features from damaged images. A dual-modal fusion mechanism under an encoder-decoder architecture is constructed. At each time step of decoding, the visual feature vector and semantic probability vector are weighted and fused, utilizing the logic of classical Chinese writing to forcibly constrain the ambiguity caused by visual defects—a fundamental technological breakthrough distinct from traditional serial recognition. A distribution mechanism quantifying the topological structure of characters and contextual semantics is designed. Through a clear weight allocation mechanism, the problem of misidentification of highly similar variant characters caused by missing strokes is accurately solved, achieving measurable error correction.

[0045] like Figure 2 As shown, this application provides a deep learning-based calligraphy and painting recognition device, comprising: The data processing module 201 is used to acquire the image of the calligraphy and painting work to be identified, and to preprocess and enhance the image of the calligraphy and painting work to obtain the target calligraphy and painting work image; Feature extraction module 202 is used to extract features from the target calligraphy and painting image through a pre-constructed feature extraction network to obtain the character structure features corresponding to the target image; The text recognition module 203 is used to input the glyph structure features into the optical character recognition and semantic analysis hybrid model to obtain the initial recognized text; The matching and error correction module 204 is used to perform semantic error correction and matching between the initial identified text and the pre-built calligraphy dictionary database to obtain the target identified text, and to convert the target identified text into a searchable text format and output it.

[0046] The calligraphy and painting recognition device based on deep learning provided in this application embodiment can execute the above-mentioned method and technical solution. Its principle and beneficial effects are similar, and will not be repeated here.

[0047] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processing unit of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processing unit of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0048] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0049] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0050] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this application are indicated by the claims.

[0051] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. A deep learning-based method for calligraphy and painting recognition, characterized in that, include: Acquire images of the calligraphy and painting works to be identified, and preprocess and enhance the images of the calligraphy and painting works to obtain the target calligraphy and painting work images; The target calligraphy and painting image is used to extract features by a pre-constructed feature extraction network to obtain the character structure features corresponding to the target image; The glyph structure features are input into the hybrid model of optical character recognition and semantic analysis to obtain the initial recognized text; The initial identified text is semantically corrected and matched with a pre-built calligraphy dictionary database to obtain the target identified text, which is then converted into a searchable text format and output.

2. The deep learning-based calligraphy and painting recognition method according to claim 1, characterized in that, Acquire images of the calligraphy and painting works to be identified, and preprocess and enhance the images to obtain the target calligraphy and painting work image, including: Acquire images of calligraphy and painting works to be identified by staff through human-computer interaction or collect images of calligraphy and painting works to be identified from a specified data source; The images of the calligraphy and painting works are sequentially subjected to noise reduction, tilt correction and contrast stretching to obtain the pre-processed images of the calligraphy and painting works. A generative adversarial network-based image inpainting model is used to perform pixel-level enhancement and restoration on the preprocessed calligraphy and painting images to obtain the target calligraphy and painting image.

3. The deep learning-based calligraphy and painting recognition method according to claim 1, characterized in that, The method also includes: Using a deep learning-based object detection algorithm, the signature area, inscription area, and seal area are located and cropped from the target image; The seal area is segmented and binarized using ink color thresholds within a specific color space to separate red or white ink marks and obtain the target data area. Feature extraction is performed on the target data region to obtain the glyph structure features corresponding to the target image; The glyph structure features are input into the hybrid model of optical character recognition and semantic analysis to obtain the initial recognized text; The initial identified text is semantically corrected and matched with a pre-built calligraphy dictionary database to obtain the target identified text, which is then converted into a searchable text format and output.

4. The deep learning-based calligraphy and painting recognition method according to claim 1, characterized in that, Before use, the hybrid model of optical character recognition and semantic analysis also includes: Construct a specialized training dataset containing authentic works by famous calligraphers throughout history, damaged rubbings, cursive script, and blurred signatures; The basic optical character recognition sub-model is adjusted using the aforementioned specialized training dataset and combined with the contextual semantic analysis sub-model, so that the model can learn the grammar and textual logic of the context while learning the shape of individual characters. The optical character recognition and semantic analysis hybrid model adopts an encoder-decoder architecture. In each step of decoding and generating text, the current frame image feature vector extracted by the optical character recognition sub-model is weighted and fused with the context semantic probability vector calculated by the semantic analysis sub-model based on the generated preceding text, and finally outputs the initial recognized text.

5. The deep learning-based calligraphy and painting recognition method according to claim 1, characterized in that, The initial identified text is semantically corrected and matched with a pre-built calligraphy dictionary database to obtain the target identified text, including: The edit distance between each character in the initial identified text and each glyph topological feature in the pre-constructed calligraphy dictionary database is obtained, and a similarity score is obtained based on the edit distance; the calligraphy dictionary database stores glyph topological features and their corresponding standard text labels; Based on the similarity score and combined with the contextual probability distribution output by the semantic analysis sub-model, candidate replacements are performed on characters with confidence scores below a preset threshold to obtain the target identification text.

6. The deep learning-based calligraphy and painting recognition method according to claim 5, characterized in that, A similarity score is obtained based on the edit distance, including: ； in, Indicates the characters in the initial identified text. Topological features of glyphs in a pre-built calligraphy dictionary database Similarity score between them Weighting coefficients representing the similarity of character structures. Weight coefficients representing the semantic probability of context. Indicates the characters in the initial identified text. Topological features of glyphs in a pre-built calligraphy dictionary database Edit distance between Indicates the characters in the initial identified text. dimensionality of feature vectors This represents the topological features of each character form in a pre-built calligraphy dictionary database. dimensionality of feature vectors Indicates taking and The maximum value between This indicates the glyph topological features in the current context. The semantic probability value of occurrence.

7. The deep learning-based calligraphy and painting recognition method according to claim 6, characterized in that, The method also includes: Based on the target identified text, a preset expert interpretation knowledge graph is invoked to generate an expert interpretation result corresponding to the target identified text.

8. The deep learning-based calligraphy and painting recognition method according to claim 7, characterized in that, Based on the target identified text, a preset expert interpretation knowledge graph is invoked to generate an expert interpretation result corresponding to the target identified text, including: Based on the target text, key entities are extracted, including the author's name, chronological information, place name, studio name, and seal owner; Based on the key entity input, experts are dispatched to interpret the knowledge graph to perform related queries and output an interpretation report containing auxiliary criteria for judging the authenticity of calligraphy and painting, the author's life background, the historical origin of the inscriptions and literary interpretations, thus obtaining the expert interpretation results corresponding to the target identification text.

9. The deep learning-based calligraphy and painting recognition method according to claim 8, characterized in that, The method also includes: The image of the calligraphy or painting to be identified, the target identification text, and the corresponding expert interpretation results are associated and stored, and then displayed through a visualization device.

10. A calligraphy and painting recognition device based on deep learning, characterized in that, include: The data processing module is used to acquire images of calligraphy and painting works to be identified, and to preprocess and enhance the images of calligraphy and painting works to obtain the target calligraphy and painting work image; The feature extraction module is used to extract features from the target calligraphy and painting image through a pre-constructed feature extraction network to obtain the character structure features corresponding to the target image; The text recognition module is used to input the glyph structure features into the hybrid model of optical character recognition and semantic analysis to obtain the initial recognized text; The matching and error correction module is used to perform semantic error correction and matching between the initial identified text and the pre-built calligraphy dictionary database to obtain the target identified text, and to convert the target identified text into a searchable text format and output it.

Images