Handwritten poetry recognition method based on deep learning and scenic spot knowledge graph

Abstract

The handwritten poetry recognition method integrating deep learning with scenic spot knowledge graph belongs to the field of electronic information, and solves the problem that the poetry text extracted from the handwritten poetry image of the scenic spot by the text recognition method is difficult to cover the main text of the poetry. The present invention interacts with the user's smart phone, and uses its integrated Beidou / GPS sensor to sense the user's location and orientation information. The user collects the image data of handwritten poetry in the scenic spot through the camera of the smart phone; the poetry text position information in the scenic spot handwritten poetry image data is detected by the scenic spot handwritten poetry detection technology based on FPN, and the specific regional location information of the poetry text is extracted; the poetry text information in the area is identified by the scenic spot handwritten poetry recognition method based on ACE, and the preliminary poetry text recognition result is obtained; the preliminary identified poetry text result is corrected and improved by the scenic spot knowledge graph, and the final poetry text recognition result is returned to the user.

Description

Technical Field

[0001] The present invention belongs to the field of electronic information and is a method for recognizing handwritten poems in scenic spots based on deep learning and knowledge graph. Background Art

[0002] With the rapid development of my country's economy, scenic spot visits have become an indispensable part of people's daily life. At the same time, in order to increase their popularity and attract tourists, different scenic spots in the scenic area usually quote poems by famous poets of ancient and modern times to enhance the cultural atmosphere of the scenic spot. These quoted poems usually imitate the handwriting of the quoted poets, resulting in different styles of handwritten poems in scenic spots, and also making it difficult for today's tourists to fully recognize all the words in the quoted poems. Therefore, the recognition method of handwritten poems in scenic spots has obviously become a key issue. At present, there are relatively mature technical methods in this field at home and abroad, which can be mainly divided into two types: the first is to use scenic spot poetry image text detection and recognition model training methods separately, in which the text detection method mainly draws on the algorithm in the field of target detection to perform regression detection on the text box of the poem text; while the text recognition method mainly draws on the algorithm in the field of speech recognition to feature encode the image area of ​​the poem text, and then use the model to decode the content information of the poem text. The second is an end-to-end recognition method that combines text detection with text recognition models. This method can optimize the text detection model through text recognition results, but the computational complexity is greater than that of the first method. Although the above two methods have achieved good recognition results, the recognition effect is poor due to the complex fonts, poor imaging effect and complex background of the text content of the handwritten poetry images in the scenic area. Given that more and more scenic spots are building knowledge graphs for the scenic spots to facilitate tourists to retrieve scenic spot information in the scenic spots, a new solution has been brought to the problem of poor text recognition effect of handwritten poetry images in scenic spots. Summary of the invention

[0003] In order to solve the problem that the poem text extracted from the handwritten poem image of the scenic spot using the traditional text recognition method is difficult to cover the main text of the poem, the present invention proposes a scenic spot handwritten poem recognition method that integrates deep learning and scenic spot knowledge graph. The present invention mainly interacts with the user's smart phone, and uses its integrated Beidou / GPS sensor to perceive the user's location and orientation information. The user collects the image data of handwritten poems in the scenic spot through the camera of the smart phone; the poem text position information in the scenic spot handwritten poem image data is detected by the FPN-based scenic spot handwritten poem detection technology, and the specific regional location information of the poem text is extracted; the poem text information in the area is recognized by the ACE-based scenic spot handwritten poem recognition method, and the preliminary poem text recognition result is obtained; the preliminary recognized poem text result is corrected and improved by the scenic spot knowledge graph, and the final poem text recognition result is returned to the user for display.

[0004] The present invention comprises the following steps:

[0005] Step 1: Obtain the handwritten poetry text image of the scenic spot to be identified and the relevant attribute information of the image, including the geographical location of the image shooting, the image background texture, and the text direction, and use the spatial domain enhancement algorithm to reduce the noise in the image so that the structural similarity index between the original image and the denoised image is greater than 0.9 to obtain the preprocessing result.

[0006] Step 2: Input the preprocessed scenic spot handwritten poetry image into the feature extraction network based on VGG16 to extract the features of the poetry text, and obtain the handwritten poetry text weight sequence of the scenic spot image through the poetry text classifier, and train the classifier at the same time.

[0007] Step 3: Input the extracted handwritten poetry text feature map into the FPN-based handwritten poetry detection network for fusion to obtain the single-character Gaussian heat map of the handwritten poetry text, and then obtain the text area position information of the scenic spot handwritten poetry image through the multi-character text box linking algorithm.

[0008] Step 4: Crop the scenic area handwritten poetry image according to the extracted text image position information, and input the cropped area into the ACE-based handwritten poetry recognition network in turn to perform Encoder-Decoder processing of the handwritten poetry text to obtain the recognition result of the scenic area handwritten poetry image.

[0009] Step 5. Input the relevant attribute information of the scenic spot handwritten poetry image into the scenic spot knowledge graph and obtain the search knowledge result set through graph search. Use the recognition result obtained in step 4 and the search knowledge result set through the scenic spot handwritten poetry matching algorithm to obtain the final text recognition result of the scenic spot handwritten poetry image.

[0010] Effects of the Invention

[0011] The handwritten poetry recognition method of the present invention, which integrates the vision of the smart phone design with the scenic spot knowledge graph, can facilitate the tourists in the scenic spot to recognize the text of the handwritten poetry in the scenic spot, enhance their understanding of the scenic spot culture, and at the same time increase the flow of tourists in the scenic spot and promote the relevant culture of the scenic spot. By introducing deep learning to perform text recognition on the handwritten poetry images in the scenic spot, and extracting relevant poetry information from the scenic spot knowledge graph based on the tourist location and orientation information collected by the Beidou / GPS sensor, as well as the poetry-related attribute information, the text recognition results are corrected to assist tourists in accurately identifying the text of the handwritten poetry images.

[0012] Difficulty of invention

[0013] (1) A detection technology for handwritten poems in scenic spots based on FPN was designed. The difficulty lies in the accurate prediction of large and small text areas in the images of handwritten poems in scenic spots, while also ensuring low latency requirements.

[0014] (2) A method for recognizing handwritten poems in scenic spots based on ACE was designed to perform text recognition on the poem images in the detected text area. Considering that this method will be applied to smartphone interaction, the difficulty lies in ensuring the recognition accuracy and real-time requirements of the poem text.

[0015] (3) A poetry correction technology based on knowledge graph was designed to correct and improve the initial recognition results of poetry texts. The difficulty lies in how to fuse and match the recognized texts with the knowledge entities in the scenic area knowledge graph. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] Figure 1 This is the framework diagram for recognizing handwritten poems in scenic spots.

[0017] Figure 2 It is a multi-feature module network structure.

[0018] Figure 3 It is the overall system architecture diagram.

[0019] Figure 4 This is a diagram of the steps for acquiring and processing scenic spot knowledge.

[0020] Figure 5 This is a process diagram of scenic spot knowledge cloud recognition service.

[0021] Figure 6 It is a flow chart for matching scenic spot poetry results.

[0022] Figure 7 This is the architecture diagram of the fusion of knowledge graph and scenic spot poetry recognition algorithm.

[0023] The core algorithm of the present invention

[0024] (1) Detection technology of handwritten poems in scenic spots based on FPN

[0025] Architecture Figure 1 As shown in the scenic area handwritten poetry image text area detection module, the entire structure is divided into three parts, namely: handwritten poetry spatial feature extraction network, character key point calibration algorithm and multi-character text box linking algorithm. First, the scenic area handwritten poetry image is input into the handwritten poetry spatial feature extraction network to extract the handwritten poetry text features; then the extracted poetry text features are marked with single character key points through the character key point calibration algorithm; finally, the marked single character key points are processed by the multi-character text box linking algorithm to obtain the scenic area handwritten poetry text area coordinate information, and this information is passed as input to the scenic area handwritten poetry image text character recognition module. Its detailed description is as follows:

[0026] The present invention uses VGG16 to extract features from images of handwritten poems in scenic spots, and obtains feature maps of poem texts of images of handwritten poems in scenic spots. Since the features of convolutional networks are hierarchical, features at different levels can complement each other's information, so the multi-layer feature fusion of a single model can directly and effectively incorporate the network into it, while also improving the accuracy of the network. The backbone network for the fusion of multiple feature modules in the present invention uses a feature pyramid network. In order to solve the problem of multi-scale target detection, the feature pyramid network uses a convolutional neural network to extract features of different scales from the bottom up from the same picture, and its layers are interconnected using existing networks. The present invention uses a 7-layer pyramid network, and the layers are interconnected to promote feature reuse. In the pyramid network of the present invention, a bottom-up path and a top-down path are horizontally paired for data transmission. Its network structure is as follows: Figure 2 shown.

[0027] 1) Bottom-up pathway: refers to the upward flow of data. Figure 2 As shown on the left, each layer includes a convolutional layer, a pooling layer, an activation function layer, and a recurrent layer. This path can obtain 7 multi-scale feature maps, labeled {c1, c2, c3, c4, c5, c6, c7}. Different feature maps record text features at different levels. Low-level features reflect the boundaries of shallow-level poetry and text features, while high-level features reflect deeper-level poetry and text features.

[0028] 2) Top-down pathway: refers to the downward flow of data, such as Figure 2As shown on the right. By upsampling the feature map, a high-resolution feature map with strong semantic information is provided, which is crucial for the detection of handwritten poetry texts in scenic spots. The outputs of the previous layers are used as the input of the current layer to extract richer features. At the same time, a deformable convolution module Def-Incept is added to extract the features of some deformed texts, and then a multi-layer feature map {p1, p2, p3, p4, p5, p6, p7} is generated. Finally, a convolution operation is performed on the feature map, which reduces the number of parameters and eliminates the confounding effect caused by deformation. The following formula (1) shows the feature extraction process.

[0029]

[0030] In the formula, Conv represents the convolution operation, Represents feature fusion, UpSample represents upsampling operation, and Def-Incept represents deformable convolution operation. The feature pyramid network module constructed by the present invention can fuse shallow features with deep features, and then add deep semantic information to the shallow feature map, and fuse the features of the two to generate more small target features, which is very helpful for improving the model's detection ability for small text images. The output information of this operation is processed by a Gaussian kernel function to obtain a Gaussian heat map of the key nodes of the characters of the image text.

[0031] The present invention designs a multi-character text box linking algorithm. The algorithm takes the Gaussian heat map of the character key nodes of the image text as a prerequisite, and obtains the final scenic spot handwritten poetry image text detection frame by calculating the link relationship between the character key nodes. The calculation process of the multi-character text box linking algorithm is introduced in detail below.

[0032] First, by analyzing and calculating the Gaussian heat map of the key points of each character in the image text of the handwritten poetry in the scenic area, the maximum diameter of the Gaussian heat map of each character is obtained, and a square box with the maximum diameter as the side length is drawn to mark the text box position of a single character. This can be used to avoid the rotation angle of the Gaussian heat map shape, which makes it difficult for the text box to fully contain the character area. Secondly, half of the diagonal length of the single-character text box is selected as the initial value of the radius r of the outward radiating circle, and the maximum value of the radius r is set according to the maximum side length of the input image. Then, with a step size of The radius of the radiation circle is continuously increased, and the exploration is carried out in the direction of the text input by the user. If another character text box is encountered, it stops and links the current character text box with the center of the encountered text box; if the radius r reaches the maximum value, it means that the text box link is over. Finally, the linked text boxes are integrated to obtain the final result of the location of the text detection area of ​​the scenic spot handwritten poetry image.

[0033] (2) ACE-based recognition method for handwritten poems in scenic spots

[0034] Architecture Figure 1 The Chinese character recognition network module is shown in the figure. By using the text recognition network of the scenic spot handwritten poems, the local text area obtained by the scenic spot handwritten poems text detection is first subjected to image normalization processing to make the data more standardized; then the processed image data is input into the handwritten poems character feature extraction network to serialize and encode the poem text features; finally, the serialized code of the poem text features is decoded by the character recognizer to obtain the preliminary scenic spot handwritten poems text recognition results. Its detailed description is as follows:

[0035] After normalizing the image of handwritten poetry text in the scenic area, character feature extraction will be performed on the processed text image. The present invention uses a convolutional recurrent neural network (CRNN) as the backbone network to extract features from the image text of handwritten poetry in the scenic area. The network structure of the character feature extraction is mainly composed of two parts: a convolutional layer and a recurrent network layer. First, the normalized poetry text image is input into the convolutional layer to extract the convolution feature map of the image; then, the extracted convolution feature map is input into the recurrent network layer to continue to extract the text sequence feature based on the convolution feature. The feature contains the contextual information of the poetry text, making the character recognition result more accurate; finally, the extracted feature is output for further research and analysis.

[0036] The present invention will use the Aggregation Cross-Entropy (ACE) algorithm to decode the feature sequence of the scenic spot handwritten poetry image text to realize the recognition of the scenic spot handwritten poetry image text. The goal of the algorithm is to multiply the gain value by the high-frequency part of the image that expresses specific content, and then reorganize it to obtain a better image. Therefore, calculating the gain coefficient of the high-frequency part is the core of the ACE algorithm. In the initial stage of model training, due to the uniform distribution of different character categories at different times, the present invention sets the gain coefficient to 1; in the subsequent training stage, due to the different times, the probability of a certain category will be much greater than the probability of other categories, the present invention sets the gain coefficient to the number of characters in the text in the image. The ACE algorithm can adapt well to the recognition of long texts, and can solve the alignment problem caused by indefinite length sequences, which provides great help for the present invention to decode the feature sequence of scenic spot handwritten poetry text.

[0037] (3) Poetry correction technology based on knowledge graph

[0038] Architecture Figure 1The present invention provides knowledge reasoning and related content knowledge recommendation functions, specifically: taking the description information of poetry queried by the user as the search condition, using the scenic spot knowledge graph to perform graph search, and obtaining the search result set C.

[0039] Secondly, the search result set C and the preliminary recognition result x of the scenic spot handwritten poem text image are segmented by the word segmentation algorithm f(·) and the obtained search result set keyword matrix S n×m And the preliminary recognition result matrix E of the scenic spot handwritten poetry text image 1×m , where n represents the number of entities in the search result set, m represents the number of processed keywords, k represents the index position of the entity in the search result set, f(·) represents the word segmentation function for text segmentation, C[k] represents a single text in the search result set, and x represents the preliminary recognition result of the scenic spot handwritten poem text image. Its calculation formulas are shown in (2) and (3).

[0040] S k×m =f(C[k]) (2)

[0041] E 1×m =f(x) (3)

[0042] Then, the entity keywords in the matrix obtained above are calculated using the distribution model for generating word vectors. Each entity keyword vector v e The calculation formula is shown in (4), where e represents a single sample in the matrix, g(·) represents the distribution model function that generates the word vector, i represents the index position of the entity keyword, and S n×m [e] represents a single sample data in the keyword matrix of the search result set, S n×m [e][i] means gradually taking the entity keyword data of a single sample in the keyword matrix of the search result set.

[0043]

[0044] The obtained vector v of the poem text e After normalization, the final text vector v of the poem text is obtained, and the n text vectors are combined to generate the search result vector set V. The calculation formula is shown in (5).

[0045]

[0046] After the preliminary recognition result of the scenic spot handwritten poetry text image is also processed as above, the final preliminary recognition result vector q of the scenic spot handwritten poetry text image is obtained.

[0047] Finally, the similarity between the two poetry text vectors is calculated by using the preliminary recognition result vector q of the scenic spot handwritten poetry text image and the text vector v of each entity in the search result vector set V through VSM. The poetry text vector with the highest similarity between the preliminary recognition result vector q of the scenic spot handwritten poetry text image and the search result vector set V is taken, and the text Result in the corresponding search result set C is output as the scenic spot handwritten poetry text recognition result. The calculation formula is shown in (6) and (7), where VSM(·) represents the vector space model calculation function, j represents the index position of a single search result vector, s1 and s2 are two texts, and their word frequencies are represented by a and b respectively. t and b t It indicates the index position of the frequency of words in the text.

[0048]

[0049]

[0050] The knowledge nodes with "association" relationship with the retrieval result Result knowledge node are obtained by using the knowledge reasoning method. The triple representation can be described as (?, association, Result). The relevant poetry knowledge node information is obtained through knowledge reasoning and calculation to realize the knowledge recommendation service. Compared with the existing technology, this system has the function of making the fragmented knowledge related to scenic spot poetry more relevant. By extracting and representing the original data, the relationship between the poetry ontology and the poetry feature entity is established, so that users can retrieve the complete information of poetry more accurately according to the content characteristics of the poetry part. The scenic spot poetry knowledge map provides the scenic spot cultural knowledge and poetry related resource recommendation service, which helps tourists to accept the scenic spot cultural knowledge more effectively and intuitively. DETAILED DESCRIPTION

[0051] In order to enable those skilled in the art to better understand the technical solutions in the content of the present invention, the technical solutions in the content of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the present invention. Obviously, the described implementation examples are only part of the implementation examples of the implementation examples of the present invention, rather than all the implementation examples. Based on the implementation examples in the implementation examples of the present invention, all other implementation examples obtained by ordinary technicians in the field should belong to the scope of protection of the embodiments of the present invention.

[0052] As pointed out in the above background technology section of this application, how to improve the accuracy of scenic spot poetry retrieval and recognition has become a key issue. At present, there are two main methods for retrieving and identifying poetry: (1) traditional scenic spot poetry database retrieval system (2) scenic spot poetry recognition method based on computer vision. Among them, for the first method, a very rich scenic spot poetry data set is required, and the database retrieval performance requirements are high. In addition, the system is based on keywords or shallow semantic analysis, resulting in poor results and huge time and labor costs. For the second method, a currently popular artificial intelligence method is used to collect scenic spot poetry images for large-scale model training and learning, but this method only uses image single-modal data, and the recognition accuracy is low, which cannot meet most scenic spot poetry retrieval and recognition scenarios.

[0053] In view of this, the present invention proposes a handwritten poetry recognition method that integrates deep learning and scenic spot knowledge graph, which can solve the problems mentioned in the above-mentioned related technologies.

[0054] The specific implementation of the present invention will be further described below in conjunction with the accompanying drawings of the present invention.

[0055] Figure 3 The system architecture diagram of an example of the present invention is shown. The system mainly includes a knowledge acquisition and processing module, a knowledge storage module, and a knowledge application module. The basic layer includes the knowledge acquisition and processing module, the database layer and the cache layer include the knowledge storage module, and the service end and the API end include the knowledge application module.

[0056] The knowledge acquisition and processing module processes the original knowledge of scenic spot poetry knowledge related books, websites, poetry Excel spreadsheets and XML files through three processes: data cleaning, knowledge processing and knowledge representation, and obtains the scenic spot poetry feature entity and scenic spot poetry ontology relationship network. Figure 4 shows the overall step diagram of the knowledge acquisition and processing module.

[0057] For example, taking the Guo Moruo Former Residence Scenic Area as an example, the poetry data about Guo Moruo Former Residence on the web page is collected. The structured knowledge after data cleaning is shown in Table 1 (the part that cannot be displayed is omitted). The knowledge processing of the scenic area poems includes Chinese word segmentation, part-of-speech tagging, named entity recognition and word vector relationship calculation. After filtering, the characteristic entities of the scenic area poems are obtained as shown in Table 2. Through the triple knowledge representation method, a series of relations are obtained, such as ("When it is time to give up the pen and volunteer, I will leave my wife, abandon my children and cut off the lotus root silk", belongs to "Returning to the Country Miscellaneous Poems"), (Guo Moruo, belongs to "Returning to the Country Miscellaneous Poems"), ("Ruining the family to help the country, returning to the country to fight the war, and composing poems to express my ambitions", belongs to "Returning to the Country Miscellaneous Poems"), and similarly there is also the relationship ("Returning to the Country Miscellaneous Poems", belongs to, Guo Moruo's Former Residence), as well as related content knowledge relations such as ("Tengu", associated, "Returning to the Country Miscellaneous Poems").

[0058] The knowledge storage module uses the Neo4j graph database to provide scenic spot poetry knowledge graph storage services, saving the relationship between the poetry ontology and feature entities and the related content knowledge ontology relationship. The poetry ontology attributes include ID, name, address, picture, longitude and latitude, content background and related content. The attribute data types are shown in Table 3.

[0059] This system adopts a microservice-oriented design pattern for platform design. Based on the SOA architecture, the core business lines of the system are divided into user authentication services, user authority control services, poetry feature entity extraction services, poetry knowledge reasoning and retrieval services, scenic spot poetry text image recognition services, and poetry knowledge recommendation services. The API interface is designed and implemented using the Restful specification. The MySQL object-relational database is used to store user information and system log records. Considering the scalability of the platform and support for high concurrency, Redis is used for distributed caching. The Docker container technology is used to encapsulate the knowledge service application of this system platform, which is convenient for distributed application deployment, making the knowledge service system highly portable and scalable. The Kubernetes platform is used to manage containers, so that the system platform can be automatically deployed, expanded, and managed, making this system highly available.

[0060] The knowledge application module includes commonly used multi-user services, such as: user login, user registration, historical search record query management; knowledge retrieval and reasoning, and the use of image text detection and recognition models to assist knowledge reasoning; based on the poetry ontology knowledge, recommend poetry content resource knowledge ontologies associated with it, such as: related books, related poems, and related scenic spots.

[0061] Table 1 Example of structured knowledge representation of scenic spots

[0062]

[0063] Table 2 Example of entity representation of scenic spot poems

[0064]

[0065] Table 3 Attribute data type table

[0066] Property Name type describe ID int Knowledge point ID Name string Knowledge point name Image string Poetry pictures Address string Location LatitudeLongitude string Latitude and longitude Background string background Related Content string Related content

[0067] Figure 5 The process of users performing scenic spot knowledge cloud recognition services is shown. Users log in to the system through identity authentication, input poetry feature information and upload scenic spot poetry text images, obtain relevant poetry ontology knowledge through knowledge reasoning module and poetry image recognition module, and then obtain complete poetry resource ontology knowledge based on ontology knowledge reasoning, integrate knowledge and push it to users to complete the knowledge service process. Figure 6 A flow chart of scenic spot poetry result matching is shown. Figure 7 The diagram shows the architecture of the fusion of deep learning and scenic spot knowledge graph.

Claims

1. A handwritten poetry recognition method integrating deep learning with scenic spot knowledge graph, characterized in that: The following steps are involved: Step 1: Obtain the handwritten poem text image of the scenic spot to be identified and the relevant attribute information of the image, including the geographical location of the image shooting, the image background texture, and the text direction, and use the spatial domain enhancement algorithm to reduce the noise in the image so that the structural similarity index between the original image and the denoised image is greater than 0.9, and obtain the preprocessing result; Step 2: Input the preprocessed scenic spot handwritten poetry image into the feature extraction network based on VGG16 to extract the feature of the poetry text, and obtain the handwritten poetry text weight sequence of the scenic spot image through the poetry text classifier, and train the classifier at the same time; Step 3: Input the extracted handwritten poetry text feature map into the handwritten poetry detection network based on FPN for fusion, obtain the single-character Gaussian heat map of the handwritten poetry text, and then obtain the text area position information of the scenic spot handwritten poetry image through the multi-character text box linking algorithm; Step 4: Crop the scenic area handwritten poetry image according to the extracted text image position information, and input the cropped area into the ACE-based handwritten poetry recognition network to perform Encoder-Decoder processing of the handwritten poetry text to obtain the recognition result of the scenic area handwritten poetry image; Step 5: Input the relevant attribute information of the scenic spot handwritten poem image into the scenic spot knowledge graph, and obtain the search knowledge result set through graph search. Use the recognition result obtained in step 4 and the search knowledge result set through the scenic spot handwritten poem matching algorithm to obtain the final text recognition result of the scenic spot handwritten poem image; The details are as follows: (1) Detection technology of handwritten poems in scenic spots based on FPN It is divided into three parts: handwritten poetry spatial feature extraction network, character key point calibration algorithm and multi-character text box linking algorithm. First, the scenic spot handwritten poetry image is input into the handwritten poetry spatial feature extraction network to extract the handwritten poetry text features. Then the extracted poem text features are marked by the character key point calibration algorithm to mark the single character key points; finally, the marked single character key points are processed by the multi-character text box linking algorithm to obtain the coordinate information of the scenic spot handwritten poem text area, and the information is passed as input to the scenic spot handwritten poem image text character recognition module; its detailed description is as follows: The feature extraction of the scenic spot handwritten poetry image is carried out through VGG16, and the poetry text feature map of the scenic spot handwritten poetry image is obtained. The backbone network of the multi-feature module fusion uses a feature pyramid network; a 7-layer pyramid network is adopted, and the layers are interconnected to promote feature reuse; in the pyramid network, a bottom-up path and a top-down path are used for horizontal pairing for data transmission; 1) Bottom-up pathway: refers to the upward flow of data; each layer includes a convolution layer, a pooling layer, an activation function layer and a loop layer; this pathway obtains 7 multi-scale feature maps, marked as {c1, c2, c3, c4, c5, c6, c7}, different feature maps record different levels of text features, low-level features reflect the shallower level of poetry text feature boundaries, and high-level features reflect the deeper level of poetry text features; 2) Top-down path: refers to the downward flow of data. By upsampling the feature map, the output of the previous layers is used as the input of the current layer. At the same time, the deformable convolution module Def-Incept is added to extract the features of the partially deformed text, and then a multi-layer feature map {p1, p2, p3, p4, p5, p6, p7} is generated. Finally, a convolution operation is performed on the feature map, which reduces the number of parameters and eliminates the confounding effect caused by deformation. The following formula (1) shows the feature extraction process; In the formula, Conv represents the convolution operation, Indicates feature fusion, UpSample indicates upsampling operation, and Def-Incept indicates deformable convolution operation. The output information of this operation is processed by Gaussian kernel function to obtain Gaussian heat map of key nodes of characters in image text. The multi-character text box linking algorithm takes the Gaussian heat map of the key character nodes of the image text as a prerequisite, and calculates the link relationship between the key character nodes to obtain the final scenic spot handwritten poetry image text detection box; the calculation process of the multi-character text box linking algorithm will be introduced in detail below; Firstly, the Gaussian heat map of the key points of each character in the image text of the handwritten poems in the scenic area is analyzed and calculated to obtain the maximum diameter of the Gaussian heat map of each character. A square box is drawn with the maximum diameter as the side length to mark the text box position of a single character. Secondly, half of the diagonal length of the single-character text box is selected as the initial value of the radius r of the outward radiating circle, and the maximum value max of the radius r is set according to the maximum side length of the input image. Then, the step length is set to The radius of the radiation circle is continuously increased, and the exploration is carried out in the direction of the text input by the user. If another character text box is encountered, it stops and links the current character text box with the center of the encountered text box. If the radius r reaches the maximum value, it means that the text box link is ended. Finally, the linked text boxes are integrated to obtain the final result of the location of the text detection area of ​​the scenic spot handwritten poetry image. (2) ACE-based recognition method for handwritten poems in scenic spots By using the text recognition network of scenic spot handwritten poems, the local text area obtained by the scenic spot handwritten poems text detection is first subjected to image normalization processing to make the data more standardized; then the processed image data is input into the handwritten poems character feature extraction network to serialize and encode the poem text features; finally, the serialized code of the poem text features is decoded by the character recognizer to obtain the preliminary scenic spot handwritten poems text recognition results; its detailed description is as follows: After normalizing the image of handwritten poetry text in the scenic area, character features will be extracted from the processed text image. The convolutional recurrent neural network CRNN will be used as the backbone network to extract features from the image text of handwritten poetry in the scenic area. The character feature extraction network structure is mainly composed of two parts: a convolutional layer and a recurrent network layer. First, the normalized poetry text image is input into the convolutional layer to extract the convolutional feature map of the image. Then, the extracted convolutional feature map is input into the recurrent network layer to continue extracting text sequence features based on the convolutional features, and the extracted features are output. The aggregate cross entropy ACE algorithm will be used to decode the feature sequence of the scenic spot handwritten poetry image text. In the initial stage of model training, the gain coefficient is set to 1; in the subsequent training stage, the gain coefficient is set to the number of characters in the text in the image; (3) Poetry correction technology based on knowledge graph Taking the description information of the poem queried by the user as the search condition, the scenic spot knowledge graph is used for graph search to obtain the search result set C; Secondly, the search result set C and the preliminary recognition result x of the scenic spot handwritten poem text image are segmented by the word segmentation algorithm f(·) and the obtained search result set keyword matrix S n×m And the preliminary recognition result matrix E of the scenic spot handwritten poetry text image 1×m , where n represents the number of entities in the search result set, m represents the number of processed keywords, k represents the index position of the entity in the search result set, f(·) represents the word segmentation function for text segmentation, C[k] represents a single text in the search result set, and x represents the preliminary recognition result of the scenic spot handwritten poetry text image; its calculation formula is shown in (2) and (3); S k×m =f(C[k]) (2) E 1×m =f(x) (3) Then, the entity keywords in the matrix obtained above are calculated using the distribution model for generating word vectors; each entity keyword vector v e The calculation formula is shown in (4), where e represents a single sample in the matrix, g(·) represents the distribution model function that generates the word vector, i represents the index position of the entity keyword, and S n×m [e] represents a single sample data in the keyword matrix of the search result set, S n×m [e][i] represents the entity keyword data of a single sample in the keyword matrix of the search result set being gradually taken; The obtained vector v of the poem text e After normalization, the final text vector v of the poem text is obtained, and the n text vectors are combined to generate the search result vector set V; The calculation formula is shown in (5); After the preliminary recognition result of the scenic spot handwritten poem text image is also processed as above, the final preliminary recognition result vector q of the scenic spot handwritten poem text image is obtained; Finally, the similarity between the preliminary recognition result vector q of the scenic spot handwritten poetry text image and the text vector v of each entity in the search result vector set V is calculated through VSM. The poem text vector with the highest similarity between the preliminary recognition result vector q of the scenic spot handwritten poetry text image and the search result vector set V is taken, and the corresponding text Result in the search result set C is output as the scenic spot handwritten poetry text recognition result. The calculation formula is shown in (6) and (7), where VSM(·) represents the vector space model calculation function, j represents the index position of a single search result vector, s1 and s2 are two texts, and their word frequencies are represented by a and b respectively. t and b t Indicates, t represents the index position of the frequency of the text word; The knowledge reasoning method is used to obtain the knowledge nodes that are "associated" with the search result Result knowledge node. The triple representation can be described as (?, association, Result). The relevant poetry knowledge node information is obtained through knowledge reasoning and calculation to realize the knowledge recommendation service.

Images