A method and system for Thangka image recognition
By improving the encoder and decoder structure and combining ResNeXt-50, MobileNet and Transformer networks, the problem of automatic description of Thangka images was solved, and accurate description and digital protection of Thangka images were achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- QINGHAI UNIVERSITY
- Filing Date
- 2022-09-19
- Publication Date
- 2026-07-17
AI Technical Summary
Existing technologies struggle to effectively understand and describe the content of Thangka images, especially lacking multimodal learning research when machines automatically describe Thangka images.
An improved encoder structure is adopted, which combines ResNeXt-50 and MobileNet convolutional neural networks to extract Thangka image features, and the Transformer network is used to optimize the decoder for text generation. Layer normalization technology is used to improve model performance.
It enables accurate description of Thangka images, supports large-scale searching and digital preservation of Thangka images, and simplifies users' understanding of Thangka cultural resources.
Smart Images
Figure CN115690453B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of computer image description technology, and specifically to a method and system for Thangka image recognition. Background Technology
[0002] As a regional and religious art form, Thangka is the core competitiveness of the cultural and artistic industry in the Qinghai-Tibet region. However, because the main deities, ritual implements, and historical allusions depicted in Thangka paintings involve knowledge of religion, region, history, and ethnicity, understanding the content of Thangka images remains very difficult for most people. Currently, the understanding of Thangka imagery is primarily based on the expertise of specialists, with little research on the descriptive aspects of Thangka images.
[0003] Automatic image description, also known as image captioning, can be simply understood as enabling machines to "talk about pictures." Specifically, it involves automatically generating descriptive text for input images. This task requires the system to establish a mapping from images to corresponding text descriptions. This necessitates not only recognizing salient objects in the image but also possessing the linguistic ability to describe the most salient aspects of the image. The former involves feature extraction from computer vision, while the latter involves text generation.
[0004] From a computer vision perspective, automatic image description, compared to other image understanding tasks, not only requires identifying targets and their attributes in an image, but also determining the relationships between targets and describing these features using grammatically and semantically correct text. From a text generation perspective, automatic image description is viewed as a translation problem of "translating" an image into text. Therefore, the encoder-decoder framework from machine translation is introduced to solve the automatic image description task. The encoder-decoder framework consists of two parts: the encoder part understands the input image, and current mainstream image understanding methods use Convolutional Neural Networks (CNNs) to extract image features; the decoder part uses the image feature vectors extracted by the CNN network as input to generate descriptive text, which is essentially still a text generation task in the field of NLP. Therefore, the base network used is a Recurrent Neural Network (RNN), especially a Long Short-Term Memory (LSTM) network to generate text sequences. The Transformer network is constructed based on the self-attention mechanism, which, compared to the natural temporal structure of LSTM, is more in line with the compositional rules of Thangka paintings, and can further improve the performance of the entire model.
[0005] In conclusion, Thangka is a unique cultural resource with distinctive ethnic characteristics. However, research on the integration of information technology for the protection and development of Thangka, especially on multimodal learning tasks involving Thangka images, is limited. Therefore, how to introduce deep neural networks to assist in understanding the content of Thangka images remains a worthy research topic. Summary of the Invention
[0006] To address the aforementioned technical problems, the present invention aims to provide a method and system for Thangka image recognition.
[0007] This invention protects a method for Thangka image recognition, specifically including the following steps:
[0008] S1, Input the acquired Thangka image into the encoder to extract the main feature vector of the Thangka image;
[0009] S2, input the main feature vector and the other main feature vectors obtained into the decoder to obtain target feature information;
[0010] S3, after fusing the main feature vector and the target feature information text, the corresponding text description is obtained.
[0011] Furthermore, in step S1, the encoder consists of two convolutional neural networks, specifically including a main network ResNeXt-50 and a branch network MobileNet running in parallel with the main network. The branch network MobileNet acts as an auxiliary classifier to simultaneously extract and output the main image and other feature vectors of the main image in the Thangka image. The main image feature vector is finally output by the encoder by fusing the extracted features from the two convolutional neural networks.
[0012] In the optimized ResNeXt-50 network, a global average pooling layer is added after the feature vector is output from the last convolutional layer to control the scale of the feature vector. The vector output from the global average pooling layer is passed through a fully connected layer to obtain the final dominant feature vector.
[0013] Furthermore, in step S2, the decoder adopts a transformer-based model unit structure, which uses two multi-head self-attention units and one feedforward fully connected unit, with the feedforward fully connected unit added between the two multi-head self-attention units for construction; and uses layer normalization (LN) to constrain the scale problem of multi-term embedding accumulation.
[0014] This invention also protects a Thangka image recognition system, including a front-end page and a back-end part;
[0015] The front-end page is used for user interaction and includes the system homepage, the Thangka image upload page, and the Thangka image description text display page;
[0016] The backend is used for uploading Thangka images and generating descriptive text;
[0017] The Thangka image upload involves storing the uploaded Thangka image as a temporary file on the server, and the file's storage path on the server is then passed to the main deity detection part of the calling model.
[0018] The method for generating the descriptive text is the same as the steps described in the above-mentioned Thangka image recognition method.
[0019] Furthermore, the description text generation also includes a template method, which is used to display the generated Thangka image text and its extended content (religious meaning). The extended content is stored in the server in the form of static data and can be directly called when needed.
[0020] Compared with existing technologies, the present invention has the following beneficial effects:
[0021] To improve feature extraction performance at the image level, this invention improves the encoder structure by using two convolutional neural networks to extract image features and fusing the extracted feature vectors as the encoder output. The ResNet-50 network structure is modified by adding a parallel convolutional tail chain to simultaneously extract features from the main image and other features of the Thangka. The decoder is modified by using a Transformer network to optimize the text generation model of the Long Short-Term Memory network and applying conditional layer normalization to the Transformer network, ultimately resulting in a deep learning-based Thangka image description algorithm. This method introduces image description into Thangka scenes, better aiding in the understanding of Thangka images. Combined with text information search technology, it allows for convenient searching of large-scale Thangka images, which is of great significance for the digital protection and development of Thangka cultural resources. The system designed based on this method is simple to operate, has a concise interface, and is suitable for all users. Attached Figure Description
[0022] Figure 1 This is a flowchart of the method of the present invention;
[0023] Figure 2 The network structure is ResNeXt50;
[0024] Figure 3 The formula for obtaining the final image feature vector by passing the vector output by the global average pooling layer through a fully connected layer;
[0025] Figure 4 Encoder structure;
[0026] Figure 5 Here is the formula for the encoder's loss function;
[0027] Figure 6 The formula for the BCEwithLogits function used for the encoder loss function;
[0028] Figure 7 Flowchart for BCE with Logits Loss;
[0029] Figure 8 The formula for calculating BCEwithLogits function;
[0030] Figure 9 It is a Transformer unit structure;
[0031] Figure 10 This is a system flowchart of the present invention;
[0032] Figure 11 This is the homepage of the system of this invention;
[0033] Figure 12 This is the image upload page for the system of this invention;
[0034] Figure 13 This is the text display page for the Thangka image description of the system of this invention. Detailed Implementation
[0035] The technical solution of the present invention will be clearly and completely described below with reference to the embodiments. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0036] Example 1
[0037] Optimization process of the Thangka image recognition algorithm model of this invention
[0038] 1. Methodological Approach to Thangka Image Recognition
[0039] First, in order to improve the feature extraction effect at the image end, the encoder structure was improved. Two convolutional neural networks were used to extract the features of the image, and the extracted feature vectors were fused as the output of the encoder.
[0040] Second, the ResNet-50 network structure was modified by adding a parallel convolutional tail chain to extract features from the main deity and other features of the Thangka simultaneously.
[0041] Third, the decoder was modified by using the Transformer network to optimize the text generation model of the Long Short-Term Memory network and by using conditional layer normalization to normalize the Transformer network.
[0042] 2. Encoder
[0043] In the Thangka image description task, the encoder's task is to extract the image features of the input Thangka image, while the convolutional neural network processes the image continuously through multiple layers of convolution, which can effectively extract information at different scales in the image.
[0044] Typically, convolutional layers of different depths extract information from different levels of an image. High-level features in a convolutional neural network usually contain more abstract and complex semantic information, while low-level features focus more on spatial information such as edges and contours. Therefore, increasing the depth of a convolutional neural network helps extract more abstract semantic information. However, when the convolutional neural network becomes too deep, gradient explosion or vanishing extraction can occur due to the need for backpropagation. To alleviate this, this application uses ResNet-50 as the base network for image feature extraction and optimizes it by adding a parallel tail chain, resulting in the main network ResNeXt-50 for the encoder part. The specific structure of ResNeXt-50 is shown in the attached figure. Figure 2 As shown.
[0045] The ResNeXt-50 network in this application adds a global average pooling layer after the feature vector output from the last convolutional layer to control the scale of the feature vector. The vector output from the global average pooling layer is then passed through a fully connected layer to obtain the final image feature vector. The formula is attached. Figure 3 As shown in the diagram; where img represents the input Thangka image, convi represents the vector output by the last convolutional layer, and featurei represents the final output image features. The value of i is either role or act; if i is role, it indicates the main network; if i is act, it indicates a parallel network.
[0046] Thangka images primarily describe the type of the main deity, as well as their ritual implements, hand gestures, and postures. Typically, attention mechanisms are added to the image processing layer to better extract information about the main deity and other related details. However, as painted images, Thangka images often lack depth of field, making it difficult to distinguish between foreground and background. Therefore, to better extract Thangka image features, this paper uses MobileNet as a branch network to extract features from Thangka images. Finally, the feature vectors from the two convolutional neural networks are fused to obtain the final image feature vector output by the encoder. The encoder structure is attached. Figure 4 As shown.
[0047] The basic network was modified by adding a branch as an auxiliary classifier to simultaneously extract and output features such as the main deity, hand gestures, and posture of the Thangka image. Therefore, the encoder's loss function formula is attached. Figure 5 As shown.
[0048] Where, loss base The loss function of the ResNeXt-50 network consists of two parts: the loss functions of the main network and the loss functions of the parallel network. trim This represents the loss function of the MobileNet network. The encoder's loss function is calculated using the BCEwithLogits function, as shown in the appendix. Figure 6 As shown.
[0049] Where, base out1 and base out2 `basetrim` represents the output of the main and parallel networks of the ResNeXt-50 network, `basetrim` represents the output of the MobileNet network, and `RoleTarget` and `ActTarget` represent the true values of the principal deity category and the principal deity's hand gesture. The BCEwithLogits function first normalizes the input vector using the Sigmoid function, and then calculates the loss using the binary cross-entropy function. The calculation process is shown in the attached figure. Figure 7 As shown, by appendix Figure 7 The formula for calculating BCEwithLogits is attached. Figure 8 As shown.
[0050] Using the BCEwithLogitis function can make the classification of the main deity more accurate, and can also extract more precise semantic information such as ritual implements, hand gestures, and postures, ultimately resulting in a more accurate description of the Thangka image.
[0051] 3. Decoder
[0052] The decoder part references the Transformer model, using its self-attention mechanism and fully connected feedforward neural network layers to construct self-attention sublayers and fully connected feedforward neural network sublayers, respectively. The Transformer network abandons the recurrent structure in RNNs and uses a multi-head self-attention mechanism entirely to learn the information features in the text. This allows for better modeling of dependencies in long texts while performing parallel training, resulting in significant improvements in both performance and efficiency. Furthermore, to increase the model's generalization ability, the extracted Thangka image feature vectors are masked in the input part, as shown in the following formula:
[0053] feature encoding =attention_mask(feature) act )
[0054] Therefore, in the Thangka image description model, the objective function that ultimately needs to be optimized is the log-maximum likelihood function:
[0055]
[0056] Where W is a matrix composed of the embedding vectors of each word in the sentence, which can be represented as {W1, W2, ..., W...} L}, where L is the sentence length and W i Let p(W) represent the word embedding vector of the i-th word in the sentence. i |W 1:i-1 feature encoding ) represents the probability of generating the i-th word based on the first i-1 words and the image feature vector.
[0057] In the field of machine vision, normalization is usually performed at the batch level. However, in text data, the information correlation between different batches is not high, and the lengths of different sentences are also inconsistent. Therefore, normalization at the batch level does not have a significant effect on reducing variance and may even lose the differences between different sentences. Therefore, the Transformer network uses Layer Normalization (LN) to constrain the scale problem of multi-word embedding accumulation, which effectively reduces variance.
[0058] In the Thangka image description task, the category of the main deity in the Thangka has a significant impact on the description. Determining the category of the main deity determines most of the semantic information in the Thangka description, including information such as ritual implements and hand gestures. Therefore, Conditional Layer Normalization (CLN) is used as the normalization method to construct self-attention units and feedforward fully connected units, as shown in the attached diagram. Figure 9 As shown.
[0059] The decoder network is constructed using two self-attention units and one fully connected unit, with the fully connected unit added between the two self-attention units. Assuming N represents the number of images and γc is the conditional input for the conditional layer normalization, the model's loss function is as follows:
[0060]
[0061] Example 2
[0062] A method for Thangka image recognition specifically includes the following steps:
[0063] S1, the acquired Thangka image is input into an encoder to extract the main feature vector of the Thangka image; wherein, the encoder consists of two convolutional neural networks, specifically including a main network ResNeXt-50 and a branch network MobileNet running in parallel with the main network. The branch network MobileNet acts as an auxiliary classifier to simultaneously extract and output the main feature vector and other feature vectors of the main image of the Thangka image; the main feature vector is finally output by the encoder by fusing the extracted features from the two convolutional neural networks; the ResNeXt-50 network adds a global average pooling layer after outputting the feature vector at the end of the last convolutional layer to control the scale of the feature vector, and the vector output by the global average pooling layer is passed through a fully connected layer to obtain the final main feature vector;
[0064] S2, the main feature vector and other obtained main feature vectors are input into the decoder to obtain target feature information; the decoder adopts a transformer-based model unit structure, which uses two multi-head self-attention units and one feedforward fully connected unit, with the feedforward fully connected unit added between the two multi-head self-attention units; layer normalization (LN) is used to constrain the scale problem of multi-term embedding accumulation;
[0065] S3, after fusing the main feature vector and the target feature information text, the corresponding text description is obtained.
[0066] Example 3
[0067] 1. System Design Concept
[0068] The goal is to enable users to generate descriptive text for local Thangka digital images via a browser, aiding their understanding of the Thangka image's content. Since the generated text involves Buddhist knowledge, and the template-based text generation method is entirely controllable, this system chooses the template method to generate the Thangka image descriptive text. To further enhance user comprehension, extended content has been added to the model-generated text. This extended content also relates to specific areas of knowledge concerning the main deity, allowing users to learn more about Thangka images. (Appendix) Figure 10 This is the main process of the system of the present invention.
[0069] 2. System Implementation
[0070] 1) Implementation of the front-end page
[0071] This section covers Web and JSP technologies, primarily focusing on human-computer interaction. Web technologies include HTML, CSS, and JavaScript. HTML stands for Hyper Text Markup Language, a language that uses tags to describe web pages. Browsers organize content using these tags within HTML files. CSS describes how HTML elements are displayed on screens, paper, or other media. Its purpose is to separate content from presentation; content is organized by HTML tags, while its presentation is controlled by CSS selectors and attribute values. JavaScript is a programming language for programming web pages. By programming the behavior of web pages using JavaScript, simple human-computer interactions and some animations can be achieved. Web pages developed using HTML, CSS, and JavaScript are static. However, the Thangka descriptions displayed in this system are dynamically generated based on uploaded Thangka images. Therefore, JSP technology was used to develop a dynamic display page. Java code is embedded in the HTML document of the display page to programmatically display the Thangka descriptions dynamically generated by the backend through model calls.
[0072] 2) Implementation of the backend
[0073] This part of the development involves some Java EE technologies as well as hybrid Java and Python development techniques. Java EE is a set of standards for developing web applications, and the technologies that can be used include JSP technology for the front end, Servlet technology for the back end, and JDBC technology for accessing database data. The backend logic of this system mainly includes image uploading and calling deep learning models to generate image descriptions, all of which are developed under the Java EE standard. Since the main development environment of this system is Java EE, but the deep learning network part is developed using Python, hybrid Java and Python development techniques are needed to call the Python deep learning model. There are three main methods for Java to call Python deep learning models.
[0074] The first method is to save the various parameters of the trained model in the form of a structured document, and then use Java to reimplement the network structure in the background and read the structured document containing the saved parameters. This way, the previously trained model can be reproduced. However, the disadvantages of this approach are obvious. On the one hand, the workload is very large because the many deep learning frameworks in Python cannot be used, and the entire network can only be implemented step by step from the bottom layer. On the other hand, because the network can only be implemented step by step from the bottom layer, the probability of bugs will increase dramatically.
[0075] The second method is to package the Python code into a JAR file and directly call the Python code in the JAR file from the Java code. The disadvantage of this method is that many third-party libraries cannot be packaged into JAR files, so many deep learning frameworks cannot be used. The disadvantages are basically the same as the first method: it is labor-intensive and prone to errors.
[0076] The third method is to create a thread using Java's thread class, call the Python interpreter through this thread, run the Python code directly with parameters passed to the Python interpreter, and then return the result of the Python code to the Java code for use.
[0077] The method used in this system is the third method, which uses a process class object to run the model trained in deep learning. This approach can ensure the high efficiency and correctness of Python code execution. In Java, you only need to get the result of the Python code to proceed to the next step.
[0078] 3. System Display
[0079] A Thangka image recognition system includes a front-end page and a back-end component; the front-end page is used for user interaction and includes the system homepage (attached). Figure 11 Thangka image upload page (attached) Figure 12 ) and Thangka image description text display page (attached) Figure 13 The backend is used for uploading Thangka images and generating descriptive text. Uploading Thangka images involves storing them as temporary files on the server, with the file's storage path on the server being passed to the model's main deity detection section. The descriptive text generation method follows the steps described in the Thangka image recognition method. The descriptive text generation also includes a template method to display the generated Thangka image text and its extended content (religious meaning). This extended content is stored as static data on the server and can be directly accessed when needed.
[0080] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.
Claims
1. A method for Thangka image recognition, characterized in that, Specifically, the steps include the following: S1, the acquired Thangka image is input into the encoder to extract the main feature vector of the Thangka image. The encoder consists of two convolutional neural networks, specifically including a main network ResNeXt-50 and a branch network MobileNet that runs parallel to the main network. The branch network MobileNet acts as an auxiliary classifier to simultaneously extract and output the main feature vector and other feature vectors of the main feature of the Thangka image. By fusing the extracted features from the two convolutional neural networks, the final main feature vector output by the encoder is obtained. S2, in the Thangka image description task, the main deity feature vector and other obtained main deity feature vectors are input to the decoder. The decoder adopts a transformer-based model unit structure and uses layer normalization (LN) to constrain the scale problem of multi-term embedding accumulation. The layer normalization is specifically Conditional Layer Normalization (CLN), which is used as the normalization method to construct self-attention units and feedforward fully connected units. The transformer model unit structure uses two multi-head self-attention units and one feedforward fully connected unit. The feedforward fully connected unit is added between the two multi-head self-attention units to construct the decoder network. The loss function of the model is: Where N represents the number of images, γc is the conditional input for conditional layer normalization; target feature information is obtained through the above process; W is a matrix composed of the embedding vectors of each word in the sentence; feature encoding To mask the extracted Thangka image feature vector in the input part; S3, after fusing the main deity feature vector and the target feature information text, the corresponding text description is obtained; the text description generation also includes a template method, which is used to display the generated Thangka image text and the extended content containing religious meaning; the extended content is stored in the server in the form of static data and can be directly called when needed; this extended content is also domain knowledge related to the main deity.
2. The method for Thangka image recognition according to claim 1, characterized in that, The ResNeXt-50 network adds a global average pooling layer after the feature vector is output from the last convolutional layer to control the scale of the feature vector. The vector output from the global average pooling layer is passed through a fully connected layer to obtain the final dominant feature vector.
3. A Thangka image recognition system, characterized in that, Including the front-end page and the back-end; The front-end page is used for user interaction and includes the system homepage, the Thangka image upload page, and the Thangka image description text display page; The backend is used for uploading Thangka images and generating descriptive text; The Thangka image upload involves storing the uploaded Thangka image as a temporary file on the server, and the file's storage path on the server is passed to the main deity detection part of the calling model. The method for generating the descriptive text is the step described in claim 1 or 2.