An efficient student Chinese painting appreciation system and method based on the LLaVA model

Abstract

This invention relates to an efficient student Chinese painting appreciation system and method based on the LLaVA model, including a client, a server, and an improved LLaVA model; the client is used to collect electronic images of traditional Chinese paintings and textual appreciation and analysis pairs; the client is also used to process the collected total dataset D CZO The system is divided into several parts. The server has an improved LLaVA model deployment port, which communicates with the client via HTTP. The improved LLaVA model incorporates the ConvNext v2 model, the PainterProjecter algorithm, and the Large Language Model (LLM). The ConvNext v2 model extracts meticulous brushwork and artistic conception features from electronic images of Chinese paintings. The PainterProjecter algorithm fuses and aligns the features of Chinese painting and the features of Chinese appreciation text. The Qwen2.5-7B-Chat combined with the HalfLoRA large language model is used to output appreciation and analysis text and to transmit the output appreciation and analysis text to the client. This system enables intelligent appreciation of students' Chinese paintings, greatly helping to improve students' creative abilities through intelligent education.

Description

Technical Field

[0001] This invention belongs to the field of intelligent education technology, specifically relating to an efficient student Chinese painting appreciation system and method based on the LLaVA model. Background Technology

[0002] Chinese traditional culture contains boundless wisdom and charm, and with the development of the times, it is increasingly admired and loved by people around the world. Chinese painting, as a treasure of traditional culture, carries a rich historical legacy and profound cultural spirit. It is not only a display of visual art but also a medium for conveying emotions and philosophical thoughts. Through brush and ink, paintings express feelings through scenery and convey aspirations through objects. Since ancient times, countless literati and scholars have used Chinese painting to express their temperament, record the times, and express their personal ideals and pursuits, forming a unique artistic tradition of "expressing one's aspirations and feelings through painting."

[0003] However, students and enthusiasts often encounter numerous obstacles on the path of learning and creating Chinese paintings, especially feeling confused about appreciating masterpieces and evaluating their own work. Appreciating Chinese painting requires not only understanding techniques but also a deep understanding of the cultural connotations and the artist's emotional expression, which is a major challenge for beginners. Each creation, though a process of gradual improvement in individual skill and flashes of inspiration, often progresses slowly or even stagnates due to a lack of high-level guidance.

[0004] Due to the scarcity of professional guidance resources, many students and enthusiasts struggle to obtain direct feedback and instruction, which undoubtedly limits their artistic exploration and the release of their creativity. How to improve themselves without frequent direct guidance has become a major challenge for many Chinese painting learners.

[0005] Therefore, developing a Chinese painting appreciation system that can assist learning and provide objective evaluation and in-depth analysis is particularly important for promoting the inheritance and development of Chinese painting art. Summary of the Invention

[0006] The purpose of this invention is to overcome the shortcomings of the prior art and provide an efficient student Chinese painting appreciation system and method based on the LLaVA model, which solves the technical problem that students and enthusiasts find it difficult to obtain direct feedback and guidance due to the scarcity of professional guidance resources for Chinese painting.

[0007] To address the aforementioned problems, the technical solution of this invention is: an efficient student Chinese painting appreciation system based on the LLaVA model, comprising a client, a server, and an improved LLaVA model;

[0008] The client is used to collect electronic images of traditional Chinese paintings and textual appreciation and critique pairs, and to organize these pairs into one-to-one question-and-answer pairs. The organized electronic images of traditional Chinese paintings and their appreciation and critique texts form the overall dataset D. CZO The client is also used to process the collected total dataset D CZO The dataset is split, with one part used for training and the other for testing, where D... CTr D represents the training dataset. CCs D represents the test dataset and the training dataset. CTr Used for training the improved LLaVA model; Test dataset D CCs Used to test the trained, improved LLaVA model

[0009] The server has an improved LLaVA model deployment port, which communicates with the client via the HTTP protocol;

[0010] The improved LLaVA model incorporates the ConvNext v2 model, the PainterProjecter algorithm, and the Large Language Model (LLM). The ConvNext v2 model extracts meticulous brushwork and artistic conception features from electronic images of Chinese paintings. The PainterProjecter algorithm deeply fuses and aligns Chinese painting features with Chinese appreciation text features. The Large Language Model (LLM) includes the Qwen2.5-7B-Chat model and the HalfLoRA module. The Qwen2.5-7B-Chat combined with the HalfLoRA Large Language Model is used to output appreciation and analysis texts, and to transmit the output appreciation and analysis texts to the client for students to view and copy.

[0011] Optionally, the ConvNext v2 model is divided into four stages, each containing convolutional kernels of different sizes to focus on local features. The ConvNext v2 model also has GRN layers to extract features from the global image information of the Chinese painting. Moreover, the Chinese paintings in different stages have been downsampled from the previous stage, and the Chinese paintings in different stages are of different sizes.

[0012] Optionally, the PainterProjecter algorithm includes a multi-scale extended attention mechanism, a feedforward neural network, a HOSC activation function, and a multi-head cross-attention mechanism;

[0013] The multi-scale extended attention mechanism is used to re-extract cross-connection Chinese painting features from the features output in ConvNext v2, and the same applies to Chinese text features.

[0014] The FFN feedforward neural network and HOSC activation function are used to perform nonlinear transformations on the features output by the multi-scale extended attention mechanism;

[0015] A multi-head cross-attention mechanism is used to align and associate features between Chinese painting image features and Chinese text features;

[0016] The FFN feedforward neural network and HOSC activation function are also used to further fuse the aligned Chinese painting image features and text features, and output them to the LLM large language model.

[0017] Optionally, the multi-scale extended attention mechanism has different r values, and the different r values ​​represent different extension capabilities, so as to better associate features between Chinese painting images and appreciation texts in neighboring locations;

[0018]

[0019] ,

[0020] The formula for the HOSC activation function is as follows: .

[0021] Optionally, the Qwen2.5-7B-Chat model serves as a large language model that integrates the input of Chinese painting image features, text input, and prompt word input. The HalfLoRA module is located in the Qwen2.5-7B-Chat branch of the LLM model. The HalfLoRA module is used to add the Q and K matrices in the attention mechanism of the Qwen2.5-7B-Chat model using the HalfLoRA method, as shown in the following formula:

[0022]

[0023]

[0024]

[0025] Here, φ is a hyperparameter that can be adjusted by the user.

[0026] Optionally, the training dataset D CTr When training the improved LLaVA model, the number of training iterations EPOCH is less than H. e H e The size is a hyperparameter; the training dataset D CTr When performing mini-batch partitioning, the value of Batch is less than or equal to B. ca The value of miniBatch is less than or equal to M. cm B ca and M cm These are all hyperparameters, and hyperparameters have training error.

[0027] Optionally, H e The value is 6, B ca The value is 64, M cm The value is 16, the hyperparameter has training error, and MSE is used, with a learning rate of 16. The initial value is set to 0.00001.

[0028] Optionally,

[0029] D CZO = D CTr + D CCs

[0030] D CTr = 0.7∗D CZO

[0031] D CCs = 0.3∗D CZO .

[0032] Optionally, it also includes a camera for capturing electronic images of Chinese paintings and transmitting them to the client.

[0033] Optionally, the improved LLaVA model is deployed on an Ubuntu 22.04 system; its hardware includes a machine with two RTX 4090 graphics cards, and the client accesses and interacts with the model by accessing the server IP and a fixed port 8501.

[0034] Another objective of this invention is to provide a method for constructing an efficient student Chinese painting appreciation system based on the LLaVA model, comprising the following steps:

[0035] Step 1: Collect and organize traditional Chinese paintings and textual appreciation and analysis pairs. Organize these pairs into one-to-one question-and-answer pairs. The collected and organized data will form the overall dataset D. CZO ;

[0036] Step 2: Divide the Chinese painting dataset:

[0037] The total dataset D from step one CZO The dataset is split, with one part used for training and the other for testing; where D CTr D represents the training dataset. CCs This represents the test dataset;

[0038] Step 3: Improve the LLaVA model, including:

[0039] a) The Vision Encoder in the LLaVA model is improved to the ConvNext v2 model. The ConvNext v2 model is used to extract the fine brushwork features and artistic conception features in Chinese electronic images. The ConvNext v2 model is divided into four stages. Each stage contains convolutional kernels of different sizes to focus on local features. The ConvNext v2 model also has GRN layers to extract features from the global image information of Chinese paintings. Moreover, the Chinese paintings in different stages have been downsampled from the previous stage, and the Chinese paintings in different stages are of different sizes.

[0040] b. The LLaVA model incorporates the PainterProjecter algorithm for deep fusion and alignment of Chinese painting features and Chinese text features. The PainterProjecter algorithm includes a Multi-Scale Dilated Attention mechanism, a Feed Forward Network, a HOSC activation function, and a Multi-Head CrossAttention mechanism. First, the PainterProjecter algorithm uses the Multi-Scale Dilated Attention mechanism to extract Chinese painting features from the output of ConvNext v2 across connections, and the same applies to Chinese text features. Then, it uses an FFN feedforward network and a HOSC activation function to perform a nonlinear transformation on the features output by the Multi-Scale Dilated Attention mechanism. Next, it uses a Multi-Head CrossAttention mechanism to align and associate the Chinese painting image features and Chinese text features. Finally, these aligned Chinese painting image features and text features are further fused using a Feed Forward Network and a HOSC activation function, and the final output is fed into the LLM large language model.

[0041] In Multi-Scale Dilated Attention, different values ​​of r represent different expansion capabilities, in order to better associate features between Chinese painting images and appreciation texts in neighboring locations.

[0042]

[0043] ,

[0044] The formula for the HOSC activation function is as follows: ;

[0045] c. Improve the LLM model in the LLaVA model: Use the Qwen2.5-7B-Chat model as a large language model that integrates the understanding of Chinese painting image feature input, text input, and prompt word input;

[0046] d. Add the HalfLoRA module to the Qwen2.5-7B-Chat branch of the LLM model; the HalfLoRA module adds the Q and K matrices in the attention mechanism of the Qwen2.5-7B-Chat model using the HalfLoRA method; the formula is as follows:

[0047]

[0048]

[0049] ,

[0050] φ is an adjustable hyperparameter;

[0051] Step 4: Train the improved LLaVA model:

[0052] Using the training dataset D of Chinese paintings CTr To train the improved LLaVA model from step three, first ensure that the number of training iterations, EPOCH, is less than H. e H e The size is a hyperparameter; secondly, the training dataset D... CTr For mini-batch partitioning, the batch size must be less than or equal to B. ca The value of miniBatch is less than or equal to M. cm B ca and M cm These are all hyperparameters; hyperparameters also have training error;

[0053] Step 5: Model Validation

[0054] The trained and improved LLaVA model was tested on the Chinese painting test dataset. CCs ;

[0055] Step Six: Model Deployment

[0056] The improved LLaVA model is deployed to the Ubuntu 22.04 system, where it runs on the server; the client accesses and interacts with the model.

[0057] Step 7: System Deployment;

[0058] The system takes photos of students' Chinese paintings using a camera and saves them. The saved images are then transmitted to a server via HTTP. The improved LLaVA model on the server provides an electronic evaluation of the Chinese paintings.

[0059] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0060] 1. The efficient student Chinese painting appreciation system based on the LLaVA model provided by this invention utilizes an artificial intelligence multimodal model as its foundation, combined with high-definition camera equipment and local area network servers. By acquiring student Chinese paintings as electronic images, the system can accurately appreciate and analyze student-created Chinese paintings, helping students improve their creative skills. This can serve as an intelligent system for students to appreciate Chinese paintings. The efficient student Chinese painting appreciation system of this invention can provide students with very convenient appreciation and evaluation, and it has great practical application significance.

[0061] 2. The efficient student Chinese painting appreciation system based on the LLaVA model provided by this invention improves the Vison Encoder in the LLaVA model to ConvNext v2. ConvNext v2 can extract the meticulous brushwork features of Chinese painting and the artistic conception structure of the overall structure of Chinese painting at multiple scales.

[0062] 3. The efficient student Chinese painting appreciation system based on the LLaVA model provided in this invention designs the PainterProjecter multimodal alignment network. It aligns and fuses multi-scale image features containing overall artistic conception features and subtle brushwork features extracted from ConvNext v2 with the corresponding appreciation and evaluation text features. It can ensure that subtle brushwork features are associated with brushwork description text features, and at the same time, associate and fuse artistic conception image features with text description artistic conception features.

[0063] 4. The large language model in the improved LLaVA model of this invention combines Qwen2.5-7B-Chat with the HalfLoRA designed in this paper as a whole large language model. It can not only ensure the accurate recognition of Chinese text, but also ensure the embedding of knowledge in the vertical domain of Chinese painting, and ensure the accurate output of the system in this invention. Attached Figure Description

[0064] Figure 1 This is a schematic diagram of the appreciation system of the present invention;

[0065] Figure 2 This is a flowchart of the appreciation method of the appreciation system of the present invention;

[0066] Figure 3This is a schematic diagram of the PainterProjecter algorithm of the present invention. Detailed Implementation

[0067] The present invention will now be described in further detail with reference to the accompanying drawings and embodiments.

[0068] Example 1: As Figures 1-3 As shown, this embodiment provides an efficient student Chinese painting appreciation system based on the LLaVA model, including a client, a server, and an improved LLaVA model;

[0069] The client is used to collect and organize traditional Chinese paintings and their corresponding textual appreciation and critiques. It matches the organized texts with the images, and the compiled digital images of Chinese paintings and their accompanying textual appreciations form the overall dataset D. CZO Specifically, high-definition cameras were used to capture Chinese paintings. Some paintings could be directly collected as digital images from the internet. These Chinese paintings included landscape paintings and flower-and-bird paintings. The digital images of Chinese paintings also included paintings by some students and school teachers, totaling 360 pieces.

[0070] The client is also used to split Chinese painting image and text pairs, that is, to split the total collected dataset D CZO Split the dataset. Total dataset D CZO It was split into a Chinese painting training dataset and a test dataset; where D CTr D represents the training dataset. CCs This represents the test dataset;

[0071] D CZO = D CTr +D CCs

[0072] D CTr = 0.7∗D CZO

[0073] D CCs = 0.3∗D CZO ;

[0074] Training dataset D for Chinese painting CTr This is used to train the improved LLaVA model. First, ensure that the number of training iterations, EPOCH, is less than H. e H e The size of H is a hyperparameter; in this embodiment, H... e The chosen value is 6. Next, the training dataset D needs to be... CTr For mini-batch partitioning, the batch size must be less than or equal to B. caTo better ensure training speed and prevent oscillations and excessive errors in mini-batch gradients, this embodiment also adds a mini-batch, the value of which must be less than or equal to M. cm B ca and M cm These are all hyperparameters. In this embodiment, B... ca The value is 64, M cm The value is 16. Additionally, there are hyperparameters such as training error; this embodiment uses MSE and the learning rate. The initial value is set to 0.00001.

[0075] The ViT model used in the original LLaVA model simply divides the image into nine equal parts and extracts features sequentially. However, the meticulous brushwork in the Chinese painting image in this embodiment is mainly reflected in the subtle details of ink application, which are very small details that cannot be met by simple segmentation. Secondly, the design and arrangement of the artistic conception and blank space in Chinese paintings require a holistic perspective. Furthermore, the large language model used in the LLaVA model is the Llama model. However, Llama models are mostly trained on native English speakers, while the Chinese painting appreciation and analysis in this embodiment is in simplified Chinese, which places very high demands on Chinese language comprehension. The Llama model's training in Chinese is insufficient. Secondly, computation using only a large Chinese model is still insufficient for appreciating Chinese paintings because current large models have general capabilities. However, they lack basic knowledge and understanding of specialized vocabulary in the vertical field of Chinese painting. Finally, the alignment between the Chinese painting features and the text for appreciation directly determines the mutual understanding between text and image. Therefore, this embodiment needs further improvement in alignment capabilities. Therefore, this embodiment improves the LLaVA model.

[0076] The improved LLaVA model incorporates the ConvNext v2 model, PainterProjecter algorithm, Qwen2.5-7B-Chat model, and HalfLoRA module.

[0077] Specifically, given the inherent limitations of the ViT model in the LLaVA model in extracting the meticulous brushwork and artistic conception of Chinese paintings, this embodiment improves the Vision Encoder in the LLaVA model to the ConvNext v2 model. The ConvNext v2 model consists of four stages, each containing convolutional kernels of different sizes to focus on local features. The ConvNext v2 model also includes GRN layers to extract features from the global image information of the Chinese painting. Furthermore, the Chinese paintings in different stages have undergone downsampling from the previous stage, resulting in different sizes of Chinese paintings in different stages. All of these provide a solid foundation for extracting the meticulous brushwork information features of Chinese paintings.

[0078] To address the limited ability of the LLaVA model to align Chinese text with Chinese paintings, this embodiment designs the PainterProjecter algorithm to deeply fuse and align Chinese painting features and Chinese appreciation text features. The PainterProjecter algorithm includes a Multi-Scale Dilated Attention mechanism, a Feed Forward Network, a HOSC activation function, and a Multi-Head Cross Attention mechanism. In this embodiment, the PainterProjecter algorithm first uses the Multi-Scale Dilated Attention mechanism to extract cross-connection Chinese painting features from the features output in ConvNext v2, and the same applies to Chinese text features. Then, it uses an FFN feedforward network and a HOSC activation function to perform a nonlinear transformation on the features output by the Multi-Scale Dilated Attention mechanism. Next, it uses the Multi-Head Cross Attention mechanism to align and associate the Chinese painting image features and Chinese text features. Finally, these aligned Chinese painting image features and text features are further fused using a Feed Forward Network and a HOSC activation function, and the final output is fed into the LLM.

[0079] The Multi-Scale Dilated Attention mechanism uses different values ​​of r, where different values ​​of r represent different dilation capabilities. This allows for better feature association between Chinese painting images and appreciation texts located in neighboring positions.

[0080]

[0081] ,

[0082] The above formula is the formulaic expression for Multi-Scale Dilated Attention. Attn msda It is Multi-Scale Dilated Attention, which consists of three matrices: Q, K, and V, representing the operations performed on these three matrices.

[0083] The formula for the HOSC activation function is as follows: ,

[0084] In this embodiment, .

[0085] To address the issue of LLM's poor performance in understanding Chinese characters within LLaVA, this embodiment improves the LLM model within the LLaVA model. This embodiment uses the Qwen2.5-7B-Chat model as a large language model that integrates and interprets Chinese painting image features, text input, and prompt words.

[0086] To address the limitation of the improved LLaVA model in understanding specialized terms and concepts within the field of Chinese painting, this embodiment introduces the HalfLoRA module, which is added to the Qwen2.5-7B-Chat branch of the LLM model within the improved LLaVA model. This embodiment utilizes HalfLoRA in conjunction with Qwen2.5-7B-Chat to make the improved LLaVA model more suitable for the vertical field of Chinese painting appreciation and analysis. The HalfLoRA design adds the Q and K matrices to the attention mechanism of the Qwen2.5-7B-Chat model using the HalfLoRA method, as shown in the following formula:

[0087]

[0088]

[0089]

[0090] The above is the formula for HalfLoRA, where B1, B2, and A represent the embedding matrices. r1, r2, and r3 represent different rank combinations. Dim(B1) represents the dimension of the B1 matrix, and its calculation formula is the original weight Dim(W) in Qwen2.5. * Divide the dimension of matrix B2 by the splitting coefficient φ. dim(B2) represents the dimension of matrix B2.

[0091] Meanwhile, the r dimension in B1 is 4, the r dimension in B2 is 16, and φ is an adjustable hyperparameter. In this embodiment, the value of φ is 2.

[0092] Training dataset D CTr This dataset is used to train an improved LLaVA model. The trained model learns the unique features of Chinese paintings and can align and integrate these features with textual appreciation and analysis. The large language model learns to associate Chinese paintings with the textual appreciation, thus providing a corresponding evaluation based on the image features of the Chinese painting. Test dataset D CCs The improved LLaVA model is used to test the trained model. The model that passes the test can be used as the final model in this embodiment.

[0093] After successful testing, the improved LLaVA model is deployed to the Ubuntu system. Specifically, first, the RTX driver needs to be installed on the Ubuntu system; second, the CUDA dependency packages need to be installed; then, the corresponding dependency packages need to be installed on Ubuntu. The improved model is then copied to the Ubuntu system and started.

[0094] When in use, students and enthusiasts can capture electronic images of Chinese paintings through a camera. These electronic images can be uploaded to the server through a client. Specifically, the client uploads the captured electronic images to the server via the HTTP protocol.

[0095] The client and server communicate via the HTTP protocol, primarily using port 8501. Port 8501 on the server is the port on which the improved LLaVA model is deployed in this embodiment. After receiving the electronic image of the Chinese painting through port 8501, the LLaVA model can perform feature extraction.

[0096] The improved LLaVA model is used for feature extraction and output of text information. Specifically, the LLaVA model extracts fine brushwork features and artistic conception features from Chinese electronic images through the ConvNext v2 model, then associates image features with text features through painterProjecter, and finally outputs appreciation and analysis text through the large language model Qwen2.5-7B-Chat combined with HalfLoRA.

[0097] The Qwen2.5-7B-Chat combination HalfLoRA's large language model transmits the output appreciation and critique text to the client via the HTTP protocol. After receiving the text transmitted via the HTTP protocol, the client outputs the text to the computer screen. Students improve their creative abilities by appreciating and analyzing the Chinese paintings they have created based on the output results.

[0098] This embodiment presents an efficient student Chinese painting appreciation system based on the LLaVA model. The improved LLaVA model is deployed on the Ubuntu 22.04 system. The hardware includes a machine with two RTX 4090 graphics cards. The improved LLaVA model runs on the server, and clients access and interact with the model by accessing the server's IP address and a fixed port 8501.

[0099] The workflow of an efficient student Chinese painting appreciation system based on the LLaVA model in this embodiment is as follows:

[0100] I. High-definition camera captures Chinese paintings. Chinese paintings include landscape paintings and flower-and-bird paintings, including famous paintings from the ancient Song, Ming, and Qing dynasties, as well as contemporary works by renowned artists and paintings by students and teachers.

[0101] II. Compilation of Chinese Painting Images and Texts. This involves pairing the collected Chinese paintings with their corresponding textual analyses.

[0102] 3. Split the total dataset. The total dataset is split into a Chinese painting training dataset and a test dataset.

[0103] Fourth, ensure that the number of training iterations (EPOCH) is less than H. e H e The selected value is 6; secondly, the training dataset D needs to be... CTr For mini-batch partitioning, the batch size must be less than or equal to B. ca To better ensure training speed and prevent gradient oscillations and excessive errors in mini-batch, the value of this MiniBatch should be less than or equal to M. cm B ca and M cm These are all hyperparameters. This embodiment uses B. ca The value is 64. M cm The value is 16. Other hyperparameters include training error and learning rate. The initial value is set to 0.00001.

[0104] V. Improve the LLaVA model.

[0105] 6. Train the improved LLaVA model. This allows the improved LLaVA model to be optimized for the vertical domain of Chinese painting in this embodiment.

[0106] VII. Validating the Model on the Test Set. The trained, improved LLaVA model is tested on the test dataset, and the optimal model is saved.

[0107] 8. Model Deployment. Use the saved optimal model as the baseline model, copy it to the Ubuntu system, and install its dependencies.

[0108] 9. System Deployment. Connect the client and server with a network cable, enable port 8051 and SSH service on the server, connect the camera to the client computer, run the model and set it as a system service.

[0109] 10. Students and enthusiasts can capture electronic images of Chinese paintings using cameras, and these electronic images can be uploaded to the server via a client.

[0110] 11. The client uploads the collected electronic images to the server via the HTTP protocol.

[0111] 12. HTTP protocol upload to server for reception.

[0112] Thirteen, Improved LLaVA Model Feature Extraction Output Text Information. The LLaVA model extracts features from the received electronic images of Chinese paintings. The LLaVA model extracts the meticulous brushwork features and artistic conception features from the Chinese electronic images through the ConvNext v2 model, then associates the image features with the text features through painterProjecter, and finally outputs the appreciation and analysis text through the large language model Qwen2.5-7B-Chat combined with HalfLoRA.

[0113] 14. HTTP Transmission. The Qwen2.5-7B-Chat combined with HalfLoRA's large language model transmits the output appreciation and critique text to the client via the HTTP protocol.

[0114] 15. Client Reception. The client receives the streaming text transmitted via the HTTP protocol and converts it to UTF-8 file format.

[0115] 16. Output the results. Output the converted text to the computer screen for students to view and copy.

[0116] In response to the booming development of intelligent education, this embodiment utilizes an improved LLaVA model to meticulously create a highly efficient appreciation system designed for the accurate evaluation and analysis of students' Chinese painting works, effectively identifying their shortcomings. In today's society, the revival of traditional culture is unstoppable, and Chinese painting, as a leading example, has become a treasure of the global art world, highly respected. Chinese painting is profound and extensive; within its subcategories, whether it's the majestic landscape painting or the vibrant flower-and-bird painting, each category contains unique meticulous techniques and profound artistic conception, requiring painters to seek a perfect balance between delicate depiction, overall composition, and freehand expression. Students' daily practice works often require highly professional scrutiny as a basis for modification and appreciation. However, in reality, this is often influenced by differences in teachers' individual artistic cultivation and subjective preferences, which can lead to certain biases. Different people may have vastly different evaluations of the same work, bringing uncertainty to students' learning feedback. These challenges have spurred an urgent need for a universal and efficient Chinese painting appreciation system. This system should transcend the limitations of traditional teaching, using objective and comprehensive standards to promote the improvement of students' painting skills and the deepening of their artistic understanding. To address this issue, this embodiment utilizes an artificial intelligence multimodal model as a foundation, combined with high-definition camera equipment and a local area network server to develop an efficient Chinese painting appreciation system for students. Specifically, firstly, a high-definition camera is used to digitally photograph the paper-based Chinese paintings created by students. The ultra-high-definition photographs are stored on the client side, which then transmits the paintings via HTTP to a server deployed with the improved LLaVA model of this embodiment. The improved LLaVA model extracts features and performs semantic understanding on the transmitted paintings, ultimately generating an appreciation analysis. To better integrate the unique feature information of the Chinese painting field, this embodiment improves the VisionEncoder model in the LLaVA model to the ConvNextv2 algorithm, which is more capable of extracting detailed features and overall structure of Chinese paintings. Secondly, to better align the Chinese paintings with the lengthy appreciation text, this invention designs the PainterProjecter model. Furthermore, to ensure the quality of the generated appreciation and analysis, this paper uses the Qwen2.5-7B-Chat model and improves it to incorporate a HalfLoRA model structure. In summary, this embodiment of an efficient student Chinese painting appreciation system based on the LLaVA model can very well meet students' requirements. It has significant theoretical and educational practical application value and a wide range of application scenarios.

[0117] Example 2: This example provides an appreciation method for an efficient student Chinese painting appreciation system based on the LLaVA model, including the following steps:

[0118] Step 1: Capturing Digital Images of Chinese Paintings with a Camera. First, use a high-definition camera to capture digital images of the Chinese paintings. Some paintings can be directly collected from the internet. These Chinese paintings include landscape paintings and flower-and-bird paintings. The digital images also include paintings by students and school teachers, totaling 360 pieces.

[0119] Step 2: The client organizes the Chinese painting image and text pairs. The appreciation and analysis of the Chinese painting electronic images collected in Step 1 are collected and organized. The organized text and images are matched one-to-one, and the organized Chinese painting electronic images and appreciation and analysis texts are used as the total dataset.

[0120] Step 3: Split the Chinese painting image and text pairs. The dataset D of Chinese painting image and text pairs... CZO The dataset is split into a Chinese painting training dataset (Training dataset) and a testing dataset (testing dataset). D CTr This represents the training dataset, D. CCs This represents the test dataset.

[0121] D CZO = D CTr + D CCs

[0122] D CTr = 0.7∗D CZO

[0123] D CCs = 0.3∗D CZO ;

[0124] Step 4: Hyperparameter Selection. Training Dataset D for Chinese Painting CTr The improved model is then trained. First, ensure that the number of training iterations (EPOCH) is less than H. e H e The size of is a hyperparameter; in this embodiment, it is chosen to be 6. Next, the training dataset D needs to be... CTr For mini-batch partitioning, the batch size must be less than or equal to B. ca To better ensure training speed and prevent oscillations and excessive errors in mini-batch gradients, this embodiment also adds a mini-batch, where the value of the mini-batch is less than or equal to M. cm B ca and M cm These are all hyperparameters. This embodiment uses B. ca The value is 64. M cm The value is 16. Additionally, hyperparameters include training error; in this embodiment, MSE and the learning rate are used. The initial value is set to 0.00001.

[0125] Step 5: Improve the LLaVA model. The original LLaVA model uses the ViT model, which simply divides the image into nine equal parts and extracts features sequentially. However, the meticulous brushwork in the Chinese painting images in this embodiment is mainly reflected in the subtle details of ink application, which are very small details that cannot be met by simple segmentation. Secondly, the design and arrangement of the artistic conception and blank space in Chinese paintings require a holistic perspective. Furthermore, the large language model used in the LLaVA model is the Llama model. However, the Llama model is mostly trained on English as the native language, while the Chinese painting appreciation and analysis in this paper is in simplified Chinese, which places very high demands on Chinese language comprehension. The Llama model's training in Chinese is insufficient. Secondly, computation using only a large Chinese model is still insufficient for appreciating Chinese paintings. This is because current large models have general capabilities, but they lack basic knowledge and understanding of specialized vocabulary in the vertical field of Chinese painting. Finally, the alignment between the features of the Chinese painting and the text of the appreciation directly determines the mutual understanding between the text and the image. Therefore, this embodiment needs to further improve the alignment capability. Thus, this embodiment improves the LLaVA model.

[0126] Addressing the inherent limitations of the ViT model within the LLaVA model in extracting the meticulous brushwork and artistic conception of Chinese paintings, this embodiment improves the Vision Encoder in the LLaVA model to a ConvNext v2 model. It consists of four stages, each containing convolutional kernels of varying sizes to focus on local features, as well as GRN layers to extract features from the global image information of the Chinese painting. Furthermore, the Chinese paintings in different stages have undergone downsampling from the previous stage, resulting in paintings of different sizes in each stage. All of these factors provide a solid foundation for extracting the meticulous brushwork features of Chinese paintings.

[0127] To address the limited ability of the LLaVA model to align Chinese text with Chinese paintings, this embodiment designs the PainterProjecter algorithm to deeply fuse and align Chinese painting features and Chinese text features. The PainterProjecter algorithm includes a Multi-Scale Dilated Attention mechanism, a Feed Forward Network, a HOSC activation function, and a Multi-Head Cross Attention mechanism. In this embodiment, the PainterProjecter algorithm first uses the Multi-Scale Dilated Attention mechanism to extract Chinese painting features from the output features of ConvNext v2 across connections, and the same applies to the Chinese text features. Then, FFN and HOSC are used to perform a nonlinear transformation on the features output by the Multi-Scale Dilated Attention mechanism. Next, the Multi-Head Cross Attention mechanism aligns and associates the Chinese painting image features and Chinese text features. Finally, after aligning these features, the aforementioned Chinese painting image features and text features are further fused through the Feed Forward Network and the HOSC activation function, and the final output is given to the LLM.

[0128] In Multi-Scale Dilated Attention, different values ​​of r represent different expansion capabilities, which can better associate features between Chinese painting images and appreciation texts in neighboring locations.

[0129]

[0130] ,

[0131] The above formula is the formulaic expression for Multi-Scale Dilated Attention. Attn msda It is Multi-Scale Dilated Attention, which consists of three matrices: Q, K, and V, representing the operations performed on these three matrices.

[0132] The formula for the HOSC activation function is as follows: In this invention, .

[0133] To address the issue of LLM's poor performance in understanding Chinese characters within LLaVA, this embodiment improves the LLM model within the LLaVA model. This embodiment uses the Qwen2.5-7B-Chat model as a large language model that integrates the understanding of Chinese painting image features, text input, and prompt words.

[0134] To address the limitation of the improved LLaVA model in understanding specialized terms and concepts within the field of Chinese painting, this embodiment introduces the HalfLoRA module, which is added to the Qwen2.5-7B-Chat branch of the LLM model within the improved LLaVA model. This embodiment utilizes HalfLoRA in conjunction with Qwen2.5-7B-Chat to make the improved LLaVA model more suitable for the vertical field of Chinese painting appreciation and analysis. The HalfLoRA design adds the Q and K matrices from the attention mechanism of the Qwen2.5-7B-Chat model using the HalfLoRA method. The formula is as follows:

[0135]

[0136]

[0137]

[0138] The above is the formula for HalfLoRA, where B1, B2, and A represent the embedding matrices. r1, r2, and r3 represent different rank combinations. Dim(B1) represents the dimension of the B1 matrix, and its calculation formula is the original weight Dim(W) in Qwen2.5. * Divide the dimension of matrix B2 by the splitting coefficient φ. dim(B2) represents the dimension of matrix B2.

[0139] Meanwhile, the r dimension in D1 is 4, the r dimension in D2 is 16, and φ is an adjustable hyperparameter. In this embodiment, the value of φ is 2.

[0140] Step 6: Train the model. Train the improved LLaVA model from Step 5 using the hyperparameters from Step 4. The trained model can learn the unique features of Chinese painting and align and integrate these features with the text for appreciation and analysis. The large language model learns to associate Chinese painting with the text for appreciation, and thus can give its corresponding evaluation based on the image features of Chinese painting.

[0141] Step 7: DCC Testing. The model trained in Step 6 is tested on the test dataset. The model that passes the test can be used as the final model in this embodiment.

[0142] Step 8: Model and System Deployment. Deploy the model from Step 7 to the Ubuntu system. First, install the RTX driver on the Ubuntu system; second, install the CUDA dependency packages; then, install the corresponding dependency packages on Ubuntu; copy the improved model to the Ubuntu system and start the copied model.

[0143] Step 9: Collection of Student Chinese Paintings. Students and enthusiasts can use a camera to capture digital images of Chinese paintings, which can then be uploaded to the server via a client application.

[0144] Step 10: The client uploads the digital image of the Chinese painting. The client uploads the acquired digital image to the server via the HTTP protocol.

[0145] Step 11: HTTP Protocol Upload and Server Reception. The client and server transmit data via the HTTP protocol, primarily using port 8501. Port 8501 on the server is the port where the improved LLaVA model is deployed in this embodiment. After receiving the electronic image of the Chinese painting through port 8501, the LLaVA model can perform feature extraction.

[0146] Step 12: Feature extraction and output of text information using the improved LLaVA model. The LLaVA model extracts features from the received electronic images of Chinese paintings. The LLaVA model extracts meticulous brushwork and artistic conception features from the Chinese electronic images using the ConvNext v2 model, then associates image features with text features using painterProjecter, and finally outputs the appreciation and analysis text using the large language model Qwen2.5-7B-Chat combined with HalfLoRA.

[0147] Step 13: HTTP transmission. The Qwen2.5-7B-Chat combined with HalfLoRA's large language model transmits the output appreciation and critique text to the client via the HTTP protocol.

[0148] Step 14: Client Reception. After receiving the text transmitted via the HTTP protocol, the client outputs the text to the computer screen. Students use the output to appreciate and analyze their own Chinese paintings, thereby improving their creative abilities.

Claims

1. A highly efficient student Chinese painting appreciation system based on the LLaVA model, characterized in that, This includes clients, servers, and an improved LLaVA model; The client is used to collect electronic images of traditional Chinese paintings and textual appreciation and critique pairs, and to organize these pairs into one-to-one question-and-answer pairs. The organized electronic images and textual appreciation and critique pairs constitute the total dataset D. CZO The client is also used to process the collected total dataset D CZO The dataset is split, with one part used for training and the other for testing, where D... CTr D represents the training dataset. CCs D represents the test dataset and the training dataset. CTr Used for training the improved LLaVA model; Test dataset D CCs Used to test the trained, improved LLaVA model; The server has an improved LLaVA model deployment port, which communicates with the client via the HTTP protocol; The improved LLaVA model incorporates the ConvNext v2 model, the PainterProjecter algorithm, and the Large Language Model (LLM). The ConvNext v2 model extracts meticulous brushwork and artistic conception features from traditional Chinese painting electronic images. The PainterProjecter algorithm deeply fuses and aligns Chinese painting features with Chinese appreciation text features. The Large Language Model (LLM) includes the Qwen2.5-7B-Chat model and the HalfLoRA module. The Qwen2.5-7B-Chat combined with the HalfLoRA Large Language Model is used to output appreciation and critique texts and to transmit the output appreciation and critique texts to the client for students to view and copy. The ConvNext v2 model is divided into four stages, each containing convolutional kernels of different sizes to focus on local features. The ConvNext v2 model also has GRN layers to extract features from the global image information of Chinese paintings. Moreover, the Chinese painting features in different stages have been downsampled from the previous stage, and the Chinese painting features in different stages are of different sizes. The PainterProjecter algorithm includes a multi-scale expanded attention mechanism, a feedforward neural network, a HOSC activation function, and a multi-head cross attention mechanism. The multi-scale extended attention mechanism is used to extract cross-connection Chinese painting features from the features output in ConvNext v2, and the same applies to Chinese text features. The FFN feedforward neural network and HOSC activation function are used to perform nonlinear transformations on the features output by the multi-scale extended attention mechanism; A multi-head cross-attention mechanism is used to align and associate features of Chinese paintings and Chinese text. The FFN feedforward neural network and HOSC activation function are also used to further fuse the aligned Chinese painting image features and text features, and output them to the LLM large language model. The Qwen2.5-7B-Chat model serves as a large language model that integrates and understands Chinese painting feature input, text input, and prompt word input. The HalfLoRA module is located in the Qwen2.5-7B-Chat branch of the LLM model. The HalfLoRA module is used to add the Q and K matrices in the attention mechanism of the Qwen2.5-7B-Chat model using the HalfLoRA method, as shown in the following formula: Here, φ is an adjustable hyperparameter.

2. The efficient student Chinese painting appreciation system based on the LLaVA model according to claim 1, characterized in that, The multi-scale extended attention mechanism has different r values, and the different r values ​​represent different extension capabilities, so as to better associate features between Chinese painting images and appreciation texts in non-neighboring locations. , The formula for the HOSC activation function is as follows: .

3. The efficient student Chinese painting appreciation system based on the LLaVA model according to claim 1, characterized in that, Training dataset D CTr When used to train the improved LLaVA model, the number of training iterations EPOCH is less than H. e H e The size is a hyperparameter; the training dataset D CTr When performing mini-batch partitioning, the value of Batch is less than or equal to B. ca The value of miniBatch is less than or equal to M. cm B ca and M cm These are all hyperparameters.

4. The efficient student Chinese painting appreciation system based on the LLaVA model according to claim 3, characterized in that, H e The value is 6, B ca The value is 64, M cm The value is 16, the hyperparameter has training error, and MSE is used, with a learning rate of 16. The initial value is set to 0.00001.

5. The efficient student Chinese painting appreciation system based on the LLaVA model according to claim 1, characterized in that, D CZO = D CTr + D CCs D CTr = 0.7∗D CZO D CCs = 0.3∗D CZO 。 6. The efficient student Chinese painting appreciation system based on the LLaVA model according to claim 1, characterized in that, The improved LLaVA model is deployed on the Ubuntu 22.04 system; Its hardware includes a machine with two RTX 4090 graphics cards. Clients access and interact with the model by accessing the server IP and a fixed port 8501.

7. A highly efficient student-based Chinese painting appreciation system based on the LLaVA model, characterized in that, Includes the following steps: Step 1: Collect and organize traditional Chinese paintings and textual appreciation and analysis pairs. Organize these pairs into one-to-one question-and-answer pairs. The collected and organized data will form the overall dataset D. CZO ; Step 2: Divide the Chinese painting dataset: The total dataset D from step one CZO The dataset is split, with one part used for training and the other for testing; where D CTr D represents the training dataset. CCs This represents the test dataset; Step 3: Improve the LLaVA model, including: a) The Vision Encoder in the LLaVA model is improved to the ConvNext v2 model. The ConvNext v2 model is used to extract the fine brushwork features and artistic conception features in the electronic image of traditional Chinese painting. The ConvNext v2 model is divided into four stages. Each stage contains convolutional kernels of different sizes to focus on local features. The ConvNext v2 model also has GRN layers to extract features from the global image information of Chinese painting. Moreover, the Chinese painting features in different stages have been downsampled from the previous stage, and the Chinese painting features in different stages are of different sizes. b. The LLaVA model incorporates the PainterProjecter algorithm for deep fusion and alignment of Chinese painting features and Chinese textual features. The PainterProjecter algorithm includes a Multi-Scale Dilated Attention mechanism, a Feed Forward Network, a HOSC activation function, and a Multi-Head CrossAttention mechanism. First, the PainterProjecter algorithm uses the Multi-Scale Dilated Attention mechanism to extract Chinese painting features from the output of ConvNext v2 across connections, and the same applies to Chinese text features. Then, it uses an FFN feedforward network and a HOSC activation function to perform a non-linear transformation on the features output by the Multi-Scale Dilated Attention mechanism. Next, it uses the Multi-Head CrossAttention mechanism to align and associate Chinese painting features and Chinese text features. Finally, these aligned Chinese painting features and text features are further fused using a Feed Forward Network and a HOSC activation function, and the final output is fed into the LLM large language model. In Multi-Scale Dilated Attention, different values ​​of r represent different expansion capabilities, in order to better associate features between Chinese painting images and appreciation texts in non-neighboring locations. The formula for the HOSC activation function is as follows: ; c. Improve the LLM model in the LLaVA model: Use the Qwen2.5-7B-Chat model as a large language model that integrates and understands Chinese painting feature input, text input, and prompt word input; d. Add the HalfLoRA module to the Qwen2.5-7B-Chat branch of the LLM model; the HalfLoRA module adds the Q and K matrices in the attention mechanism of the Qwen2.5-7B-Chat model using the HalfLoRA method; the formula is as follows: φ is an adjustable hyperparameter; Step 4: Train the improved LLaVA model: Using the training dataset D of Chinese paintings CTr To train the improved LLaVA model from step three, first ensure that the number of training iterations, EPOCH, is less than H. e H e The size is a hyperparameter; secondly, the training dataset D... CTr For mini-batch partitioning, the batch size must be less than or equal to B. ca The value of miniBatch is less than or equal to M. cm B ca and M cm These are all hyperparameters; Step 5: Model Validation The trained and improved LLaVA model was tested on the Chinese painting test dataset. CCs ; Step Six: Model Deployment The improved LLaVA model is deployed to the Ubuntu 22.04 system, where it runs on the server; the client accesses and interacts with the model. Step 7: System Deployment; The system takes photos of students' Chinese paintings using a camera and saves them. The saved images are then transmitted to a server via HTTP. The improved LLaVA model on the server provides an electronic evaluation of the Chinese paintings.

Images