Efficient student Chinese painting appreciation system and appreciation method based on LLaVA model

Abstract

The invention relates to an efficient student Chinese painting appreciation system and appreciation method based on an LLaVA model. The system comprises a client, a server and an improved LLaVA model; the client is used for collecting traditional Chinese painting electronic images and text appreciation and evaluation pairs; the client is also used for splitting the collected total data set DCZO, the server is provided with an improved LLaVA model deployment port, and the deployment port and the client carry out transmission through an http protocol; the improved LLaVA model has a ConvNext v2 model, a PainterProjecter algorithm and a large language model LLM, the ConvNext v2 model is used for extracting artistic features and artistic conception features in a Chinese painting electronic image, and the PainterProjecter algorithm is used for fusing and aligning Chinese painting features and Chinese appreciation character features; and the large language model of the Qwen2.5-7B-Chat combined HalfLoRA is used for outputting appreciation and evaluation characters and transmitting the output appreciation and evaluation characters to the client. Intelligent appreciation of Chinese paintings of students is achieved, and great help is provided for the students to improve creative ability through the intelligent education system.

Description

Technical Field

[0001] The invention belongs to the field of intelligent education technology, and specifically relates to an efficient student Chinese painting appreciation system and appreciation method based on an LLaVA model. Background Art

[0002] Chinese traditional culture contains endless wisdom and charm. With the development of the times, it is increasingly respected and loved by the world. As a treasure of traditional culture, Chinese painting carries a heavy historical accumulation and profound cultural spirit. It is not only a display of visual art, but also a medium for conveying emotions and philosophical thoughts. Paintings are made of brush and ink, expressing emotions in scenery and aspirations in objects. Since ancient times, countless literati and scholars have expressed their temperament, recorded the times, and expressed their personal ideals and pursuits through Chinese paintings, forming a unique artistic tradition of "using paintings to express aspirations and emotions".

[0003] However, on the road of learning and creating Chinese paintings, students and enthusiasts often encounter many obstacles, especially in appreciating the masterpieces of masters and evaluating their own works. The appreciation of Chinese paintings requires not only an understanding of the techniques, but also a deep understanding of the cultural connotations behind them and the emotional expression of the artist, which is a big problem for beginners. Although each creation is a process of gradual improvement of individual skills and flashes of inspiration, the lack of high-level guidance often makes this process slow or even stagnant.

[0004] Due to the scarcity of professional guidance resources, many students and enthusiasts find it difficult to obtain direct feedback and guidance, which undoubtedly limits the pace of their artistic exploration and the release of their creativity. How to improve oneself without frequent direct guidance has become a major challenge facing many Chinese painting learners.

[0005] Therefore, developing a Chinese painting appreciation system that can assist learning, provide objective evaluation and in-depth analysis is particularly important for promoting the inheritance and development of Chinese painting art. Summary of the invention

[0006] The purpose of the present invention is to overcome the shortcomings of the prior art and provide an efficient student Chinese painting appreciation system and appreciation method based on the LLaVA model, which solves the technical problems that students and enthusiasts find it difficult to obtain direct feedback and guidance due to the scarcity of professional guidance resources for Chinese painting.

[0007] In order to solve the above problems, the technical solution of the present invention is: an efficient student Chinese painting appreciation system based on LLaVA model, including a client, a server and an improved LLaVA model;

[0008] The client is used to collect electronic images of traditional Chinese paintings and textual appreciation and analysis pairs, and to organize the electronic images of traditional Chinese paintings and textual appreciation and analysis pairs into a one-to-one corresponding question-answer pair format. The organized electronic images of traditional Chinese paintings and textual appreciation and analysis texts are used as the total data set D CZO The client is also used to collect the total data set D CZO After splitting, one part of the data set is used for training and the other part is used for testing. CTr represents the training data set, D CCs Denotes the test data set, training data set D CTr Used to train the improved LLaVA model; test data set D CCs Used to test the trained improved LLaVA model

[0009] The server has an improved LLaVA model deployment port, and the deployment port is transmitted with the client via the http protocol;

[0010] The improved LLaVA model has the ConvNext v2 model, the PainterProjecter algorithm and the large language model LLM. The ConvNextv2 model is used to extract the meticulous features and artistic conception features in the electronic images of Chinese paintings. The PainterProjecter algorithm is used to deeply fuse and align the features of Chinese paintings and the features of Chinese appreciation texts. The large language model LLM includes the Qwen2.5-7B-Chat model and the HalfLoRA module. The large language model of Qwen2.5-7B-Chat combined with HalfLoRA is used to output appreciation and analysis texts, and to transmit the output appreciation and analysis texts to the client for students to view and copy.

[0011] Optionally, the ConvNext v2 model is divided into four stages. Each stage contains convolution kernels of different sizes to focus on local features. The ConvNext v2 model also has a GRN layer to extract features from the global graph information of Chinese paintings. Moreover, the Chinese paintings in different stages have all been downsampled in the previous stage, and different stages contain Chinese paintings of different sizes.

[0012] Optionally, the PainterProjecter algorithm includes a multi-scale dilated attention mechanism, a feed-forward neural network, a HOSC activation function, and a multi-head criss-cross attention mechanism;

[0013] The multi-scale dilated attention mechanism is used to cross-connect the features output from ConvNextv2 to re-extract the features of Chinese paintings, and the same is true for the features of Chinese text;

[0014] The FFN feedforward neural network and HOSC activation function are used to perform nonlinear transformation on the features output by the multi-scale dilation attention mechanism;

[0015] The multi-head cross attention mechanism is used to align and associate Chinese painting image features with Chinese text features;

[0016] The FFN feedforward neural network and HOSC activation function are also used to further fuse the aligned Chinese painting image features and text features, and output them to the LLM large language model.

[0017] Optionally, there are different r in the multi-scale expansion attention mechanism, and the difference of r indicates the difference of expansion ability, so as to better associate the features of the Chinese painting images and the appreciation texts at non-neighboring positions;

[0018] Attn msda =Attention(Q hj ,K r ,V r )

[0019]

[0020] The HOSC activation function formula is HOSC(x,γ)=tanh(γsinx).

[0021] Optionally, the Qwen2.5-7B-Chat model is used as a large language model that integrates and understands Chinese painting image feature input, text input, and prompt word input; the HalfLoRA module is located in the Qwen2.5-7B-Chat branch of the LLM model, and the HalfLoRA module is used to add the Q, K matrices in the attention mechanism in the Qwen2.5-7B-Chat model by the HalfLoRA method, and the formula is as follows:

[0022] z=Φ Qwen2.5(x) +B1Ax+B2(r1×r1+r2×r2+r3×r3)Ax

[0023]

[0024]

[0025] in, It is a hyperparameter that can be adjusted by yourself.

[0026] Optionally, the training dataset D CTr When used to train the improved LLaVA model, the number of training iterations EPOCH is less than H e , H e The size of the training dataset D is a hyperparameter. CTrWhen performing small batch division, the value of Batch is less than or equal to B ca ; The value of miniBatch is less than or equal to M cm , where B ca and M cm They are all hyperparameters, and hyperparameters have training errors.

[0027] Optionally, H e The value of B is 6. ca The value is 64, M cm The value of is 16, the hyperparameter has training error, MSE is used, and the initial value of the learning rate ζ is set to 0.00001.

[0028] Optionally,

[0029] D CZO =D CTr +D CCs

[0030] D CTr =0.7*D CZO

[0031] D CCs =0.3*D CZO .

[0032] Optionally, a camera is also included, and the camera is used to collect electronic images of Chinese paintings and transmit them to the client.

[0033] Optionally, the improved LLaVA model is deployed in the Ubuntu 22.04 system; its hardware includes two RTX4090 graphics card machines, and the client accesses and interacts with the model by accessing the server IP and fixed port 8501.

[0034] Another object of the present invention is to provide a method for constructing an efficient student Chinese painting appreciation system based on the LLaVA model, comprising the following steps:

[0035] Step 1: Collect and organize the pairs of traditional Chinese paintings and text appreciation and analysis, and organize the pairs of traditional Chinese paintings and text appreciation and analysis into a one-to-one corresponding question-answer pair format. The collected and organized data is used as the total data set D CZO ;

[0036] Step 2: Chinese painting data set division:

[0037] The total data set D in step 1 CZO After splitting, one part of the data set is used for training and the other part is used for testing; CTr represents the training data set, D CCs represents the test data set;

[0038] Step 3: Improve the LLaVA model, including:

[0039] a. Improve the Vision Encoder in the LLaVA model to the ConvNextv2 model, which is used to extract the fine brushwork features and artistic conception features in Chinese electronic images; the ConvNextv2 model is divided into four stages; each stage contains convolution kernels of different sizes to focus on local features, and the ConvNext v2 model also has a GRN layer to extract features from the global image information of Chinese paintings; and the Chinese paintings in different stages have all been downsampled in the previous stage, and different stages contain Chinese paintings of different sizes;

[0040] b. The LLaVA model has a PainterProjecter algorithm to deeply fuse and align Chinese painting features and Chinese appreciation text features; the PainterProjecter algorithm includes Multi-Scale Dilated Attention, Feed Forward Network, HOSC activation function and Multi Head CrossAttention; the PainterProjecter algorithm first uses the Multi-Scale Dilated Attention mechanism to cross-connect the features output in ConvNext v2 to extract Chinese painting features, and the same is true for Chinese text features; then the FFN feedforward neural network and HOSC activation function are used to perform nonlinear transformation on the features output by the Multi-Scale Dilated Attention mechanism; then the Multi Head Cross Attention mechanism is used to align and associate Chinese painting image features and Chinese text features; finally, the aligned Chinese painting image features and text features are further fused through the FeedForward Networ feedforward neural network and HOSC activation function, and finally output to the LLM large language model;

[0041] There are different r in Multi-Scale Dilated Attention, and the different r represent different expansion capabilities, so as to better associate the features of Chinese painting images and appreciation texts in non-adjacent positions;

[0042] Attn msda =Attention(Q hj ,K r ,V r )

[0043]

[0044] The HOSC activation function formula is HOSC(x,γ)=tanh(γsinx);

[0045] c. Improve the LLM model in the LLaVA model: Use the Qwen2.5-7B-Chat model as a large language model that integrates and understands Chinese painting image feature input, text input, and prompt word input;

[0046] d. Add the HalfLoRA module to the Qwen2.5-7B-Chat branch of the LLM model; the HalfLoRA module adds the Q, K matrices in the attention mechanism of the Qwen2.5-7B-Chat model to the HalfLoRA method; the formula is as follows:

[0047] z=Φ Qwen2.5(x) +B1Ax+B2(r1×r1+r2×r2+r3×r3)Ax

[0048]

[0049]

[0050] It is a hyperparameter that can be adjusted;

[0051] Step 4: Train the improved LLaVA model:

[0052] Using the Chinese painting training dataset D CTr Train the improved LLaVA model in step 3. First, ensure that the number of training iterations EPOCH is less than H e , H e The size of is a hyperparameter; secondly, the training data set D CTr Divide into small batches, the value of Batch should be less than or equal to B ca ; The value of miniBatch is less than or equal to M cm , where B ca and M cm They are all hyperparameters; hyperparameters also have training errors;

[0053] Step 5: Model verification:

[0054] The trained improved LLaVA model is tested on the Chinese painting test dataset. CCs ;

[0055] Step 6: Model deployment:

[0056] Deploy the improved LLaVA model to the Ubuntu 22.04 system and run the model on the server; the client accesses and interacts with the model;

[0057] Step 7: System deployment;

[0058] The Chinese paintings created by students are photographed and saved through a camera, and the saved pictures are transmitted to the server through the http protocol. The improved LLaVA model in the server will give an electronic version of the appreciation and evaluation of the Chinese paintings.

[0059] Compared with the prior art, the present invention has the following beneficial effects:

[0060] 1. The efficient student Chinese painting appreciation system based on the LLaVA model provided by the present invention uses an artificial intelligence multimodal model as a basis, combines high-definition camera equipment and local area network servers and other equipment, and collects student Chinese paintings as electronic images. The Chinese painting appreciation system designed by the present invention can accurately appreciate and analyze the Chinese paintings created by students, and help students improve their own creative level. This can be used as an intelligent system for students to appreciate Chinese paintings. The efficient student Chinese painting appreciation system of the present invention can provide students with very convenient appreciation and evaluation, and it has very great practical application significance.

[0061] 2. The efficient student Chinese painting appreciation system based on the LLaVA model provided by the present invention improves the Vison Encoder in the LLaVA model into ConvNext v2. ConvNext v2 can extract the meticulous features of Chinese painting and the artistic conception structure of the overall structure of Chinese painting at multiple scales.

[0062] 3. The efficient student Chinese painting appreciation system based on the LLaVA model provided by the present invention designs the PainterProjecter multimodal alignment network. It aligns and fuses the multi-scale image features containing overall artistic conception features and subtle brushwork features extracted from ConvNext v2 with the corresponding appreciation and tasting text features. It can ensure that the subtle brushwork features are associated with the brushwork description text features, and at the same time, associate and fuse the artistic conception image features with the text description artistic conception features.

[0063] 4. The large language model in the improved LLaVA model of the present invention combines Qwen2.5-7B-Chat with the HalfLoRA designed in this paper as an overall large language model, which can not only ensure the accurate recognition of Chinese text, but also ensure the embedding of vertical field knowledge of Chinese painting, and ensure the accurate output of the system in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0064] Figure 1 is a schematic diagram of the appreciation system of the present invention;

[0065] Figure 2 is a flow chart of an appreciation method of an appreciation system of the present invention;

[0066] Figure 3 Schematic diagram of the PainterProjecter algorithm of the present invention. DETAILED DESCRIPTION

[0067] The present invention is further described in detail below in conjunction with the accompanying drawings and embodiments.

[0068] Example 1: Figure 1-Figure 3 As shown, this embodiment provides an efficient student Chinese painting appreciation system based on the LLaVA model, including a client, a server and an improved LLaVA model;

[0069] The client is used to collect and organize traditional Chinese paintings and textual appreciation and analysis pairs, and to match the organized texts and images one by one. The organized electronic images of Chinese paintings and appreciation and analysis texts are used as the total data set D CZO Specifically, we first used Hikvision high-definition cameras to collect Chinese paintings. Some paintings can be directly collected from the Internet. These Chinese paintings include Chinese landscape paintings and Chinese flower-and-bird paintings, including paintings by some ancient celebrities in the Song, Ming and Qing dynasties, such as Su Shi, Tang Bohu, and the Eight Eccentrics of Jiangnan. The electronic images of Chinese paintings also include paintings by some contemporary celebrities such as Qi Baishi, Zhang Daqian, and Xu Beihong. The electronic images of Chinese paintings also include paintings by some students and teachers in schools, a total of 360 paintings.

[0070] The client is also used to split the Chinese painting image and text pairs, that is, to collect the total data set D CZO Split. Total data set D CZO It is split into a Chinese painting training data set and a test data set; D CTr represents the training data set, D CCs represents the test data set;

[0071] D CZO =D CTr +D CCs

[0072] D CTr =0.7*D CZO

[0073] D CCs =0.3*D CZO ;

[0074] Chinese painting training dataset D CTr Used to train the improved LLaVA model. First, ensure that the number of training iterations EPOCH is less than H e , H e The size of is a hyperparameter. In this example, H eThe selected one is 6. Secondly, the training data set D CTr Divide into small batches, the value of Batch should be less than or equal to B ca In order to better ensure the training speed and ensure that the small batch gradient will not oscillate and the error is too large, this embodiment also adds miniBatch, and the value of MiniBatch must be less than or equal to M cm , where B ca and M cm All of them are hyperparameters. In this embodiment, B ca The value is 64, M cm The value of is 16. At the same time, the hyperparameter also has a training error. In this embodiment, MSE is used, and the initial value of the learning rate ζ is set to 0.00001.

[0075] The ViT model used in the original LLaVA model only cuts the image into 9 equal parts when extracting image features, and then extracts these image features in turn. However, the fine brushwork in the Chinese painting image in this embodiment is mainly reflected in the subtle details of the ink, which are very small details, and the requirements of this embodiment cannot be met by simple cutting alone. Secondly, the design and arrangement of structures such as artistic conception and blank space in Chinese paintings need to focus on the whole. In addition, the large language model of the LLaVA model uses the llama model. However, the llama model is mostly trained for native English speakers, but the Chinese painting appreciation and analysis in this embodiment is simplified Chinese, which has very high requirements for Chinese language understanding, and the llama model is not enough for Chinese training. Secondly, there is still no way to appreciate Chinese paintings using only the Chinese large model, because the current large models all have general capabilities. However, this still lacks basic knowledge and understanding of proprietary vocabulary in the field of Chinese painting in the vertical field. Finally, the alignment ability between the Chinese painting features and the text appreciation text directly determines the mutual understanding between the text and the image. Therefore, this embodiment needs to further improve the alignment ability. Therefore, this embodiment improves the LLaVA model.

[0076] The improved LLaVA model has the ConvNext v2 model, PainterProjecter algorithm, Qwen2.5-7B-Chat model and HalfLoRA module.

[0077] Specifically, the ViT model in the LLaVA model has natural defects in extracting the meticulousness and artistic conception of Chinese paintings. In this embodiment, the Vision Encoder in the LLaVA model is improved to the ConvNext v2 model. The ConvNext v2 model is divided into four stages. Each stage not only contains convolution kernels of different sizes to focus on local features, but also has a GRN layer to extract features from the global graph information of Chinese paintings. Moreover, the Chinese paintings in different stages have all been downsampled in the previous stage. Therefore, the Chinese paintings in different stages are of different sizes. All of these provide a good foundation for extracting the meticulousness information features of Chinese paintings.

[0078] In view of the limited ability of the LLaVA model to align Chinese characters with Chinese paintings, this embodiment designs the PainterProjecter algorithm to deeply fuse and align Chinese painting features and Chinese appreciation text features. The PainterProjecter algorithm includes Multi-Scale Dilated Attention, Feed Forward Network, HOSC activation function and Multi Head Cross Attention. The PainterProjecter algorithm of this embodiment first cross-connects the features output in ConvNextv2 through the multi-scale dilation attention mechanism to extract Chinese painting features, and the same is true for Chinese text features; then uses the FFN feedforward neural network and HOSC activation function to perform nonlinear transformation on the features output by the multi-scale dilation attention mechanism; then uses the Multi Head Cross Attention mechanism to align and associate Chinese painting image features and Chinese text features; finally, the aligned Chinese painting image features and text features are further fused through the FeedForward Networ feedforward neural network and HOSC activation function, and finally output to LLM.

[0079] The Multi-Scale Dilated Attention mechanism has different r, and different r represent different dilation capabilities, which can better associate the features of Chinese painting images and appreciation texts in non-neighboring positions.

[0080] Attn msda =Attention(Q hj ,K r ,V r )

[0081]

[0082] The above formula is the formula expression of Multi-Scale Dilated Attention. msda It is Multi-Scale Dilated Attention, which is composed of three matrices: Q matrix, K matrix and V matrix, which represent the operation of these three matrices.

[0083] The HOSC activation function formula is HOSC(x,γ)=tanh(γsinx),

[0084] In this embodiment, γ=2.

[0085] In view of the problem that LLM in LLaVA is not friendly to Chinese understanding, this embodiment improves the LLM model in the LLaVA model. This embodiment uses the Qwen2.5-7B-Chat model as a large language model that integrates and understands Chinese painting image feature input, text input, and prompt word input.

[0086] Since the improved LLaVA model still cannot understand the proper nouns and specific concepts in the field of Chinese painting, this embodiment designs a HalfLoRA module, which is added to the Qwen2.5-7B-Chat branch of the LLM model in the improved LLaVA model. This embodiment designs HalfLoRA and Qwen2.5-7B-Chat to work together so that the improved LLaVA model can be more suitable for the vertical field of Chinese painting appreciation and analysis. The design of HalfLoRA adds the Q, K matrices in the attention mechanism in the Qwen2.5-7B-Chat model to the HalfLoRA method, and the formula is as follows:

[0087] z=Φ Qwen2.5(x) +B1Ax+B2(r1×r1+r2×r2+r3×r3)Ax

[0088]

[0089]

[0090] The above is the formula of HalfLoRA. B1, B2, and A represent the embedding matrix. r1, r2, and r3 represent different rank combinations. Dim(B1) represents the dimension of the B1 matrix, and its calculation formula is the original weight Dim(W * ) divided by the split coefficient dim(B2) represents the dimension of the B2 matrix.

[0091] At the same time, the r dimension in B1 is 4, and the r dimension in B2 is 16. is an adjustable hyperparameter. The value of is 2.

[0092] Training dataset D CTr Used to train the improved LLaVA model. The trained model can learn the unique features of Chinese paintings and align and fuse these features with the text appreciation and analysis text. The large language model learns to associate Chinese paintings with text appreciation text, so that the corresponding evaluation of Chinese paintings can be given according to the image features of Chinese paintings. Test Dataset D CCs It is used to test the trained improved LLaVA model. The model after successful testing can be used as the final model used in this embodiment.

[0093] After the test is successful, the improved LLaVA model is deployed to the Ubuntu system. Specifically, you need to first install the RTX driver in the Ubuntu system. Secondly, install the cuda dependency package. Then, install the corresponding dependency package in Ubuntu. Copy the improved model to the Ubuntu system and start the copied model.

[0094] When in use, students and enthusiasts can collect electronic images of Chinese paintings through Hikvision cameras, and these electronic images of Chinese paintings can be uploaded to the server through the client. Specifically, the client uploads the electronic images collected by Hikvision to the server via the http protocol.

[0095] The client and the server transmit data via the http protocol, mainly using port 8501 for transmission. Port 8501 of the server is exactly the port where the improved LLaVA model in this embodiment is deployed. After the LLaVA model receives the electronic image of the Chinese painting via port 8501, it can extract the features of the model.

[0096] The improved LLaVA model is used to extract features and output text information. Specifically, the LLaVA model extracts the meticulous features and artistic conception features in Chinese electronic images through the ConvNextv2 model, then associates the image features with the text features through painterProjecter, and then outputs the appreciation and analysis text through the large language model Qwen2.5-7B-Chat combined with HalfLoRA.

[0097] Qwen2.5-7B-Chat combined with HalfLoRA's large language model transmits the output appreciation text to the client through the http protocol. After receiving the text transmitted from the http protocol, the client outputs the text to the computer screen. Students can improve their creative ability by appreciating and analyzing the Chinese paintings they created based on the output results.

[0098] In this embodiment, an efficient student Chinese painting appreciation system based on the LLaVA model is deployed in the improved LLaVA model to the Ubuntu 22.04 system. Its hardware includes two RTX4090 graphics card machines, and the improved LLaVA model runs in the server. The client accesses and interacts with the model by accessing the server IP and fixed port 8501.

[0099] The workflow of an efficient student Chinese painting appreciation system based on the LLaVA model in this embodiment is as follows:

[0100] 1. Hikvision high-definition cameras collect Chinese paintings. Chinese paintings include Chinese landscape paintings and Chinese flower-and-bird paintings, including some famous scenic spots in ancient Song, Ming and Qing dynasties. Chinese painting electronic images also include some contemporary celebrity paintings. Chinese painting electronic images also include some paintings of students and school teachers.

[0101] 2. Arrangement of Chinese painting images and texts. Pair the collected Chinese paintings with the corresponding text appreciation.

[0102] 3. Split the total dataset. The total dataset is split into a Chinese painting training dataset and a test dataset.

[0103] Fourth, first ensure that the number of training iterations EPOCH is less than H e , H e The selected one is 6; secondly, the training data set D CTr Divide into small batches, the value of Batch should be less than or equal to B ca ; In order to better ensure the training speed and ensure that the small batch gradient does not oscillate and the error is too large, the value of this MiniBatch should be less than or equal to M cm , where B ca and M cm These are all hyperparameters. This example uses B ca The value of M is 64. cm The value of is 16. At the same time, the hyperparameters also have training errors, and the initial value of the learning rate ζ is set to 0.00001.

[0104] 5. Improved LLaVA model.

[0105] 6. Training the improved LLaVA model In this way, the improved LLaVA model can be tuned to the vertical field of Chinese painting in this embodiment.

[0106] 7. Test set validation model: The trained improved LLaVA model is tested on the test data set and the optimal model is saved.

[0107] 8. Model deployment: Save the optimal model as the baseline model, copy it to the Ubuntu system and install its dependent packages.

[0108] 9. System deployment. Connect the client to the server through a network cable and enable port 8051 and ssh service on the server, connect the Hikvision camera to the client computer, run the model and set it as a system service.

[0109] 10. Students and enthusiasts can collect electronic images of Chinese paintings through Hikvision cameras, and these electronic images of Chinese paintings can be uploaded to the server through the client.

[0110] 11. The client uploads the electronic images collected by Hikvision to the server via the HTTP protocol.

[0111] 12. Upload the data to the server through http protocol.

[0112] 13. The improved LLaVA model extracts features and outputs text information. The LLaVA model extracts features from the received electronic images of Chinese paintings. The LLaVA model extracts the fine brushwork features and artistic conception features in the Chinese electronic images through the ConvNext v2 model, then associates the image features with the text features through painterProjecter, and then outputs the appreciation and analysis text through the large language model Qwen2.5-7B-Chat combined with HalfLoRA.

[0113] 14. HTTP transmission. Qwen2.5-7B-Chat combined with HalfLoRA's large language model transmits the output appreciation and tasting text to the client through the http protocol.

[0114] 15. Client reception: The client receives the streaming text transmitted from the http protocol and converts it into the utf8 file format.

[0115] 16. Output the results. Output the converted text to the computer screen for students to view and copy.

[0116] In the face of the vigorous development of intelligent education, this embodiment uses the improved LLaVA model to carefully create an efficient appreciation system that aims to accurately appreciate and analyze Chinese paintings created by students and can effectively identify their shortcomings. In today's society, the wave of revival of traditional culture is unstoppable. As one of the best, Chinese painting has become a treasure in the global art field and is highly respected. The art of Chinese painting is profound and profound. Under the subdivision, whether it is a magnificent landscape painting or a vibrant flower and bird painting, each category contains unique meticulous techniques and profound artistic conception, requiring painters to seek a perfect balance between delicate portrayal and overall composition and freehand expression. And students' daily exercises often require extremely professional scrutiny as a basis for modification and appreciation. In reality, they will be affected by the differences in teachers' personal artistic accomplishments and subjective preferences, and there may be certain deviations. Different people may have very different evaluations of the same work, which brings uncertainty to students' learning feedback. It is these challenges that have given rise to an urgent need for a popular and efficient Chinese painting appreciation system, which should be able to transcend the limitations of traditional teaching and promote the improvement of students' painting skills and the deepening of their artistic understanding with objective and comprehensive standards. In response to this problem, this embodiment uses an artificial intelligence multimodal model as a basis, combined with high-definition camera equipment and LAN servers and other equipment to develop a set of efficient Chinese painting appreciation systems for students. Specifically, first use a Hikvision high-definition camera to electronically shoot the paper Chinese paintings created by students, and store the ultra-high-definition Chinese paintings shot by the client. The client transmits the Chinese paintings to the server where the improved LLaVA model of this embodiment is deployed through the http protocol. The improved LLaVA model will extract features and semantic understanding of the transmitted Chinese paintings, and finally generate an appreciation analysis of the Chinese paintings. In order to better combine the unique feature information in the field of Chinese painting. In this embodiment, the VisionEncoder model in the LLaVA model is improved to be more capable of extracting the ConvNextv2 algorithm of the detailed features and overall structure of Chinese paintings. Secondly, in order to better align Chinese paintings with longer appreciation texts, the present invention designs a PainterProjecter model. In addition, in order to ensure the quality of the generation of appreciation and analysis, this paper uses the Qwen2.5-7B-Chat model and improves it into a model structure that integrates HalfLoRA. In summary, the efficient student Chinese painting appreciation system based on the LLaVA model in this embodiment can meet the requirements of students very well. It has great theoretical significance and educational and practical application significance, and it has a very wide range of application scenarios.

[0117] Embodiment 2: This embodiment provides an appreciation method of an efficient student Chinese painting appreciation system based on the LLaVA model, comprising the following steps:

[0118] Step 1: Cameras collect electronic images of Chinese paintings. First, use Hikvision high-definition cameras to collect Chinese paintings. Some paintings can be collected directly from the Internet. These Chinese paintings include Chinese landscape paintings and Chinese flower and bird paintings, including paintings by some ancient celebrities in the Song, Ming and Qing dynasties such as Su Shi, Tang Bohu, and the Eight Eccentrics of Jiangnan. Electronic images of Chinese paintings also include paintings by some contemporary celebrities such as Qi Baishi, Zhang Daqian, and Xu Beihong. Electronic images of Chinese paintings also include paintings by some students and school teachers, a total of 360 paintings.

[0119] Step 2: The client organizes the Chinese painting images and text pairs. The appreciation and analysis of the electronic images of Chinese paintings collected in step 1 are collected and organized, and the organized texts and images are matched one by one. The organized electronic images of Chinese paintings and the appreciation and analysis texts are used as the total data set.

[0120] Step 3: Split Chinese painting images and text pairs. CZO The total dataset is split into the Chinese painting training dataset and the testing dataset. CTr represents the training data set, D CCs It represents the test data set.

[0121] D CZO =D CTr +D CCs

[0122] D CTr =0.7*D CZO

[0123] D CCs =0.3*D CZO ;

[0124] Step 4: Hyperparameter selection. Training dataset D of Chinese paintings CTr Train the improved model. First, ensure that the number of training iterations EPOCH is less than H e , H e The size of is a hyperparameter, and this embodiment selects 6. CTr Divide into small batches, the value of Batch should be less than or equal to B ca In order to better ensure the training speed and ensure that the small batch gradient does not oscillate and the error is too large. This embodiment also adds miniBatch, and the value of MiniBatch must be less than or equal to M cm , where B ca and M cm These are all hyperparameters. This example uses B ca The value of M is 64.cm The value of is 16. At the same time, the hyperparameters also include training error. The MSE used in this embodiment and the initial value of the learning rate ζ are set to 0.00001.

[0125] Step 5: Improve the LLaVA model. The ViT model used in the original LLaVA model only cuts the image into 9 equal parts when extracting image features, and then extracts these image features in turn. However, the fine brushwork in the Chinese painting image in this embodiment is mainly reflected in the subtle details of the ink, which are very small details, and the requirements of this patent cannot be met by simple cutting alone. Secondly, the design and arrangement of structures such as artistic conception and blank space in Chinese paintings need to focus on the whole. In addition, the large language model of the LLaVA model uses the llama model. However, the llama model is mostly trained for native English speakers, but the Chinese painting appreciation and analysis in this article is simplified Chinese, which has very high requirements for Chinese language understanding, and the llama model is not trained enough in Chinese. Secondly, there is still no way to appreciate Chinese paintings using only the Chinese large model. This is because the current large models all have general capabilities, but this still lacks basic knowledge and understanding of proprietary vocabulary in the field of Chinese painting in the vertical field. Finally, the alignment ability between the Chinese painting features and the text appreciation text directly determines the mutual understanding between the text and the image. Therefore, this embodiment needs to further improve the alignment capability. Therefore, this embodiment improves the LLaVA model.

[0126] The ViT model in the LLaVA model has natural defects in extracting the meticulousness and artistic conception of Chinese paintings. In this embodiment, the Vision Encoder in the LLaVA model is improved to the ConvNextv2 model. It is divided into four stages. Each stage not only contains convolution kernels of different sizes to focus on local features, but also has a GRN layer to extract features from the global image information of Chinese paintings. Moreover, the Chinese paintings in different stages have all been downsampled in the previous stage. Therefore, the Chinese paintings in different stages are of different sizes, which provides a good foundation for extracting the meticulousness information features of Chinese paintings.

[0127] In view of the limited ability of the LLaVA model to align Chinese characters with Chinese paintings, this embodiment designs the PainterProjecter algorithm to deeply fuse and align Chinese painting features and Chinese appreciation text features. The PainterProjecter algorithm includes a Multi-Scale Dilated Attention multi-scale dilation attention mechanism, a Feed Forward Network feedforward neural network, a HOSC activation function, and a Multi Head Cross Attention multi-head cross attention mechanism. The PainterProjecter algorithm designed in this embodiment first cross-connects the features output in ConvNext v2 through a multi-scale dilation attention mechanism to extract Chinese painting features, and the same is true for Chinese text features. Then, FFN and HOSC are used to perform nonlinear transformation on the features output by the multi-scale dilation attention mechanism, and then the Chinese painting image features and Chinese text features are aligned and associated through the MultiHead Cross Attention multi-head cross attention mechanism. Finally, after these features are aligned, the above-mentioned Chinese painting image features and text features are further fused through the Feed Forward Network feedforward neural network and the HOSC activation function, and finally output to LLM.

[0128] There are different r in Multi-Scale Dilated Attention, and different r represent different expansion capabilities, which can better associate the features of Chinese painting images and appreciation texts in non-neighboring positions.

[0129] Attn msda =Attention(Q hj ,K r ,V r )

[0130]

[0131] The above formula is the formula expression of Multi-Scale Dilated Attention. msda It is Multi-Scale Dilated Attention, which is composed of three matrices: Q matrix, K matrix and V matrix, which represent the operation of these three matrices.

[0132] The HOSC activation function formula is HOSC(x,γ)=tanh(γsinx), and in the present invention, γ=2.

[0133] In view of the problem that LLM in LLaVA is not friendly to Chinese understanding, this embodiment improves the LLM model in the LLaVA model. This embodiment uses the Qwen2.5-7B-Chat model as a large language model that integrates and understands Chinese painting image feature input, text input, and prompt word input.

[0134] Since the improved LLaVA model still cannot understand the proper nouns and specific concepts in the field of Chinese painting, this embodiment designs a HalfLoRA module, which is added to the Qwen2.5-7B-Chat branch of the LLM model in the improved LLaVA model. This embodiment designs the HalfLoRA to work together with Qwen2.5-7B-Chat so that the improved LLaVA model can be more suitable for the vertical field of Chinese painting appreciation and analysis. The design of HalfLoRA adds the Q, K matrices in the attention mechanism in the Qwen2.5-7B-Chat model to the HalfLoRA method. The formula is as follows:

[0135] z=Φ Qwen2.5(x) +B1Ax+B2(r1×r1+r2×r2+r3×r3)Ax

[0136]

[0137]

[0138] The above is the formula of HalfLoRA. B1, B2, and A represent the embedding matrix. r1, r2, and r3 represent different rank combinations. Dim(B1) represents the dimension of the B1 matrix, and its calculation formula is the original weight Dim(W * ) divided by the split coefficient dim(B2) represents the dimension of the B2 matrix.

[0139] At the same time, the r dimension in B1 is 4, and the r dimension in B2 is 16. is an adjustable hyperparameter. The value of is 2.

[0140] Step 6: Train the model. The improved LLaVA model in step 5 is trained using the hyperparameters in step 4. The trained model can learn the unique features of Chinese paintings and align and fuse these features with the text of appreciation and analysis. The large language model learns to associate Chinese paintings with text appreciation text, and thus can give corresponding evaluations of Chinese paintings based on their image features.

[0141] Step 7: Dcc test: The model trained in step 6 is tested on a test data set, and the model that succeeds the test can be used as the final model used in this embodiment.

[0142] Step 8: Model and system deployment. To deploy the model in step 7 to the Ubuntu system, you first need to install the RTX driver in the Ubuntu system; secondly, install the cuda dependency package; then, install the corresponding dependency package in Ubuntu; copy the improved model to the Ubuntu system and start the copied model.

[0143] Step 9: Student Chinese painting collection. Students and enthusiasts can collect electronic images of Chinese paintings through Hikvision cameras, and these electronic images of Chinese paintings can be uploaded to the server through the client.

[0144] Step 10: The client uploads the electronic image of the Chinese painting. The client uploads the electronic image collected by Hikvision to the server via the http protocol.

[0145] Step 11: http protocol upload and server reception. The client and server transmit via http protocol, mainly using port 8501 for transmission. Port 8501 of the server is exactly the port where the improved LLaVA model of this embodiment is deployed. After the LLaVA model receives the electronic image of the Chinese painting via port 8501, the model feature extraction can be performed.

[0146] Step 12: The improved LLaVA model extracts features and outputs text information. The LLaVA model extracts features from the received electronic images of Chinese paintings. The LLaVA model extracts the fine brushwork features and artistic conception features from the Chinese electronic images through the ConvNextv2 model, then associates the image features with the text features through painterProjecter, and then outputs the appreciation and analysis text through the large language model Qwen2.5-7B-Chat combined with HalfLoRA.

[0147] Step 13: http transmission. Qwen2.5-7B-Chat combined with HalfLoRA's large language model transmits the output appreciation and tasting text to the client via the http protocol.

[0148] Step 14: Client receiving. After receiving the text transmitted by the http protocol, the client outputs the text to the computer screen. Students can appreciate and analyze the Chinese paintings they created based on the output results to improve their own creative ability.

Claims

1. An efficient student Chinese painting appreciation system based on LLaVA model, characterized by: Includes client, server and improved LLaVA model; The client is used to collect electronic images of traditional Chinese paintings and textual appreciation and analysis pairs, and to organize the electronic images of traditional Chinese paintings and textual appreciation and analysis pairs into a one-to-one corresponding question-answer pair format. The organized electronic images of traditional Chinese paintings and textual appreciation and analysis texts are used as the total data set D CZO The client is also used to collect the total data set D CZO After splitting, one part of the data set is used for training and the other part is used for testing. CTr represents the training data set, D CCs Denotes the test data set, training data set D CTr Used to train the improved LLaVA model; test data set D CCs Used to test the trained improved LLaVA model The server has an improved LLaVA model deployment port, and the deployment port is transmitted with the client via the http protocol; The improved LLaVA model has the ConvNext v2 model, the PainterProjecter algorithm and the large language model LLM. The ConvNextv2 model is used to extract the meticulous features and artistic conception features in the electronic images of Chinese paintings. The PainterProjecter algorithm is used to deeply fuse and align the features of Chinese paintings and the features of Chinese appreciation texts. The large language model LLM includes the Qwen2.5-7B-Chat model and the HalfLoRA module. The large language model of Qwen2.5-7B-Chat combined with HalfLoRA is used to output appreciation and analysis texts, and to transmit the output appreciation and analysis texts to the client for students to view and copy.

2. According to the LLaVA model-based efficient student Chinese painting appreciation system of claim 1, it is characterized in that: The ConvNext v2 model is divided into four stages. Each stage contains convolution kernels of different sizes to focus on local features. The ConvNext v2 model also has a GRN layer to extract features from the global graph information of Chinese paintings. Moreover, the Chinese paintings in different stages have all been downsampled in the previous stage, and the Chinese paintings in different stages are of different sizes.

3. According to the efficient student Chinese painting appreciation system based on LLaVA model as described in claim 1, it is characterized in that: The PainterProjecter algorithm includes a multi-scale dilation attention mechanism, a feedforward neural network, a HOSC activation function, and a multi-head cross attention mechanism; The multi-scale dilated attention mechanism is used to cross-connect the features output from ConvNextv2 to re-extract the features of Chinese paintings, and the same is true for the features of Chinese text; The FFN feedforward neural network and HOSC activation function are used to perform nonlinear transformation on the features output by the multi-scale dilation attention mechanism; The multi-head cross attention mechanism is used to align and associate Chinese painting image features with Chinese text features; The FFN feedforward neural network and HOSC activation function are also used to further fuse the aligned Chinese painting image features and text features, and output them to the LLM large language model.

4. According to the LLaVA model-based efficient student Chinese painting appreciation system of claim 3, it is characterized in that: There are different r in the multi-scale expansion attention mechanism, and the different r represent the different expansion capabilities, so as to better associate the features of Chinese painting images and appreciation texts in non-neighboring positions; Attn msda =Attention(Q hj ,K r ,V r ) The HOSC activation function formula is HOSC(x,γ)=tanh(γsinx).

5. According to the LLaVA model-based efficient student Chinese painting appreciation system of claim 1, it is characterized in that: The Qwen2.5-7B-Chat model is used as a large language model that integrates the input of Chinese painting image features, text input, and prompt word input; the HalfLoRA module is located in the Qwen2.5-7B-Chat branch of the LLM model. The HalfLoRA module is used to add the Q and K matrices in the attention mechanism of the Qwen2.5-7B-Chat model using the HalfLoRA method. The formula is as follows: z=Φ Qwen2.5(x) +B1Ax+B2(r1×r1+r2×r2+r3×r3)Ax in, It is a hyperparameter that can be adjusted by yourself.

6. The efficient student Chinese painting appreciation system based on the LLaVA model according to claim 1 is characterized in that: Training dataset D CTr When used to train the improved LLaVA model, the number of training iterations EPOCH is less than H e , H e The size of the training dataset D is a hyperparameter. CTr When performing small batch division, the value of Batch is less than or equal to B ca ; The value of miniBatch is less than or equal to M cm , where B ca and M cm They are all hyperparameters, and hyperparameters have training errors.

7. According to claim 5, an efficient student Chinese painting appreciation system based on LLaVA model is characterized in that: H e The value of B is 6. ca The value is 64, M cm The value of is 16, the hyperparameter has training error, MSE is used, and the initial value of the learning rate ζ is set to 0.00001.

8. The efficient student Chinese painting appreciation system based on the LLaVA model according to claim 1 is characterized in that: D CZO =D CTr +D CCs D CTr =0.7*D CZO D CCs =0.3*D CZO 。 9. The efficient student Chinese painting appreciation system based on the LLaVA model according to claim 1 is characterized in that: The improved LLaVA model is deployed in the Ubuntu 22.04 system; Its hardware includes two RTX4090 graphics card machines. The client accesses and interacts with the model by accessing the server IP and fixed port 8501.

10. A method for constructing an efficient student Chinese painting appreciation system based on the LLaVA model, characterized in that: The following steps are involved: Step 1: Collect and organize the pairs of traditional Chinese paintings and text appreciation and analysis, and organize the pairs of traditional Chinese paintings and text appreciation and analysis into a one-to-one corresponding question-answer pair format. The collected and organized data is used as the total data set D CZO ; Step 2: Chinese painting data set division: The total data set D in step 1 CZO After splitting, one part of the data set is used for training and the other part is used for testing; CTr represents the training data set, D CCs represents the test data set; Step 3: Improve the LLaVA model, including: a. Improve the Vision Encoder in the LLaVA model to the ConvNextv2 model, which is used to extract the fine brushwork features and artistic conception features in Chinese electronic images; the ConvNextv2 model is divided into four stages; each stage contains convolution kernels of different sizes to focus on local features, and the ConvNext v2 model also has a GRN layer to extract features from the global image information of Chinese paintings; and the Chinese paintings in different stages have all been downsampled in the previous stage, and different stages contain Chinese paintings of different sizes; b. The LLaVA model has a PainterProjecter algorithm to deeply fuse and align Chinese painting features and Chinese appreciation text features; the PainterProjecter algorithm includes Multi-Scale DilatedAttention, Feed ForwardNetwork, HOSC activation function and Multi Head Cross Attention; the PainterProjecter algorithm first uses the multi-scale dilated attention mechanism to cross-connect the features output in ConvNextv2 to extract Chinese painting features, and the same is true for Chinese text features; then the FFN feedforward neural network and HOSC activation function are used to nonlinearly transform the features output by the multi-scale dilated attention mechanism; then the Multi Head Cross Attention mechanism is used to align and associate Chinese painting image features and Chinese text features; finally, the aligned Chinese painting image features and text features are further fused through the FeedForwardNetwor feedforward neural network and HOSC activation function, and finally output to the LLM large language model; There are different r in Multi-Scale Dilated Attention, and the different r represent different expansion capabilities, so as to better associate the features of Chinese painting images and appreciation texts in non-adjacent positions; Attn msda =Attention(Q hj ,K r ,V r ) The HOSC activation function formula is HOSC(x,γ)=tanh(γsinx); c. Improve the LLM model in the LLaVA model: Use the Qwen2.5-7B-Chat model as a large language model that integrates and understands Chinese painting image feature input, text input, and prompt word input; d. Add the HalfLoRA module to the Qwen2.5-7B-Chat branch of the LLM model; the HalfLoRA module adds the Q, K matrices in the attention mechanism of the Qwen2.5-7B-Chat model to the HalfLoRA method; the formula is as follows: z=Φ Qwen2.5(x) +B1Ax+B2(r1×r1+r2×r2+r3×r3)Ax It is a hyperparameter that can be adjusted; Step 4: Train the improved LLaVA model: Using the Chinese painting training dataset D CTr Train the improved LLaVA model in step 3. First, ensure that the number of training iterations EPOCH is less than H e , H e The size of is a hyperparameter; secondly, the training data set D CTr Divide into small batches, the value of Batch should be less than or equal to B ca ; The value of miniBatch is less than or equal to M cm , where B ca and M cm They are all hyperparameters; hyperparameters also have training errors; Step 5: Model verification: The trained improved LLaVA model is tested on the Chinese painting test dataset. CCs ; Step 6: Model deployment: Deploy the improved LLaVA model to the Ubuntu 22.04 system and run the model on the server; the client accesses and interacts with the model; Step 7: System deployment; The Chinese paintings created by students are photographed and saved through a camera, and the saved pictures are transmitted to the server through the http protocol. The improved LLaVA model in the server will give an electronic version of the appreciation and evaluation of the Chinese paintings.

Images