Art painting analysis method based on multi-modal large model

Abstract

The invention discloses an artistic painting analysis method based on a multi-modal large model, which comprises the following steps: firstly, acquiring an image from an artistic painting library, screening out an image with a title and an artist name by using a closed source large language model, and then marking a corresponding overall artistic analysis paragraph only paying attention to visual features, and annotation of art analysis is carried out on professional levels such as composition, color, light and shadow and the like, so that art painting analysis data is formed. And training and finely adjusting a multi-modal large model by utilizing the collected art painting analysis data to obtain an art painting generation type pre-training model for generating art painting analysis. Through experimental research, the art painting analysis capability of the multi-modal large model is remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of art painting analysis, and more specifically, relates to an art painting analysis method based on a multimodal large model. Background Art

[0002] Art painting analysis is an important part of art appreciation and creation, and usually requires a deep understanding of visual composition, artistic techniques, style, and historical context. In the past decade, with the great success of deep learning in computer vision and natural language processing, artificial intelligence for art painting analysis has developed rapidly.

[0003] Early work explored the classification and recognition of artistic paintings, relying mainly on handcrafted features. With the great success of pre-trained language models, visual language models were fine-tuned using image-text pairs, achieving some improvements in style classification, object detection, and multimodal retrieval of artistic paintings. Later, researchers proposed multiple datasets for the task of visual question answering of art, and introduced external knowledge and bias elimination strategies to allow visual language models to learn patterns for question answering of artistic paintings, achieving good performance.

[0004] Although these simple tasks have been studied in the era of deep learning, the models still lack the ability to conduct a comprehensive and in-depth analysis of art paintings due to the limitations of visual perception and language model generation capabilities. It is also expected that AI systems will be able to provide comprehensive analysis of art paintings, which will benefit art education and help humans write reviews.

[0005] In recent years, large language models and multimodal large models have demonstrated excellent understanding and generation capabilities in many fields, including text summarization, open-ended question answering, and contextual reasoning. These models can accurately perceive and understand visual content and generate detailed descriptions, such as visual storytelling and subtitle generation, showing the potential for comprehensive analysis of artistic paintings.

[0006] Large language models and multimodal large models have promoted progress in many research fields. However, although existing multimodal large models have strong natural image understanding and text generation capabilities, they still cannot accurately perform comprehensive and in-depth analysis of art paintings, including color, composition, lines, shapes, light and shadow based on visual understanding. In addition, existing multimodal large models face the problem of language bias caused by visual illusions, and are prone to mistakenly identify the current visual content as other, and then the language model generates language for the misidentified visual content based on the corresponding memory knowledge to obtain incorrect text. Therefore, for the task of art painting analysis, existing models tend to first recognize the given painting and then perform the corresponding analysis, while not paying attention to the visual content of the painting in the generation stage of the analysis text. This process of first recognition and then analysis depends heavily on the accuracy of recognition, and it is easy to fail when the given painting is unknown and does not exist in the model memory. Although the existing ShareGPT4V provides a dataset with high-quality image and text description pairs to fine-tune the open source LLaVA multimodal model to alleviate the language bias problem caused by visual illusions, it still cannot perform professional art analysis on art paintings. Summary of the invention

[0007] The purpose of the present invention is to overcome the deficiencies of the prior art and provide an artistic painting analysis method based on a multimodal large model.

[0008] To achieve the above-mentioned object of the invention, the present invention provides an artistic painting analysis method based on a multimodal large model, characterized by comprising the following steps:

[0009] (1) Data collection for art painting analysis

[0010] 1.1) Get an image from the art painting library and use a closed-source large language model to determine whether the title and artist name of the image, i.e. the art painting, are known at the same time. If so, keep it, otherwise discard it;

[0011] 1.2) Use two closed-source large language models to retrieve the learned knowledge through the title of the art painting and the name of the artist, delete the title of the art painting and the name of the artist, and each generate a paragraph of analysis text that only focuses on visual features;

[0012] 1.3) Extract the overall art analysis paragraph from the two generated analysis texts that only focus on visual features, and analyze and annotate them to obtain the corresponding command text. Extract the five most important professional level analysis paragraphs from the two generated analysis texts that only focus on visual features, select the intersection of the five professional level analysis paragraphs as the selected professional level analysis paragraph, and analyze and annotate them to obtain the command text of the selected professional level. In this way, high-quality art painting analysis data consisting of the image of the art painting, the overall analysis paragraph and the corresponding command text, and the analysis paragraphs at each professional level and the corresponding command text are obtained, wherein the professional level includes composition, light and shadow, color, shape, texture, symbol and icon, perspective, movement and posture, line quality, and scale ratio;

[0013] (2) Train to obtain a generative pre-trained model for art paintings

[0014] 2.1) Construct a multimodal large model consisting of a visual encoder, a multimodal projector, and a large language model. The visual encoder encodes the image of the art painting to obtain visual features. The multimodal projector uses a multi-layer perceptron to project the visual features into the semantic space of the language. The large language model decodes the visual features projected into the semantic space of the language in combination with the command text to obtain an overall analysis paragraph or a professional level analysis paragraph.

[0015] 2.2) Use the collected art painting images and corresponding art painting analysis data to supervisely fine-tune the multimodal large model: freeze the parameters of the visual encoder, fine-tune the multimodal projector and the large language model, set the learning rate of training fine-tuning to 2e-5, set the batch size to 16, and perform 10k steps of fine-tuning to obtain the art painting generative pre-training model;

[0016] (3) Reasoning

[0017] Input an art painting image I, a command text P containing n words for the instruction model to analyze, and output an analysis paragraph Output containing m words through the art painting generative pre-trained model.

[0018] The object of the present invention is achieved in this way.

[0019] Since there is no data set for art painting analysis at present, the present invention is based on a multimodal large model art painting analysis method. First, images are obtained from the art painting library, and a closed-source large language model is used to filter out images with both titles and artist names. Then, the corresponding overall art analysis paragraphs that only focus on visual features are annotated, and art analysis annotations are performed on professional levels such as composition, color, light and shadow, thereby forming art painting analysis data. Using the collected art painting analysis data, a multimodal large model is trained and fine-tuned to obtain an art painting generative pre-training model for the generation of art painting analysis. Through experimental research, the present invention significantly improves the art painting analysis capability of the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 It is a flowchart of a specific implementation method of the art painting analysis method based on a multi-modal large model of the present invention;

[0021] Figure 2 yes Figure 1 A flowchart of a specific implementation method of data collection for art painting analysis;

[0022] Figure 3 It is a diagram of the instruction command examples and the specific instance of the dialogue capability of the present invention. DETAILED DESCRIPTION

[0023] The specific implementation of the present invention is described below in conjunction with the accompanying drawings so that those skilled in the art can better understand the present invention. It should be noted that in the following description, when the detailed description of known functions and designs may dilute the main content of the present invention, these descriptions will be omitted here.

[0024] The present invention focuses on art painting analysis, and innovatively proposes an art painting analysis method based on a multimodal large model to improve the model's art painting perception ability and understand the artistic skills and professional levels displayed by visual elements. Since there is currently no data set for art painting analysis, in this embodiment, the present invention collects approximately 19k painting images, annotates the corresponding overall art analysis paragraphs that only focus on visual features according to the title of the painting and the name of the artist, and annotates specific aspects such as composition, color, light and shadow for art analysis, thereby forming 50k art painting analysis data. Using the collected data set, the present invention trains and fine-tunes a multimodal large model to obtain an art painting generative pre-training model, referred to as GalleryGPT, which can be used for the generation of art painting analysis. Through experimental research, the present invention significantly improves the art painting analysis capability of the multimodal large model.

[0025] Figure 1 It is a flow chart of a specific implementation of the art painting analysis method based on a multimodal large model of the present invention.

[0026] In this embodiment, if Figure 1 As shown, the art painting analysis method based on the multimodal large model of the present invention includes the following steps:

[0027] Step S1: Data collection for art painting analysis

[0028] The present invention aims to alleviate the language bias problem caused by visual illusions by fine-tuning the multimodal large model to pay more attention to the visual elements of the painting. Since there is currently no data set for art painting analysis, the present invention designs a data collection method to obtain high-quality art painting analysis data, thereby fine-tuning the multimodal large model and obtaining the art painting generative pre-training model, namely GalleryGPT.

[0029] However, the analysis of manually annotated art paintings requires the annotator to have expertise in art analysis, which is difficult and expensive, and it takes a lot of time to manually annotate a large number of art paintings. Therefore, considering that the excellent closed-source large language model has sufficient knowledge, the present invention uses it to perform high-quality analysis and annotation of art painting images. The final collected dataset includes art paintings and art painting analysis data for each art painting. The specific data collection process is as follows Figure 2 As shown, the following steps are included:

[0030] Step S1.1: Get art painting images with titles and artist names

[0031] Get an image from the art painting library and use a closed-source large language model to determine whether the title and artist name of the image, i.e. the art painting, are known at the same time. If so, keep it, otherwise discard it.

[0032] With the development of the Internet, a large number of art paintings have been digitized and stored. In this embodiment, the present invention uses the art gallery library Art Gallery as the source of paintings. In order to ensure that the closed-source large language model GPT-4 and Gemini can provide accurate art painting analysis paragraphs, 19295 famous art painting images in the art gallery library Art Gallery are first obtained, and the closed-source large language model Gemini is used to determine whether the titles and artist names of these art paintings are known, thereby filtering out paintings without specific titles and paintings judged as unknown by Gemini, and finally obtaining 18526 painting images. Among them, 13526 are used as training images, and 5000 relatively non-famous paintings are used as test images.

[0033] Step S1.2: Generate two analysis texts focusing only on visual features

[0034] Two closed-source large language models are used to retrieve the learned knowledge through the titles of art paintings and the names of artists, and the titles of art paintings and the names of artists are deleted to generate a paragraph of analysis text that only focuses on visual features.

[0035] In this embodiment, the present invention uses powerful closed-source large language models GPT-4 and Gemini to generate high-quality art analysis paragraphs. For each painting, only the title of the painting and the name of the artist are provided, and no visual information is input. The closed-source large language models GPT-4 and Gemini retrieve the learned knowledge through the title and artist's name of the painting, delete the title and artist's name of the art painting, and each generate a paragraph of analysis text that only focuses on visual features. In order to avoid the language bias problem caused by visual illusions, the closed-source large language models GPT-4 and Gemini are required not to mention the title and artist's name of the art painting in the analysis text, so that the corresponding art painting work cannot be easily identified based on the specific analysis text alone, that is, the language bias problem caused by visual illusions is avoided.

[0036] Step S1.3: Generate analysis paragraphs for both overall artistic analysis and professional level analysis

[0037] In order to make the art painting analysis data more diverse, two aspects of analysis paragraphs are generated: the overall art analysis paragraph is extracted from the two generated analysis texts that only focus on visual features, and the analysis and annotation are performed to obtain the corresponding command text; the five most important professional level analysis paragraphs are respectively extracted from the two generated analysis texts that only focus on visual features, and the intersection of the five professional level analysis paragraphs is selected as the selected professional level analysis paragraph, and the analysis and annotation are performed to obtain the command text of the selected professional level, so as to obtain high-quality art painting analysis data consisting of the image of the art painting, the overall analysis paragraph and the corresponding command text, and the analysis paragraphs at each professional level and the corresponding command text, wherein the professional level includes composition, light and shadow, color, shape, texture, symbols and icons, perspective, movement and posture, line quality, and scale ratio.

[0038] Step S2: Train to obtain a generative pre-trained model for art painting

[0039] After collecting art painting analysis data, inspired by the multimodal large model ShareGPT4V, in this embodiment, a generative pre-trained model GalleryGPT for art paintings is constructed and fine-tuned, which can analyze art paintings with a focus on visual elements.

[0040] Step S2.1: Build a multimodal large model

[0041] Construct a multimodal large model consisting of a visual encoder, a multimodal projector and a large language model. The visual encoder encodes the image of the art painting to obtain visual features. The multimodal projector uses a multi-layer perceptron to project the visual features into the semantic space of the language. The large language model decodes the visual features projected into the semantic space of the language in combination with the command text to obtain an overall analysis paragraph or a professional level analysis paragraph.

[0042] The present invention aims to make the open source multimodal large model focus more on visual elements to analyze artistic paintings. In this embodiment, a multimodal large model is constructed using the open source multimodal large model ShareGPT4V-7B as a blueprint and base model. The multimodal large model mainly consists of three parts: (1) CLIP-Large visual encoder, which converts the size of the input image to 336×336 and then converts it into 576 input tokens; (2) Multimodal projector, which uses a multilayer perceptron to project visual features into the semantic space of language; (3) Vicuna-v1.5 large language model, which adopts the Transformer decoder architecture.

[0043] Step S2.2: Use the collected art painting images and corresponding art painting analysis data to perform supervised fine-tuning on the multimodal large model to obtain an art painting generative pre-training model

[0044] In order to utilize the superior visual perception and description capabilities of the ShareGPT4V-7B base model, in this embodiment, the collected art painting images and corresponding art painting analysis data are used to perform supervised fine-tuning on the multimodal large model: the parameters of the CLIP-Large visual encoder are frozen, the multimodal projector and the Vicuna-v1.5 large language model are fine-tuned, the learning rate of the training fine-tuning is set to 2e-5, the batch size is set to 16, and 10k steps of fine-tuning are performed to obtain an art painting generative pre-trained model, which can perceive the artistic elements and techniques in the painting, and output the corresponding description of the art analysis, i.e., the analysis paragraph.

[0045] Step S3: Inference

[0046] Input an art painting image I, a command text P containing n words for the instruction model to analyze, and output an analysis paragraph Output containing m words through the art painting generative pre-trained model GalleryGPT.

[0047] In this embodiment, the command text can be expressed as P = {w1, w2, ..., w n},w i For the i-th word, i = 1, 2, ..., n, the output text analysis Output = {o1, o2, ..., o m}, where o j is the j-th word among them, j=1,2,…,m.

[0048] Experimental results:

[0049] The present invention collects data for art painting analysis and fine-tunes to obtain a generative pre-trained model GalleryGPT for art paintings, which can generate an analysis of art paintings focusing on visual elements and alleviate the language bias problem caused by visual illusions.

[0050] In order to evaluate the effectiveness of the art painting analysis method based on the multimodal large model of the present invention, in the experiment, the experiment was first conducted on the analysis test data of 5,000 art paintings. Several popular open source multimodal large models were tested at the same time, including LLaVA-1.5, Qwen-VL-Chat and ShareGPT4V. All multimodal models use the same instruction command text: "Please write a coherent art analysis for this painting." The present invention uses the subtitle description evaluation indicators BLEU, GLEU, METEOR and ROUGE to evaluate the quality of the generated analysis text.

[0051]

[0052] Table 1

[0053] Table 1 shows the experimental results on the art painting analysis test data. As shown in Table 1, the present invention outperforms all other open source multimodal large models, and benefits from the supervised fine-tuning of art painting analysis data, and achieves significant improvements compared to the base model, which verifies the effectiveness of high-quality art analysis data collection and fine-tuning multimodal large models.

[0054] In order to verify the classification and question-answering capabilities of the art painting generative pre-training model GalleryGPT on art paintings, the present invention conducts test experiments on several existing art painting style classification and question-answering datasets.

[0055]

[0056] Table 2

[0057] Table 2 shows the experimental results on the art painting style classification and question answering test datasets. As can be seen from Table 2, the art painting generative pre-training model GalleryGPT of the present invention also significantly outperforms all baseline models, demonstrating its generalization ability for downstream art painting analysis tasks.

[0058] Figure 3The examples of instruction commands and the dialog capabilities of the present invention are shown in . These examples show that the GalleryGPT model is able to follow instructions for generating analysis of art paintings. Specifically, the GalleryGPT model can provide an overall analysis of the input painting, necessary level analysis, and artistic style. In summary, these qualitative examples show that the GalleryGPT model of the present invention has superior art painting analysis capabilities and illustrates that the collected art painting analysis dataset is of high quality.

[0059] Although the above describes the illustrative specific embodiments of the present invention to facilitate those skilled in the art to understand the present invention, it should be clear that the present invention is not limited to the scope of the specific embodiments. For those of ordinary skill in the art, as long as various changes are within the spirit and scope of the present invention as defined and determined by the attached claims, these changes are obvious, and all inventions and creations using the concept of the present invention are protected.

Claims

1. A method for analyzing artistic paintings based on a multimodal large model, characterized by comprising the following steps: (1) Data collection for art painting analysis 1.1) Get an image from the art painting library and use a closed-source large language model to determine whether the title and artist name of the image, i.e. the art painting, are known at the same time. If so, keep it, otherwise discard it; 1.2) Use two closed-source large language models to retrieve the learned knowledge through the title of the art painting and the name of the artist, delete the title of the art painting and the name of the artist, and each generate a paragraph of analysis text that only focuses on visual features; 1.3) Extract the overall art analysis paragraph from the two generated analysis texts that only focus on visual features, and analyze and annotate them to obtain the corresponding command text. Extract the five most important professional level analysis paragraphs from the two generated analysis texts that only focus on visual features, select the intersection of the five professional level analysis paragraphs as the selected professional level analysis paragraph, and analyze and annotate them to obtain the command text of the selected professional level. In this way, high-quality art painting analysis data consisting of the image of the art painting, the overall analysis paragraph and the corresponding command text, and the analysis paragraphs at each professional level and the corresponding command text are obtained, wherein, The professional aspects mentioned include composition, light and shadow, color, shape, texture, symbol and icon, perspective, movement and gesture, line quality, scale and proportion; (2) Train to obtain a generative pre-trained model for art paintings 2.1) Construct a multimodal large model consisting of a visual encoder, a multimodal projector, and a large language model. The visual encoder encodes the image of the art painting to obtain visual features. The multimodal projector uses a multi-layer perceptron to project the visual features into the semantic space of the language. The large language model decodes the visual features projected into the semantic space of the language in combination with the command text to obtain an overall analysis paragraph or a professional level analysis paragraph. 2.2) Use the collected art painting images and corresponding art painting analysis data to supervisely fine-tune the multimodal large model: freeze the parameters of the visual encoder, fine-tune the multimodal projector and the large language model, set the learning rate of training fine-tuning to 2e-5, set the batch size to 16, and perform 10k steps of fine-tuning to obtain the art painting generative pre-training model; (3) Reasoning Input an art painting image I, a command text P containing n words for the instruction model to analyze, and output an analysis paragraph Output containing m words through the art painting generative pre-trained model.

2. The method for analyzing artistic paintings based on a multimodal large model according to claim 1, characterized in that: Use the closed-source large language model Gemini to determine whether you know the titles and artist names of these art paintings.

3. The artistic painting analysis method based on a multimodal large model according to claim 1 is characterized in that: Use powerful closed-source large language models GPT-4 and Gemini to generate high-quality art analysis paragraphs.

4. The method for analyzing artistic paintings based on a multimodal large model according to claim 1, characterized in that: The multimodal large model is constructed using the open source multimodal large model ShareGPT4V-7B as a blueprint and base model.

Images