An art image multi-level description automatic generation method based on deep learning

Abstract

The application relates to an artistic image multi-level description automatic generation method based on deep learning in the technical field of artificial intelligence. The method comprises the following steps: acquiring an artistic image to be described, pre-processing the artistic image to be described to obtain a pre-processed image; extracting global visual features and local visual features of the pre-processed image in a deep learning mode; querying a metaphor knowledge graph according to the global visual features and the local visual features to obtain metaphor visual features of the artistic image; querying an expert sentence library according to multi-dimensional visual features to obtain an expert description sentence corresponding to the image features; obtaining an artistic title according to the multi-dimensional visual features and a predetermined title template; obtaining an artistic detailed description according to the expert description sentence and a predetermined detailed description template; and obtaining an artistic image multi-level description according to the artistic title and the artistic detailed description. The method can realize efficient generation of artistic image descriptions.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, in particular to an artistic image multi-level description automatic generation method based on deep learning. BACKGROUND

[0002] Image description generation is a computer vision technology that extracts image features and generates simple text descriptions for images. Existing image description generation work mainly focuses on natural image fields and medical research fields. In the natural image field, visual features identified are generated by a feature decoder or are mapped to entities and filled into a predetermined template to generate descriptions that can improve image retrieval efficiency. In the medical research field, image description generation can assist doctors in quickly producing diagnosis reports through patient pathological photos, improving doctor work efficiency and reducing work burden.

[0003] Current image description generation methods mainly include template-based image description generation methods and encoder-decoder architecture-based image description methods. The template-based image description generation method identifies entities contained in an image by pre-labeling entity labels, and then fills the identified entities into a pre-defined word slot to generate a description for the image. The encoder-decoder architecture-based image description method extracts image features through an image feature encoder, and converts the image feature encoding into a language vector using an NLP-based decoder to finally generate a description for the image.

[0004] However, existing image description generation methods are not suitable for the description of ancient art images. The specific reasons include that ancient art contains rich metaphorical information, special cultural semantics in function and modeling, and artistic distortion and unclear details in patterns, cultural metaphors usually need to analyze the cultural association between multiple features, and existing computer vision technology cannot identify the high-level cultural semantics contained between the features and multiple features of the art; since ancient art contains the aesthetic ideas of craftsmen and has undergone artistic creation, the entities in ancient art images are very different from natural images and are abstract; the information contained in the labels or titles generated by the existing image description generation method is insufficient, which leads to the fact that the current image description generation method cannot well cope with the ancient art image description automatic generation task, and the accuracy and the detail level of the description content are not high. SUMMARY

[0005] Therefore, it is necessary to provide an artistic image multi-level description automatic generation method based on deep learning in view of the above technical problems.

[0006] An artistic image multi-level description automatic generation method based on deep learning, characterized in that the method comprises:

[0007] An artwork image to be described is acquired, and pre-processing is performed on the artwork image to be described to obtain a pre-processed image.

[0008] Global visual features and local visual features of the pre-processed image are extracted in a deep learning manner.

[0009] Metaphorical visual features of the artwork image are obtained by querying a metaphorical knowledge graph according to the global visual features and the local visual features; the metaphorical knowledge graph is constructed according to a large number of cultural metaphors collected from museums and expert manuals.

[0010] An expert description sentence corresponding to the image features is obtained by querying an expert sentence library according to the global visual features, the local visual features and the metaphorical visual features; the expert sentence library is constructed according to feature keywords of the artwork and introduction sentences of the keywords by experts.

[0011] An artwork title is obtained according to the global visual features, the local visual features, the metaphorical visual features and a predetermined title template.

[0012] An artwork detailed description is obtained according to the expert description sentence and a predetermined detailed description template.

[0013] An artwork image multi-level description is obtained according to the artwork title and the artwork detailed description.

[0014] In one embodiment, the global visual features and the local visual features of the pre-processed image are extracted in a deep learning manner, including:

[0015] Global visual features of the artwork image are obtained by extracting global visual features of the pre-processed image using a global feature extractor; the global feature extractor includes a CBAM attention mechanism multi-classifier based on EfficientNet-b3 network and EfficientNet-b6 network and a convolution module.

[0016] Local visual features of the artwork image are obtained by extracting local visual features of the pre-processed image using a local feature extraction module; the local feature extraction module is a YOLO V3 network.

[0017] In one of the embodiments, the global visual features include color dimension, type dimension, and shape dimension global visual features; the global feature extractor includes three independent global feature extraction modules with the same structure and parameters not shared, which include EfficientNet-b3 network, EfficientNet-b6 network, two CBAM attention mechanism modules, full connection layer, and Softmax function.

[0018] The global visual features of the pre-processed image are extracted by the global feature extraction module to obtain the global visual features of the artwork image, including:

[0019] The pre-processed image is input into the EfficientNet-b3 network of the first global feature extraction module to obtain the first intermediate state image feature.

[0020] The first intermediate state image feature is input into the first CBAM attention mechanism module of the first global feature extraction module to obtain the first attention refined feature.

[0021] The pre-processed image is input into the EfficientNet-b6 network of the first global feature extraction module to obtain the second intermediate state image feature.

[0022] The second intermediate state image feature is input into the second CBAM attention mechanism module of the first global feature extraction module to obtain the second attention refined feature.

[0023] The first refined feature and the second refined feature are fused, and the fusion result is input into the full connection layer of the first global feature extraction module, and the output result is activated by the Softmax function to obtain the color dimension global visual feature.

[0024] The pre-processed image is input into the second global feature extraction module and the third global feature extraction module respectively to obtain the type dimension and shape dimension global visual features.

[0025] In one of the embodiments, the local visual features of the pre-processed image are extracted by the local feature extraction module to obtain the local visual features of the artwork image; the local feature extraction module is YOLO V3 network, which includes:

[0026] An artwork image corresponding to an artwork is obtained, and the artwork image is labeled to obtain a sample set.

[0027] The sample set is subjected to sample enhancement by an image enhancement algorithm to obtain a fine-tuning sample set.

[0028] Fine-tune the YOLO V3 network by using the fine-tuning training sample set, to obtain a trained YOLOV3 network.

[0029] Input the preprocessed image into the trained YOLO V3 network to obtain the local visual features of the artwork image.

[0030] In one embodiment, the global visual features include color dimension, ware type dimension, and shape dimension global visual features.

[0031] According to the global visual features and the local visual features, query the metaphor knowledge graph to obtain the metaphor visual features of the artwork image, including:

[0032] Take each of the color dimension, ware type dimension, and shape dimension global visual features and the local visual features as a query variable to obtain a query variable group.

[0033] According to the query variables in the query variable group, query the metaphor knowledge graph one by one as a query target, and when a matched triple is retrieved, output a metaphor set,

[0034] Take all the metaphor sets as the metaphor visual features of the artwork image.

[0035] In one embodiment, according to the global visual features, the local visual features, and the metaphor visual features, query the expert sentence library to obtain the expert description sentence corresponding to the image features, including:

[0036] Combine the color dimension, ware type dimension, shape dimension global visual features, local visual features, and metaphor visual features into a feature set.

[0037] Input the feature set into the expert sentence library to retrieve a description sentence for each image feature through a mapping relationship.

[0038] When a single image feature retrieves 1 description sentence, take the description sentence as the expert description sentence corresponding to the image feature.

[0039] When a single image feature retrieves more than 1 description sentence, take the first description sentence as the expert description sentence corresponding to the image feature according to the retrieval sequence, and take the other description sentences as candidate sentences.

[0040] In one embodiment, according to the artwork title and the artwork detailed description, obtain the multi-level description of the artwork image, including:

[0041] Splice the artwork title and the artwork detailed description to obtain the multi-level description of the artwork image.

[0042] The method for automatically generating multi-level descriptions of artwork images based on deep learning includes: obtaining an artwork image to be described, preprocessing the artwork image to be described to obtain a preprocessed image; extracting global visual features and local visual features of the preprocessed image based on deep learning; querying a metaphor knowledge graph based on the global visual features and the local visual features to obtain metaphor visual features of the artwork image; querying an expert sentence library based on multi-dimensional visual features to obtain an expert description sentence corresponding to the image features; obtaining an artwork title based on the multi-dimensional visual features and a predetermined title template; obtaining an artwork detailed description based on the expert description sentence and a predetermined detailed description template; and obtaining a multi-level description of the artwork image based on the artwork title and the artwork detailed description. The method can realize efficient generation of artwork image descriptions. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 An application scenario of a method for automatically generating multi-level descriptions of artwork images based on deep learning in an embodiment is shown in the figure.

[0044] Figure 2 A flowchart of a method for automatically generating multi-level descriptions of artwork images based on deep learning in an embodiment is shown in the figure.

[0045] Figure 3 A structure diagram of a global feature extraction module in an embodiment is shown in the figure. DETAILED DESCRIPTION

[0046] To make the objectives, technical solutions, and advantages of the present application clearer, further detailed descriptions of the present application are provided below in combination with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0047] The method for automatically generating multi-level descriptions of artwork images based on deep learning provided by the present application can be applied in an application environment as shown in the figure. Figure 1 In the figure, a data acquisition terminal 102 transmits data to a data processing terminal 104 through a network. The data acquisition terminal 102 can be, but is not limited to, a digital camera or a video camera, and the data processing terminal 104 can be, but is not limited to, various personal computers, notebook computers, tablet computers, or industrial control computers.

[0048] In an embodiment, as shown in the figure, a method for automatically generating multi-level descriptions of artwork images based on deep learning is provided. Figure 2 In the figure, a data acquisition terminal 102 transmits data to a data processing terminal 104 through a network. The data acquisition terminal 102 can be, but is not limited to, a digital camera or a video camera, and the data processing terminal 104 can be, but is not limited to, various personal computers, notebook computers, tablet computers, or industrial control computers. Figure 1 The method includes the following steps:

[0049] Step 100: obtaining an artwork image to be described, and pre-processing the artwork image to be described to obtain a pre-processed image.

[0050] Specifically, Image art ∈R C*H*W represents each image in the data set, where C is the number of channels of the image, H and W are its height and width. The pre-processing operation of the artwork image Image art to be described includes image size cropping and regularization processing. The pre-processing is to clean the image before the image is fed into the multi-classifier network, so as to improve the training performance of the network.

[0051] Step 102: extracting global visual features and local visual features of the pre-processed image by using a deep learning-based method.

[0052] Specifically, the global visual features of the shape dimension, color dimension and vessel shape dimension of the artwork image are extracted by using a multi-classifier based on deep learning; and the local features of the artwork image are extracted by using a target recognition technology based on deep learning.

[0053] Step 104: querying the metaphor knowledge graph according to the global visual features and the local visual features to obtain metaphor visual features of the artwork image.

[0054] Specifically, the metaphor knowledge graph is constructed according to a large number of cultural metaphors collected from museums and expert working manuals. The data in the knowledge graph is constructed based on the relationship of triplets, which contains features, relationships between features and entities of metaphors, so the primitives of the metaphor knowledge graph are designed as {partOf, contains, implies}.

[0055] The visual features of the ceramic artwork image include the global visual features f c of the color dimension, the global visual features f s of the vessel shape dimension, the global visual features f m of the shape dimension and the local visual features pattern f p . The combination of visual features of multiple dimensions will generate metaphor information containing cultural background. The global visual features f c of the color dimension, the global visual features f s of the vessel shape dimension, the global visual features f m of the shape dimension and the local visual features pattern f p are associated, and the correlation between the features of each dimension is represented as a knowledge graph.

[0056] By identifying the visual features of artwork images and associating the image features with feature keywords in the metaphor knowledge graph, we can explore deep cultural metaphor features. Cultural metaphor is a relatively obscure way of expression. Through a single feature or a combination of multiple features on the artwork image, we can express the cultural origins and metaphorical connotations recognized and accepted by the folk culture, greatly improving the professionalism of expert indexing and the efficiency of description generation.

[0057] Step 106: Based on the global visual features, local visual features and metaphorical visual features, query the expert sentence library to obtain expert description sentences corresponding to the image features; the expert sentence library is constructed based on the feature keywords of the artwork and the introduction sentences of the keywords by the artwork experts.

[0058] Specifically, the expert sentence database is a pre-built professional term corpus that contains the characteristic keywords of artworks and the introduction sentences of the keywords by artwork experts (expert sentences). Usually, one characteristic keyword can retrieve multiple corresponding explanatory sentences in the expert sentence database.

[0059] Step 108: Obtain the artwork title based on the global visual features, local visual features, metaphorical visual features, and a predetermined title template.

[0060] Specifically, global visual features, local visual features and metaphorical visual features are filled into a predetermined title template to generate a title for the artwork.

[0061] For example, the type name of the color, shape and form of the artwork extracted from the preprocessed image is f c 、f s 、f m , using the YOLO V3 target recognition network, the pattern type name in the output image is f p ; The name of the metaphorical knowledge graph (topological network) used to identify metaphors between multiple features is f mp .

[0062] The final artwork title Cap is a collection of multiple features in a specific order:

[0063] Cap={key(f c )||key(f s )||key(f p )||key(f m )||key(mp)}

[0064] Among them, key(f c ) is the type name of the global feature of the color dimension, key(f s ) is the type name of the global feature of the device dimension, key(f m) is the type name of the global feature of the shape dimension, key(f p ) is the pattern type name, key(mp) is the name of the metaphor between features, and || is the concatenation operation.

[0065] Step 110: obtaining the detailed description of the artwork according to the expert description sentence and the predetermined detailed description template.

[0066] Specifically, the expert description sentence corresponding to the retrieved image feature is filled into the predetermined detailed description template to generate the detailed description of the ceramic artwork. The sentences fc.sentenc, fs.sentenc, fp.sentenc, fm.sentenc, and mp.sentence that best match the feature description are extracted and combined according to the pre-set order of the description template to obtain the detailed description of the artwork.

[0067] Step 112: obtaining the multi-level description of the artwork image according to the artwork title and the detailed description of the artwork.

[0068] In the above method for automatically generating a multi-level description of an artwork image based on deep learning, the method comprises: obtaining an artwork image to be described, preprocessing the artwork image to be described to obtain a preprocessed image; extracting global visual features and local visual features of the preprocessed image using a deep learning-based method; querying the metaphor knowledge graph according to the global visual features and the local visual features to obtain metaphor visual features of the artwork image; querying an expert sentence library according to multi-dimensional visual features to obtain expert description sentences corresponding to the image features; obtaining an artwork title according to the multi-dimensional visual features and a predetermined title template; obtaining a detailed description of the artwork according to the expert description sentences and a predetermined detailed description template; and obtaining a multi-level description of the artwork image according to the artwork title and the detailed description of the artwork. This method can realize efficient generation of artwork image descriptions.

[0069] In one embodiment, step 102 specifically comprises the following steps:

[0070] Step 200: extracting global visual features of the preprocessed image using a global feature extractor to obtain global visual features of the artwork image; the global feature extractor comprises a CBAM attention mechanism multi-classifier based on EfficientNet-b3 network and EfficientNet-b6 network and a convolution module.

[0071] Step 202: extracting local visual features of the preprocessed image using a local feature extraction module to obtain local visual features of the artwork image; the local feature extraction module is a YOLO V3 network.

[0072] In one of the embodiments, the global visual features include: color dimension, type dimension, shape dimension global visual features; the global feature extractor includes: 3 independent global feature extraction modules with the same structure and parameters not shared, as shown in Figure 3 The global feature extraction module includes EfficientNet-b3 network, EfficientNet-b6 network, 2 CBAM attention mechanism modules, fully connected layer and Softmax function; step 200 includes: inputting the preprocessed image into the EfficientNet-b3 network of the first global feature extraction module to obtain the first intermediate state image feature; inputting the first intermediate state image feature into the first CBAM attention mechanism module of the first global feature extraction module to obtain the first attention refined feature; inputting the preprocessed image into the EfficientNet-b6 network of the first global feature extraction module to obtain the second intermediate state image feature; inputting the second intermediate state image feature into the second CBAM attention mechanism module of the first global feature extraction module to obtain the second attention refined feature; fusing the first refined feature and the second refined feature, inputting the fusion result into the fully connected layer of the first global feature extraction module, and activating the output result by using the Softmax function to obtain the color dimension global visual feature; inputting the preprocessed image into the second global feature extraction module and the third global feature extraction module respectively to obtain the type dimension and shape dimension global visual features.

[0073] Specifically, for the color f c , type f s , shape f m dimensional features contained in the global visual features, three dimensional global feature extraction modules are constructed in parallel. In this process, we respectively adjust the model parameters of the dataset to finally generate three parallel global feature extraction modules as the global visual feature extractor of ancient art image.

[0074] The preprocessed image is input into the EfficientNet-b3 and EfficientNet-b6 independently, and the two networks will produce two independent intermediate state image feature extraction results. Expressed by formula is: Where I art is the preprocessed image, is the output of the EfficientNet-b3 network, i.e. the first intermediate state image feature, is the output of the EfficientNet-b6 network, i.e. the second intermediate state image feature. Two intermediate features are input into two CBAM attention mechanism modules for sequential channel attention and spatial attention refinement.

[0075] The CBAM attention mechanism module includes two independent sub-modules, a channel attention module (CAM) and a spatial attention module (SAM), which respectively perform attention feature extraction in the channel and space. In this way, not only can the parameters and computing power be saved, but also it can be integrated into the existing network architecture as a plug-and-play module.

[0076] image features in the first intermediate state The process of feature attention feature extraction is illustrated by taking the input into the CBAM attention mechanism module as an example. The image features in the first intermediate state The image features in the first intermediate state are respectively input into the channel attention module of the CBAM attention mechanism module, and the image features in the first intermediate state are respectively subjected to global max pooling and global average pooling to obtain two 1×1×C feature maps F, and then they are respectively sent into a two-layer neural network (MLP). Then, the features output by the MLP are subjected to element-wise addition operation and sigmoid activation operation to generate the final channel attention feature M c (F). c (F) = σ(MLP(avgpool(F)) + MLP(maxpool(F))), where σ is the sigmoid function, avgpool is the global max pooling, maxpool is the global average pooling, and MLP is a two-layer perceptron network with one hidden layer.

[0077] Finally, the channel attention feature M c (F) and the image features in the first intermediate state

[0077] are subjected to element-wise multiplication operation to generate the input feature of the spatial attention module

[0078] The feature is input into the spatial attention module of the CBAM attention mechanism module. First, the feature is subjected to channel-based global max pooling and global average pooling to obtain two H×W×1 feature maps F c , and then the two feature maps are concatenated based on the channel. Then, a 7×7 convolution operation is performed to reduce the dimension to 1 channel, i.e., H×W×1. Then, the sigmoid generates the spatial attention feature M s (F c ), M s (F c) = σ(f kernel ([avgpool(F c ) ; maxpool(F c )])). Finally, the features M s (F c ) and the input of this module are multiplied to get the final generated feature

[0079] The output of the EfficientNet-b3 network is The output of the EfficientNet-b6 network is Finally, the outputs of the two networks are fused together to get as the image feature vector containing global features. Then the fused features are mapped to the feature space and input to the fully connected layer, and then the class with the highest prediction probability is predicted by the softmax function as the output of the global feature classification of the target image gf = {f c , f s , f m}.

[0080] Each picture in the ceramic image label set has multiple visual labels. For example, in a certain image case summary, the global feature multi-classification label of the ceramic image is: {blue and white, hold, pot}; the global visual features of the ceramic artwork are extracted by the global visual feature extractor, and the Global Feature = {f c : blue and white, f s : hold, f m : pot} is generated.

[0081] In one embodiment, step 202 includes: obtaining an artwork image corresponding to an artwork real object, labeling the artwork image to obtain a sample set; using an image enhancement algorithm to enhance the sample set to obtain a fine-tuning sample set; using the fine-tuning training sample set to fine-tune the YOLO V3 network to obtain a trained YOLO V3 network; inputting the preprocessed image into the trained YOLO V3 network to obtain the local visual features of the artwork image.

[0082] Specifically, the pattern feature is drawn or engraved on the surface of the ceramic artwork, which is used for decoration or highlighting of specific historical background. The pattern is a local visual feature of the artwork, and the YOLO V3 network is used as a local feature extractor to extract the pattern f p feature of the ceramic artwork. The YOLO V3 network has been proven to have excellent performance in target recognition of artwork images.

[0083] Through the setting of the YOLO V3 network, the pattern f p ​The feature setting prediction frame is matched with the calibration frame, and the final ceramic artwork obtains the pattern feature classification result as the local visual feature.

[0084] The classification prediction of the artwork pattern through the network outputs the confidence of the classification, and the interval of the confidence is within [0, 1], for example, (Long Dragon pattern, 0.92). And the position coordinates (bx, by, bw, bh) of the pattern recognition frame are output. A ceramic artwork usually outputs multiple pattern feature recognition results.

[0085] In addition, due to the distortion and insufficient clarity of the pattern features of the ceramic artwork image and the ceramic artwork, the model parameter is fine-tuned using the sample set containing the annotation in the data set. In the fine-tuning process, the image enhancement algorithm of rotation and hue adjustment is used to enrich the sample categories and improve the robustness of the model.

[0086] It is worth noting that the extraction of local visual features of artwork images is not limited to using convolutional neural networks. Other methods such as using SVM to extract HOG features, bag-of-words model of features such as SURF and MSER, or Viola-Jones algorithm can also be used.

[0087] In one embodiment, the global visual features include: color dimension, vessel type dimension, and shape dimension global visual features; step 104 includes: taking each of the color dimension, vessel type dimension, and shape dimension global visual features and the local visual features as query variables to obtain a query variable group; and querying the metaphor knowledge graph one by one according to the query variables in the query variable group, and when a matching triple is retrieved, outputting a metaphor set; and taking all metaphor sets as the metaphor visual features of the artwork image.

[0088] Specifically, when the complete feature relationship of a ceramic artwork image is obtained {f c ,f s ,f m ,f p}, each independent feature f i is taken as a query variable to query the metaphor knowledge graph in a traversal manner, and each combination f i ∧f j [∧...] is inferred. When a matching triple is retrieved, the metaphor set {mp1,..,mp t}, mp k ←f i [∧f j ] is output. f mp is the metaphor visual feature. Taking "He He Er Xi" (f mp ) as an example, it is combined with "lotus" (f p ) and "box" (fm ) the metaphorical relationship is constructed.

[0089] In one embodiment, the step 106 includes: grouping the global visual features of the color dimension, the shape dimension, the form dimension, the local visual features and the metaphorical visual features into a feature set; inputting the feature set into the expert sentence library, retrieving a description sentence for each image feature through the mapping relationship; when a single image feature retrieves 1 description sentence, taking the description sentence as the expert description sentence corresponding to the image feature; when a single image feature retrieves more than 1 description sentence, taking the first description sentence as the expert description sentence corresponding to the image feature according to the retrieval sequence, and taking the other description sentences as candidate sentences.

[0090] Specifically, usually one feature keyword can retrieve a plurality of corresponding explanation sentences in the expert sentence library. All the features: the global visual feature f c of the color dimension, the global visual feature f s of the shape dimension, the global visual feature f m of the form dimension, the local visual feature pattern f p and the metaphorical visual feature mp are extracted to form a feature set {f c ,f s ,f m ,f p ,mp1,...,mp t} which is input to the expert sentence library, and the description sentences are retrieved for the feature keywords through the mapping relationship, and the obtained description sentences corresponding to the image features are taken as the components of the artwork image description, so that the artwork image description is fc.sentence||fs.sentence||fp.sentence||fm.sentence||mp.sentence. When a plurality of sentences are retrieved simultaneously, the first explanation sentence is output according to the retrieval sequence and the other sentences are taken as candidate sentences.

[0091] Retrieving a description sentence for each image feature through the mapping relationship means that each image feature corresponds to one or more expert sentences in the expert sentence library, and the recognized image feature is taken as a query keyword to query in the expert sentence library to retrieve the sentence corresponding to the current feature.

[0092] In one embodiment, the step 112 includes: splicing the artwork title and the artwork detailed description to obtain the multi-level description of the artwork image.

[0093] Specifically, the multi-level description of the artwork image Des is finally generated by combining the artwork title Cap and the artwork detailed description. The final multi-level description of the artwork image Des is the set of the artwork title Cap and the artwork detailed description. The multi-level description of the artwork image Des is:

[0094] Des = {Cap || f c .sentence||f s .sentence||f p .sentence||f m .sentence||mp.sentence}

[0095] It should be understood that, although Figure 1 the steps in the flowchart of FIG. 1 are shown in sequential order, such that each step is performed after another, in fact, these steps can be performed in other order. Unless otherwise specified, the execution of the steps is not strictly limited in sequence, and the steps can be executed in other order. Moreover, Figure 1 at least a part of the steps in the flowchart of FIG. 1 can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order of the sub-steps or stages is not necessarily sequential, but can be alternately or alternately executed with other steps or sub-steps or stages of other steps.

[0096] The technical features of the above embodiments can be combined in any manner. To make the description concise, all possible combinations of the technical features in the above embodiments are not described, however, as long as the combination of the technical features does not exist contradictions, it should be considered as the scope of the present disclosure.

[0097] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that, for ordinary skilled in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which are all within the scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for automatically generating multi-level descriptions of artwork images based on deep learning, characterized in that: The method comprises: Acquire an image of a work of art to be described, and preprocess the image of the work of art to be described to obtain a preprocessed image; The global and local visual features of the pre-processed images are extracted using a deep learning-based approach; performing a query on a metaphor knowledge graph based on the global visual features and the local visual features to obtain metaphorical visual features of the artwork image; the metaphor knowledge graph is constructed based on a large number of cultural metaphors collected from museums and expert workbooks; Based on the global visual features, the local visual features, and the metaphorical visual features, a query is performed in an expert sentence library to obtain expert description sentences corresponding to the image features; the expert sentence library is constructed based on the characteristic keywords of the artwork and the introduction sentences of the keywords by the artwork experts; Obtaining a title for the artwork based on the global visual features, the local visual features, the metaphorical visual features, and a predetermined title template; Obtaining a detailed description of the artwork according to the expert description statement and a predetermined detailed description template; A multi-level description of the artwork image is obtained according to the artwork title and the artwork detailed description.

2. The method according to claim 1, characterized in that The global and local visual features of the pre-processed image are extracted based on deep learning, including: A global feature extractor is used to extract global visual features of the preprocessed image to obtain global visual features of the artwork image; the global feature extractor includes: a CBAM attention mechanism multi-classifier based on the EfficientNet-b3 network and the EfficientNet-b6 network, and a convolution module; A local feature extraction module is used to extract local visual features of the preprocessed image to obtain local visual features of the artwork image; the local feature extraction module is a YOLO V3 network.

3. The method according to claim 2, characterized in that The global visual features include: global visual features of color dimension, device dimension, and shape dimension; the global feature extractor includes: three independent global feature extraction modules with the same structure and no shared parameters, and the global feature extraction modules include an EfficientNet-b3 network, an EfficientNet-b6 network, two CBAM attention mechanism modules, a fully connected layer, and a Softmax function; A global feature extraction module is used to extract global visual features of the preprocessed image to obtain global visual features of the artwork image, including: Inputting the preprocessed image into the EfficientNet-b3 network of the first global feature extraction module to obtain image features of the first intermediate state; Inputting the image features of the first intermediate state into the first CBAM attention mechanism module of the first global feature extraction module to obtain the first attention refined features; Inputting the preprocessed image into the EfficientNet-b6 network of the first global feature extraction module to obtain image features of the second intermediate state; Inputting the image features of the second intermediate state into the second CBAM attention mechanism module of the first global feature extraction module to obtain the second attention refined features; The first attention refinement feature and the second attention refinement feature are fused, and the fusion result is input into the fully connected layer of the first global feature extraction module, and the output result is activated by the Softmax function to obtain the global visual feature of the color dimension; The pre-processed images are input into the second global feature extraction module and the third global feature extraction module respectively to obtain global visual features of the shape dimension and the form dimension.

4. The method according to claim 2, characterized in that A local feature extraction module is used to extract local visual features of the preprocessed image to obtain local visual features of the artwork image; the local feature extraction module is a YOLO V3 network, including: Obtain artwork images corresponding to the physical artworks, annotate the artwork images, and obtain a sample set; Performing sample enhancement on the sample set using an image enhancement algorithm to obtain a fine-tuning training sample set; Fine-tune the parameters of the YOLO V3 network using the fine-tuning training sample set to obtain a trained YOLO V3 network; The preprocessed image is input into the trained YOLO V3 network to obtain local visual features of the artwork image.

5. The method according to claim 1, wherein The global visual features include: global visual features of color dimension, shape dimension, and form dimension; The metaphorical visual features of the artwork image are obtained by querying the metaphor knowledge graph based on the global visual features and the local visual features, including: The global visual features of the color dimension, the shape dimension, and the form dimension and each independent feature of the local visual features are used as query variables to obtain a query variable group; According to the query variables in the query variable group, query is performed one by one as query target in the metaphor knowledge graph, and when a matching triple is retrieved, a metaphor set is output. The collection of all metaphors is used as the metaphorical visual features of the artwork image.

6. The method according to claim 5, characterized in that According to the global visual features, the local visual features, and the metaphorical visual features, a query is performed in an expert sentence library to obtain an expert description sentence corresponding to the image features, including: The global visual features, local visual features and metaphorical visual features of the color dimension, shape dimension and format dimension are combined into a feature set; Input the feature set into an expert sentence library, and retrieve a description sentence for each image feature through a mapping relationship; When a single image feature retrieves a description statement, the description statement is used as the expert description statement corresponding to the image feature; When a single image feature retrieves more than one description sentence, the first description sentence is used as the expert description sentence corresponding to the image feature according to the retrieval sequence, and the other description sentences are used as candidate sentences.

7. The method according to claim 1, characterized in that According to the artwork title and the artwork detailed description, a multi-level description of the artwork image is obtained, including: The artwork title and the artwork detailed description are spliced ​​together to obtain a multi-level description of the artwork image.

Images