Multi-modal deep learning method for art work analysis

Abstract

The invention discloses a multi-modal deep learning method for artistic work analysis, and relates to the technical field of computer vision, and the method comprises the steps: multi-modal data preprocessing: obtaining a visual image, text description and metadata of an artistic work, the method has the advantages that valuable feature vectors are extracted from visual images, text description and metadata of artistic works through multi-modal data preprocessing, the importance of all modal features is dynamically adjusted by means of an attention mechanism in the cross-modal dynamic fusion step, style information of different sources can be effectively integrated, and the effect of improving the artistic work quality is achieved. In the style migration stage, the multi-scale feature pyramid network is used for fusing feature information of different levels, real-time feedback optimization is continuously evaluated and parameters are adjusted through a discriminator, and compared with an existing neural style migration method, the method has the advantages that complex artistic styles are better processed, core elements of specific styles are accurately reserved in the migration process, and the method is suitable for popularization and application. And the accuracy of style migration and the artistic expressive force are improved.

Description

Technical Field

[0001] The present invention relates to the technical field of computer vision, and particularly to a multi-modal deep learning method for art work analysis. Background Art

[0002] At present, with the deep integration of digital technology and art education, art work analysis is undergoing a transformation from traditional manual appreciation to an intelligent and diversified direction. Art works contain rich visual, semantic and cultural information. In-depth analysis of them not only helps the development of art education, but also provides strong support for fields such as art creation and cultural heritage protection. With the continuous progress of computer vision and deep learning technologies, traditional art work analysis methods have gradually been replaced by intelligent means. Early art work analysis mainly relied on the subjective judgment of experts and manual feature extraction. This method is inefficient and greatly affected by subjective factors, and it is difficult to meet the needs of large-scale art work analysis.

[0003] There are certain defects in the prior art. First, the prior art mostly relies on single-modal analysis, making it difficult to integrate multi-source style information, prone to losing details when dealing with complex art styles, and having poor style consistency. Second, traditional methods lack an adaptive regulation mechanism, with low style transfer accuracy for images of different complexities, unable to balance detail retention and style enhancement. Therefore, we propose a multi-modal deep learning method for art work analysis. Summary of the Invention

[0004] The purpose of the present invention is to provide a multi-modal deep learning method for art work analysis.

[0005] To achieve the above purpose, the present invention provides the following technical solution: A multi-modal deep learning method for art work analysis, the method comprising the following steps:

[0006] Step 1, multi-modal data preprocessing: Obtain the visual image, text description and metadata of the art work, perform multi-scale feature extraction on the visual image to generate a visual feature vector containing global and local features, perform semantic encoding on the text description to generate a semantic feature vector, and perform structured processing on the metadata to generate a metadata feature vector;

[0007] Step 2, cross-modal dynamic fusion: Construct a cross-modal attention mechanism, calculate the attention weights between the visual feature vector, semantic feature vector and metadata feature vector, and dynamically adjust the importance of each modal feature according to the attention weights to generate a fused feature vector;

[0008] Step 3. Hierarchical style transfer: Use a multi-scale feature pyramid network to construct a hierarchical transfer network. Input the fused feature vector into the hierarchical transfer network, retain texture details at the low level, and transfer the overall style at the high level to generate a preliminary style transfer image;

[0009] Step 4. Dynamic style intensity control: Analyze the edge density and texture entropy of the original image, and automatically adjust the style transfer intensity according to the edge density and texture entropy to generate an intermediate result image. Among them, the edge density is the proportion of edge pixels in the image calculated by the Canny edge detection algorithm, and the texture entropy is obtained by calculating the gray-level co-occurrence matrix;

[0010] Step 5. Real-time feedback optimization: Use a discriminator to evaluate the style consistency of the intermediate result image in real time, and dynamically adjust the cross-modal attention weights and hierarchical transfer network parameters according to the evaluation results to generate the final style transfer image;

[0011] Step 6. Personalized creation support: Receive user input custom style parameters, including custom style parameters such as color saturation and stroke intensity, and combine with the user's historical creation data to generate a personalized style transfer model to achieve personalized creation of art works;

[0012] Step 7. User interaction and feedback: Provide a user interaction interface to support users to adjust style parameters in real time and preview the transfer effect, and collect user feedback data for continuous optimization of system performance.

[0013] As a further solution of the present invention: In the above step 1, a ResNet network with increased attention channels is used for feature extraction of visual images, and the channel weights are adaptively adjusted by calculating the correlation between channels. The calculation formula is as follows:

[0014]

[0015] where, a c represents the attention weight of channel c, C represents the total number of channels of the feature map, w1 represents the weight matrix of the first fully connected layer, w2 represents the weight matrix of the second fully connected layer for further linearly transforming the transformed features, δ represents the ReLU activation function, and F c represents the feature map of channel c, and avg_pool represents the global average pooling operation.

[0016] As a further solution of the present invention: In the above step 2, the cross-modal attention mechanism adopts a multi-head self-attention structure, and the specific process is as follows:

[0017] The visual, text, and metadata feature vectors are linearly transformed through independent fully connected layers to generate a query matrix Q, a key matrix K, and a value matrix V. The transformation weight matrix for each modality is learnable. The query matrix Q is used to find the associations between features, the key matrix K is used to represent feature identities, and the value matrix V is used to perform weighted combination of features according to the association relationships. For each head, the attention calculation formula is as follows:

[0018]

[0019] where d k represents the dimension of the key vector divided by is to scale the dot product. Calculate the dot product of Q and the transpose of K to obtain the similarity score between each query vector and the key vector. Then, through the softmax function, the scores are transformed into a probability distribution. Finally, according to this probability distribution, a weighted sum is performed on the value matrix V to obtain the attention output of each head. The attention outputs of multiple heads are combined through a concatenation operation and then passed through a linear transformation to obtain the final cross-modal attention fusion result.

[0020] As a further solution of the present invention: In the third step, the construction process of the multi-scale feature pyramid network (FPN) is as follows:

[0021] Starting from the high-level features of the backbone network ResNet, through the upsampling operation, the high-level semantic features are gradually upsampled to the same spatial size as the low-level features. The upsampled high-level features are added element-wise to the corresponding low-level features in the backbone network, and the high-level semantic features and low-level detail features are fused through the upsampling operation. The specific calculation formula is

[0022] F up = upsample(F high ) + F low

[0023] where F up is the fused feature, F high is the high-level feature, F low is the low-level feature, and upsample is the upsampling operation. Through style transfer on the FPN features at different levels respectively, hierarchical style transfer from coarse-grained to fine-grained is realized. Finally, the transfer results of each level are fused to generate a preliminary style transfer image.

[0024] As a further solution of the present invention: In the fourth step, the edge density is calculated as follows:

[0025] Edge detection is performed on the original image through the Canny edge detection algorithm. Assume the size of the original image is M×N. After being processed by the Canny edge detection algorithm, a binary edge image is obtained, where the edge pixel value is 1 and the non-edge pixel value is 0. The edge density ρ is defined as the ratio of the total number of edge pixels E to the total number of pixels M×N in the image. Its calculation formula is as follows:

[0026]

[0027] where I ij represents the pixel value at the i-th row and j-th column in the binary edge image;

[0028] Calculation method of texture entropy: Texture entropy is calculated through the gray-level co-occurrence matrix. Assume the gray-level co-occurrence matrix is P, and its element P ij represents the frequency of the pixel pair with gray values i and j appearing at a given distance and direction. The calculation formula of texture entropy H is as follows:

[0029]

[0030] where G is the number of gray levels of the image. According to the calculated edge density and texture entropy, the style transfer intensity is automatically adjusted to generate an intermediate result image.

[0031] As a further solution of the present invention: In the fifth step, the discriminator adopts the Patch-GAN structure, and the specific evaluation method is as follows:

[0032] First, the generated image is divided into multiple patches with a size of 70×70, and each patch is discriminated for authenticity. By minimizing the following loss function, the generator G generates a style transfer image close to the real data distribution:

[0033]

[0034] where D is the discriminator, G is the generator, Pdata is the real data distribution, Pz is the noise distribution. The gradient of the loss function with respect to the parameters is calculated through the backpropagation algorithm, and the training parameters of the discriminator and the generator are updated according to the chain rule.

[0035] As a further solution of the present invention: In the sixth step, the user's historical creation data will be collected. These data include the style parameters input by the user and the corresponding generation results. The Gaussian process regression model is used to associate the user's currently input custom style parameters with the historical data, and the generation result corresponding to the currently input style parameters is predicted through this model to realize the generation of a personalized style transfer model.

[0036] As a further solution of the present invention: in step seven, the user interface adopts progressive visualization technology. During the process of the user adjusting the style parameters, the details of the generated image will be gradually rendered. The initial rendering progress is 0%, and as the parameters are adjusted, the rendering progress will continuously increase until it reaches 100% to complete the rendering. The user can view the rendering progress in real time and adjust the style parameters according to the preview effect.

[0037] Adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are as follows:

[0038] 1. By using multi-modal data preprocessing, the present invention extracts valuable feature vectors from the visual images, text descriptions and metadata of art works respectively. In the cross-modal dynamic fusion step, with the help of the attention mechanism, the importance of each modal feature is dynamically adjusted, enabling effective integration of style information from different sources. In the style transfer stage, the multi-scale feature pyramid network is used to fuse feature information at different levels, which can retain rich texture details at low levels and accurately transfer the overall style at high levels, avoiding the problem of detail loss. The dynamic style intensity control automatically adjusts the transfer intensity based on the edge density and texture entropy of the image, and the real-time feedback optimization continuously evaluates and adjusts the parameters through the discriminator, ensuring style consistency. Compared with existing neural style transfer methods, the present invention can better handle complex art styles, accurately retain the core elements of specific styles during the transfer process, and significantly improve the accuracy and artistic expressiveness of style transfer.

[0039] 2. Through multi-modal data preprocessing and dynamic feature regulation mechanism, the present invention significantly improves the transfer accuracy and adaptability of complex art styles. In the preprocessing stage, the improved ResNet network combines with the channel attention module to adaptively enhance the key feature channels and effectively capture the subtle differences of art styles. The structured processing of text and metadata supplements semantic and background information, breaking through the limitations of single-modal analysis. The cross-modal dynamic fusion adopts the multi-head self-attention mechanism to parallelly process the feature interactions in different subspaces, realizing deep fusion of multi-modal information and enhancing the feature expression ability. The hierarchical style transfer enhances and laterally connects through the top-down path of the FPN network, organically integrating high-level semantics and low-level details, accurately transferring the overall style while retaining details such as brushstrokes and textures, and solving the problem of detail loss in traditional methods for complex style transfer. The dynamic style intensity control automatically adjusts the transfer intensity based on the image edge density and texture entropy, retaining the details of the original image in high-complexity images and enhancing the style performance in low-complexity images. Compared with the prior art, through multi-modal deep fusion, adaptive feature processing and dynamic regulation mechanism, the present invention realizes the dual improvement of high-precision transfer of complex art styles and artistic expressiveness, especially significantly enhancing the adaptability of the system to different image complexities while maintaining style consistency. Description of the Drawings

[0040] Figure 1 This is a schematic diagram of the method steps in the embodiments of the present invention. Detailed implementation manners

[0041] The following further describes the detailed implementation manners of the present invention with reference to the accompanying drawings. It should be noted here that the description of these implementation manners is used to help understand the present invention, but does not limit the present invention.

[0042] In addition, the technical features involved in the various implementation manners of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0043] Please refer to the attached Figure 1 , a multi-modal deep learning method for art work analysis of the present invention, the method includes the following steps:

[0044] Step 1. Multi-modal data preprocessing: Obtain the visual image, text description and metadata of the art work, perform multi-scale feature extraction on the visual image to generate a visual feature vector including global and local features, perform semantic encoding on the text description to generate a semantic feature vector, and perform structured processing on the metadata to generate a metadata feature vector;

[0045] Step 2. Cross-modal dynamic fusion: Construct a cross-modal attention mechanism, calculate the attention weights between the visual feature vector, semantic feature vector and metadata feature vector, and dynamically adjust the importance of each modal feature according to the attention weights to generate a fused feature vector;

[0046] Step 3. Hierarchical style transfer: Use a multi-scale feature pyramid network to construct a hierarchical transfer network, input the fused feature vector into the hierarchical transfer network, retain texture details at a low level, and transfer the overall style at a high level to generate a preliminary style transfer image;

[0047] Step 4. Dynamic style intensity control: Analyze the edge density and texture entropy of the original image, automatically adjust the style transfer intensity according to the edge density and texture entropy to generate an intermediate result image, where the edge density is the ratio of edge pixels in the image calculated by the Canny edge detection algorithm, and the texture entropy is obtained by calculating the gray-level co-occurrence matrix;

[0048] Step 5. Real-time feedback optimization: Use a discriminator to evaluate the style consistency of the intermediate result image in real time, dynamically adjust the cross-modal attention weights and the parameters of the hierarchical transfer network according to the evaluation results to generate a final style transfer image;

[0049] Step 6: Personalized Creation Support: Receive custom style parameters input by the user, including custom style parameters such as color saturation and brush stroke intensity, and combine the user's historical creation data to generate a personalized style transfer model to achieve personalized creation of artworks;

[0050] Step 7: User Interaction and Feedback: Provide a user interaction interface to support the user in real-time adjusting style parameters and previewing the migration effect, and collect user feedback data for continuous optimization of system performance.

[0051] In an embodiment of the present invention: In Step 1, a ResNet network with an added attention channel is used for feature extraction of visual images, and the channel weights are adaptively adjusted by calculating the correlation between channels. The calculation formula is as follows:

[0052]

[0053] where a c represents the attention weight of channel c, C represents the total number of channels of the feature map, w1 represents the weight matrix of the first fully connected layer, w2 represents the weight matrix of the second fully connected layer for further linearly transforming the transformed features, δ represents the ReLU activation function, and F c represents the feature map of channel c, and avg_pool represents the global average pooling operation.

[0054] In an embodiment of the present invention: In Step 2, the cross-modal attention mechanism adopts a multi-head self-attention structure, and the specific process is as follows:

[0055] The visual, text, and metadata feature vectors are respectively linearly transformed through independent fully connected layers to generate a query matrix Q, a key matrix K, and a value matrix V. The transformation weight matrix of each modality is learnable. The query matrix Q is used to find the association between features, the key matrix K is used to represent feature identification, and the value matrix V is used to perform weighted combination of features according to the association relationship. For each head, the attention calculation is performed, and the calculation formula is as follows:

[0056]

[0057] where d k represents the dimension of the key vector divided by is to scale the dot product, calculate the dot product of Q and the transpose of K to obtain the similarity score between each query vector and the key vector, then convert the score into a probability distribution through the softmax function, and finally perform weighted summation on the value matrix V according to the probability distribution to obtain the attention output of each head. The attention outputs of multiple heads are combined together through a concatenation operation and then passed through a linear transformation to obtain the final cross-modal attention fusion result.

[0058] In one embodiment of the present invention: In step three, the construction process of the multi-scale feature pyramid network (FPN) is as follows:

[0059] Starting from the high-level features of the backbone network ResNet, through the upsampling operation, the high-level semantic features are gradually upsampled to the same spatial size as the low-level features. The upsampled high-level features are added element-wise to the corresponding low-level features in the backbone network, and the high-level semantic features are fused with the low-level detail features through the upsampling operation. The specific calculation formula is

[0060] F up = upsample(F high ) + F low

[0061] where F up is the fused feature, F high is the high-level feature, F low is the low-level feature, and upsample is the upsampling operation. By performing style transfer on the FPN features at different levels respectively, hierarchical style transfer from coarse-grained to fine-grained is realized. Finally, the transfer results at each level are fused to generate a preliminary style transfer image.

[0062] In one embodiment of the present invention: In step four, the edge density is calculated as follows:

[0063] The original image is subjected to edge detection through the Canny edge detection algorithm. Let the size of the original image be M×N. After being processed by the Canny edge detection algorithm, a binary edge image is obtained, where the edge pixel value is 1 and the non-edge pixel value is 0. The edge density ρ is defined as the ratio of the total number of edge pixels E to the total number of pixels M×N in the image. Its calculation formula is as follows:

[0064]

[0065] where I ij represents the pixel value at the i-th row and j-th column in the binary edge image;

[0066] The calculation method of texture entropy: The texture entropy is calculated through the gray-level co-occurrence matrix. Let the gray-level co-occurrence matrix be P, and its element P ij represents the frequency of occurrence of pixel pairs with gray values i and j at a given distance and direction. The calculation formula of the texture entropy H is as follows:

[0067]

[0068] where G is the number of gray levels of the image. According to the calculated edge density and texture entropy, the style transfer intensity is automatically adjusted to generate an intermediate result image.

[0069] In one embodiment of the present invention: In step five, the discriminator adopts a Patch-GAN structure, and the specific evaluation method is as follows:

[0070] First, divide the generated image into multiple patches of size 70×70, and perform true / false discrimination on each patch. By minimizing the following loss function, the generator G generates a style transfer image close to the real data distribution:

[0071]

[0072] where D is the discriminator, G is the generator, Pdata is the real data distribution, Pz is the noise distribution, calculate the gradient of the loss function with respect to the parameters through the backpropagation algorithm, and update the training parameters of the discriminator and the generator according to the chain rule.

[0073] In one embodiment of the present invention: In step six, the user's historical creation data will be collected. These data include the style parameters input by the user and the corresponding generation results. Use the Gaussian process regression model to associate the custom style parameters currently input by the user with the historical data, and predict the generation results corresponding to the currently input style parameters through this model to achieve the generation of a personalized style transfer model.

[0074] In one embodiment of the present invention: In step seven, the user interface adopts progressive visualization technology. During the process of the user adjusting the style parameters, the details of the generated image will be gradually rendered. The initial rendering progress is 0%, and as the parameters are adjusted, the rendering progress will continuously increase until it reaches 100% to complete the rendering. The user can view the rendering progress in real time and adjust the style parameters according to the preview effect.

[0075] Example 1. Please refer to the appendix Figure 1 , Application of an art education platform based on multi-modal deep learning:

[0076] Multi-modal data preprocessing:

[0077] Visual image preprocessing: Obtain the visual image of the art work, and use the improved ResNet network for multi-scale feature extraction. For an oil painting image of Van Gogh's "The Starry Night", the network will extract its global swirling nebula features and local brushstroke texture features, and generate a visual feature vector containing global and local features;

[0078] Text description preprocessing: Perform semantic encoding on the text description of the image. Input "This painting has strong color contrast and dynamic brushstrokes", and generate a semantic feature vector through semantic encoding to capture the style description information in the text;

[0079] Metadata Preprocessing: Structured processing of metadata (including creation date, artist, genre, size, material, collection institution, creative background, style characteristics, theme, color tone, brushstroke type, etc.):

[0080] Creation Date: Convert "1889" into time features (timestamp or time period);

[0081] Artist: Convert "Van Gogh" into categorical features (artist ID or genre label);

[0082] Genre: Convert "Post-Impressionism" into categorical features;

[0083] Size: Convert "73.7cm × 92.1cm" into numerical features (aspect ratio or standardized size);

[0084] Material: Convert "oil canvas, oil paint" into material type features;

[0085] Collection Institution: Convert "Museum of Modern Art, New York" into institutional labels;

[0086] Creative Background: Convert "Expressing inner restlessness and love for nature" into semantic features;

[0087] Style Characteristics: Convert "strong color contrast, dynamic brushstrokes" into style keyword vectors;

[0088] Theme: Convert "Starry Night, Village" into theme labels;

[0089] Color Tone: Convert "dark blue, bright yellow" into color features (such as RGB mean or dominant color distribution);

[0090] Brushstroke Type: Convert "swirls, short brushstrokes" into brushstroke feature vectors.

[0091] Cross-modal Dynamic Fusion:

[0092] Construct a cross-modal attention mechanism, and obtain the query matrix Q, key matrix K, and value matrix V by linearly transforming the visual feature vector, semantic feature vector, and metadata feature vector respectively;

[0093] Adopt a multi-head self-attention structure to calculate the attention weights between features of each modality. When analyzing an oil painting with a text description containing "strong colors", the attention mechanism will dynamically adjust the weights of the color-related channels in the visual features to enhance the capture of color features.

[0094] Dynamically adjust the importance of features of each modality according to the attention weights to generate a fused feature vector.

[0095] Hierarchical Style Transfer:

[0096] A hierarchical transfer network is constructed using a multi-scale feature pyramid network (FPN). Starting from the high-level features of ResNet, the high-level semantic features (overall style) are fused with the low-level features (texture details) through upsampling operations.

[0097] The fused feature vectors are input into the hierarchical transfer network, which retains texture details at the low level and transmits the overall style at the high level. When transferring the style of "The Starry Night", the texture of the brushstrokes is retained at the low level, and the swirling dynamic style is transmitted at the high level, generating a preliminary style transfer image.

[0098] Dynamic style intensity control:

[0099] Analyze the edge density and texture entropy of the original image. For a children's drawing with simple lines, its edge density is high and texture entropy is low, and the system will automatically reduce the style transfer intensity to retain the line features of the original image. For a watercolor painting with complex textures, the system will enhance the style transfer intensity to highlight the rendering effect of the watercolor;

[0100] Automatically adjust the style transfer intensity according to the edge density and texture entropy to generate an intermediate result image.

[0101] Real-time feedback optimization:

[0102] The style consistency of the intermediate result image is evaluated in real time through a discriminator (using a PatchGAN structure). The intermediate result image is divided into multiple 70×70 patches, and the discriminator discriminates the authenticity of each patch;

[0103] Dynamically adjust the cross-modal attention weights and hierarchical transfer network parameters according to the evaluation results. If the discriminator finds that there are differences between the style of the generated image and the target style, the system will backpropagate the error, adjust the network parameters, and regenerate the image until the discriminator can hardly distinguish between real and fake images, generating the final style transfer image;

[0104] Personalized creation support:

[0105] Custom style parameter input: The system receives custom style parameters input by the user, specifically including:

[0106] Color-related parameters: Color saturation (0%-100%), hue shift (-180° to +180°), contrast (0-200%);

[0107] Brushstroke-related parameters: Brushstroke intensity (1-10 levels, the larger the value, the more obvious the brushstroke), brushstroke size (1-50 pixels), brushstroke transparency (0%-100%);

[0108] Texture-related parameters: texture intensity (0%-100%, controlling the clarity of texture overlay), texture type (preset options such as oil painting texture, watercolor diffusion, sketch pencil marks, etc.);

[0109] Structure-related parameters: edge enhancement degree (0%-100%), geometric deformation coefficient (0-1, 0 being the original image structure, 1 being highly abstracted deformation);

[0110] Art style parameters: style mixing ratio (0%-100%, for example, 0% being the original image style and 100% being the target style), degree of abstraction (1-10 levels, the higher the value, the fewer details);

[0111] Interaction methods and real-time preview:

[0112] Users adjust parameters in the following ways:

[0113] Slider: used to continuously adjust numerical parameters, such as color saturation (slider range 0%-100%), brush size (1-50 pixels);

[0114] Drop-down menu: used to select enumerated parameters, such as texture type (oil painting, watercolor, sketch, etc.);

[0115] Knob control: used for angular parameters, such as hue shift (-180° to +180°);

[0116] Numeric input box: allows users to directly enter exact numerical values, such as contrast (0-200%);

[0117] Real-time preview mechanism:

[0118] The system adopts progressive visualization technology. The initial rendering progress is 0%. As the parameters are adjusted, the rendering details gradually increase. When the user adjusts the brush stroke intensity, the system first renders the contour lines of the image (20% progress), then fills in the colors (50% progress), and finally overlays the brush stroke texture (100% progress).

[0119] Users can view the rendering progress in real time and fine-tune the parameters according to the preview effect. When the user adjusts the color saturation from 50% to 80%, the preview window will immediately display an image with more vivid colors while retaining the brush strokes and texture details.

[0120] Example 2. Please refer to Appendix Figure 1 , Application of multi-modal deep learning in art creation assistance:

[0121] Multi-modal data preprocessing:

[0122] Visual Image Preprocessing: Obtain the visual image of the art work, and use the improved ResNet network for multi-scale feature extraction. For a Cubist painting "Les Demoiselles d'Avignon" by Picasso, the network will extract the global features of its geometric shape and the local block texture features, and generate visual feature vectors;

[0123] Text Description Preprocessing: Perform semantic encoding on the text description of the image. Input "Cubist style, emphasizing geometric structure and multi-perspective expression" to generate semantic feature vectors and capture the key information about the style in the text;

[0124] Metadata Preprocessing: Structurally process the metadata (including creation date, artist, genre, size, material, collection institution, creation background, style characteristics, theme, color tone, brushstroke type, etc.):

[0125] Creation Date: Convert "1907" into time features;

[0126] Artist: Convert "Picasso" into categorical features;

[0127] Genre: Convert "Cubism" into categorical features;

[0128] Size: Convert "243.9cm × 233.7cm" into numerical features;

[0129] Material: Convert "oil canvas, oil paint" into material type features;

[0130] Collection Institution: Convert "Museum of Modern Art, New York" into institutional labels;

[0131] Creation Background: Convert "Inspired by African masks and primitive art" into semantic features;

[0132] Style Characteristics: Convert "geometric blocks, multi-perspective splicing" into style keyword vectors;

[0133] Theme: Convert "female image, abstract expression" into theme labels;

[0134] Color Tone: Convert "tan, blue-gray" into color features;

[0135] Brushstroke Type: Convert "short straight lines, block filling" into brushstroke feature vectors.

[0136] Cross-modal Dynamic Fusion:

[0137] Respectively pass the visual, semantic, and metadata feature vectors through linear transformations to obtain the query matrix Q, the key matrix K, and the value matrix V;

[0138] Adopt a multi-head self-attention structure to calculate the attention weights between the features of each modality. When analyzing a Cubist painting with a "geometric structure" text description, the attention mechanism will enhance the weights of the channels related to geometric shapes in the visual features;

[0139] Dynamically adjust the importance of the features of each modality according to the attention weights to generate a fused feature vector;

[0140] Hierarchical style transfer:

[0141] Adopt a multi-scale feature pyramid network to construct a hierarchical transfer network. Starting from the high-level features of ResNet, fuse the high-level semantic features (the geometric structure of Cubism) with the low-level features (block texture) through upsampling operations;

[0142] Input the fused feature vector into the hierarchical transfer network, retain the details of the block texture at the low level, and transfer the geometric structure style of Cubism at the high level to generate a preliminary style transfer image;

[0143] Dynamic style intensity control:

[0144] Analyze the edge density and texture entropy of the original image. For a sketch with clear edges and simple texture, its edge density is relatively high and the texture entropy is relatively low. The system will automatically reduce the style transfer intensity to retain the lines and structure of the original image. For an oil painting with rich texture, the system will enhance the style transfer intensity to highlight the texture and color of the oil painting;

[0145] Automatically adjust the style transfer intensity according to the edge density and texture entropy to generate an intermediate result image;

[0146] Real-time feedback optimization:

[0147] Real-time evaluate the style consistency of the intermediate result image through a discriminator (adopting a PatchGAN structure). Divide the intermediate result image into multiple 70×70 patches, and the discriminator makes a true / false discrimination for each patch;

[0148] Dynamically adjust the cross-modal attention weights and the parameters of the hierarchical transfer network according to the evaluation results. If the discriminator finds that the geometric structure of the generated image is not prominent enough, the system will adjust the network parameters to regenerate the image until the discriminator believes that the generated image is consistent with the style of the real Cubist works;

[0149] Personalized creation support:

[0150] Custom style parameter input: The system receives the custom style parameters input by the user, specifically including:

[0151] Color-related parameters: color saturation (0%-100%), hue shift (-180° to +180°), contrast (0-200%);

[0152] Stroke-related parameters: stroke intensity (1-10 levels), stroke size (1-50 pixels), stroke transparency (0%-100%);

[0153] Texture-related parameters: texture intensity (0%-100%), texture type (oil painting texture, watercolor diffusion, sketch pencil marks);

[0154] Structure-related parameters: edge enhancement degree (0%-100%), geometric deformation coefficient (0-1);

[0155] Art style parameters: style mixing ratio (0%-100%), abstraction degree (1-10 levels).

[0156] Interaction method and real-time preview:

[0157] Users adjust parameters through sliders, drop-down menus, knob controls, etc. Use the slider to adjust color saturation and select the texture type through the drop-down menu;

[0158] The system adopts progressive visualization technology. The initial rendering progress is 0%. As the parameters are adjusted, the rendering details are gradually increased. When the user adjusts the stroke intensity, the system first renders the contour line of the image (20% progress), then fills in the color (50% progress), and finally overlays the stroke texture (100% progress);

[0159] Personalized model generation:

[0160] The system combines the user's historical creation data (including historical parameter settings and corresponding generation results), and uses the Gaussian process regression model to predict the generation result under the current input parameters. If the user has selected "high contrast (150%) + geometric deformation coefficient 0.8 + cubist style" multiple times in the historical data to generate works, the model will predict the generation result under the current input parameters based on these data;

[0161] User interaction and feedback:

[0162] Provide a friendly user interface, support artists to adjust style parameters in real time and preview the migration effect through interaction methods such as dragging and sliders. Artists can adjust the stroke size in real time and immediately see the changes in the strokes of the generated image;

[0163] Collect user feedback data (satisfaction ratings, modification suggestions) for continuous optimization of system performance. The system will record the artist's ratings of the generation results and optimize the model parameters based on the feedback data.

[0164] According to the content of the above embodiments, through multi-modal data fusion, hierarchical style transfer, and dynamic regulation mechanisms, high-precision transfer of complex artistic styles and personalized creation support are achieved. The multi-modal preprocessing integrates visual, text, and metadata features, breaking through the limitations of single-modal. The cross-modal attention mechanism deeply fuses multi-source information, enhancing the feature expression ability. The hierarchical transfer network fuses high-level semantics and low-level details through FPN, retaining artistic details such as brushstrokes and textures. The dynamic intensity control adaptively adjusts the transfer intensity based on the image complexity to ensure style consistency. The real-time feedback optimization iteratively optimizes the parameters through the GAN discriminator to improve the authenticity of the generated images. In addition, the system supports users to customize parameters and generate personalized style models in combination with historical data, providing real-time interaction and preview functions. Compared with the prior art, the present invention significantly improves the transfer accuracy of complex styles and the system adaptability while maintaining style consistency, and is applicable to fields such as art education and creative assistance, having significant technical advantages and application prospects.

[0165] Although the present invention is disclosed above in a preferred embodiment, it is not intended to limit the present invention. Any person skilled in the art can make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent changes, and decorations made to the above embodiments based on the technical essence of the present invention without departing from the technical solution of the present invention shall fall within the protection scope defined by the claims of the present invention.

Claims

1. A multimodal deep learning method for art work analysis, characterized in that, The method includes the following steps: Step 1, Multimodal data preprocessing: Obtain the visual image, text description, and metadata of the art work, perform multi-scale feature extraction on the visual image to generate a visual feature vector containing global and local features, perform semantic encoding on the text description to generate a semantic feature vector, and perform structured processing on the metadata to generate a metadata feature vector; Step 2, Cross-modal dynamic fusion: Construct a cross-modal attention mechanism, calculate the attention weights between the visual feature vector, semantic feature vector, and metadata feature vector, dynamically adjust the importance of each modal feature according to the attention weights, and generate a fused feature vector; Step 3, Hierarchical style transfer: Use a multi-scale feature pyramid network to construct a hierarchical transfer network, input the fused feature vector into the hierarchical transfer network, retain texture details at the low level and transfer the overall style at the high level to generate a preliminary style transfer image; Step 4, Dynamic style intensity control: Analyze the edge density and texture entropy of the original image, automatically adjust the style transfer intensity according to the edge density and texture entropy, and generate an intermediate result image; Step 5, Real-time feedback optimization: Use a discriminator to evaluate the style consistency of the intermediate result image in real time, dynamically adjust the cross-modal attention weights and hierarchical transfer network parameters according to the evaluation results, and generate the final style transfer image; Step 6, Personalized creation support: Receive user input custom style parameters, including custom style parameters such as color saturation and brush stroke intensity, combine with the user's historical creation data, generate a personalized style transfer model, and realize the personalized creation of art works; Step 7, User interaction and feedback: Provide a user interaction interface, support the user to adjust the style parameters in real time and preview the transfer effect, and collect user feedback data.

2. A multimodal deep learning method for art work analysis according to claim 1, characterized in that: In the above Step 1, the feature extraction of the visual image uses a ResNet network with an added attention channel, and adaptively adjusts the channel weights by calculating the correlation between channels. The calculation formula is as follows: Among them, a c represents the attention weight of channel c, C represents the total number of channels of the feature map, w1 represents the weight matrix of the first fully connected layer, w2 represents the weight matrix of the second fully connected layer for further linearly transforming the transformed features, δ represents the ReLU activation function, F c represents the feature map of channel c, and avg_pool represents the global average pooling operation.

3. A multimodal deep learning method for art work analysis according to claim 1, characterized in that: In the above Step 2, the cross-modal attention mechanism uses a multi-head self-attention structure. The specific process is as follows: The visual, text, and metadata feature vectors are linearly transformed through independent fully connected layers to generate a query matrix Q, a key matrix K, and a value matrix V. The transformation weight matrix of each modality is learnable. The query matrix Q is used to find the association between features, the key matrix K is used to represent feature identifiers, and the value matrix V is used to perform weighted combination of features according to the association relationship. The attention calculation is performed for each head, and the calculation formula is as follows: where, d k represents the dimension of the key vector divided by is to scale the dot product, calculate the dot product of Q and the transpose of K to obtain the similarity score between each query vector and the key vector, then transform the score into a probability distribution through the softmax function, and finally perform a weighted sum on the value matrix V according to this probability distribution to obtain the attention output of each head. The attention outputs of multiple heads are combined through a concatenation operation and then passed through a linear transformation to obtain the final cross-modal attention fusion result.

4. A multimodal deep learning method for art work analysis according to claim 1, characterized in that: In the above Step 3, the construction process of the multi-scale feature pyramid network (FPN) is as follows: Starting from the high-level features of the backbone network ResNet, gradually upsample the high-level semantic features to the same spatial size as the low-level features through upsampling operations, add the upsampled high-level features to the corresponding low-level features in the backbone network element by element, and fuse the high-level semantic features and low-level detail features through upsampling operations. The specific calculation formula is F up = upsample(F high ) + F low Among them, F up is the fused feature, F high is the high-level feature, F low is the low-level feature. Upsample is the upsampling operation. By performing style transfer on the FPN features at different levels respectively, hierarchical style transfer from coarse-grained to fine-grained is achieved. Finally, the migration results at each level are fused to generate a preliminary style transfer image.

5. A multimodal deep learning method for art work analysis according to claim 1, characterized in that: In the above Step 4, the edge density is calculated as follows: The original image is subjected to edge detection by the Canny edge detection algorithm. Assume the size of the original image is M×N. After being processed by the Canny edge detection algorithm, a binary edge image is obtained, where the edge pixel value is 1 and the non-edge pixel value is 0. The edge density ρ is defined as the ratio of the total number of edge pixels E to the total number of pixels M×N in the image. Its calculation formula is as follows: where, I ij represents the pixel value of the i-th row and j-th column in the binary edge image; Texture entropy calculation method: Calculate the texture entropy through the gray-level co-occurrence matrix. Let the gray-level co-occurrence matrix be P, and its element P ij represents the frequency of pixel pairs with gray values i and j occurring at a given distance and direction. The calculation formula for the texture entropy H is as follows: Among them, G is the gray level of the image. According to the calculated edge density and texture entropy, the style transfer intensity is automatically adjusted to generate an intermediate result image.

6. A multimodal deep learning method for art work analysis according to claim 1, characterized in that: In step 5, the discriminator adopts the Patch-GAN structure, and the specific evaluation method is as follows: First, the generated image is divided into multiple patches of size 70×70. Each patch is discriminated for authenticity. By minimizing the following loss function, the generator G generates a style transfer image close to the real data distribution: Among them, D is the discriminator, G is the generator, Pdata is the real data distribution, and Pz is the noise distribution. The gradient of the loss function with respect to the parameters is calculated by the backpropagation algorithm, and the training parameters of the discriminator and the generator are updated according to the chain rule.

7. A multimodal deep learning method for art work analysis according to claim 1, characterized in that: In step 6, the user's historical creation data will be collected. These data include the style parameters input by the user and the corresponding generation results. The Gaussian process regression model is used to associate the user's currently input custom style parameters with the historical data, and the generation result corresponding to the currently input style parameters is predicted through this model to realize the generation of a personalized style transfer model.

8. A multimodal deep learning method for analyzing artworks according to claim 1, characterized in that: In step 7, the user interface adopts progressive visualization technology. During the process of the user adjusting the style parameters, the details of the generated image will be gradually rendered. The initial rendering progress is 0%, and as the parameters are adjusted, the rendering progress will continuously increase until it reaches 100% to complete the rendering. The user can view the rendering progress in real time and adjust the style parameters according to the preview effect.

Images