Image aesthetic description and positioning method and system based on multi-modal attention

Abstract

The disclosure provides a kind of multi-modal attention-based image aesthetic description and positioning method and system, it is related to the technical field of aesthetic quality evaluation, method includes obtaining image, image comment data, image description data and various image data of description position information, and the image is preprocessed;Text information and image information in image data are input into neural network model, for text information, the high-level semantic vector feature of text is extracted;For image information, the deep semantic feature of image is extracted by convolutional neural network;Introduce attention fusion module, the high-level semantic vector feature is fused with deep semantic feature, generates image description and image region positioning.The combination of text features and image features is fully considered, and the subtle aesthetic changes can be effectively reflected.

Description

Technical Field

[0001] This disclosure relates to the field of aesthetic quality evaluation technology, specifically to a method and system for image aesthetic description and localization based on multimodal mutual attention. Background Technology

[0002] The statements in this section are merely background information relating to this disclosure and do not necessarily constitute prior art.

[0003] Aesthetic quality assessment involves fields such as computer vision, computational aesthetics, psychology, and neuroscience, and provides important references for image quality evaluation in certain scenarios (such as photography and post-editing). Traditional aesthetic quality assessment mostly focuses on providing quality ratings for images, failing to extend the task to generating descriptive text related to photographic aesthetics and / or photographic techniques.

[0004] When a human artist sees a photograph or painting, they don't simply assign a numerical score; they always describe the image's composition, lighting, color, focus, and many other aesthetic attributes. Existing methods for describing image aesthetics mostly output only a single sentence evaluation and fail to provide a comprehensive overview of these aesthetic attributes.

[0005] The inventors discovered that previous methods could only provide a description of an image, but could not show the correspondence between the description and the information within the image. Summary of the Invention

[0006] To address the aforementioned issues, this disclosure proposes an image aesthetic description and localization method and system based on multimodal mutual attention. By utilizing multimodal features based on mutual attention and combining them with neural networks to construct the relationship between image features and text features, a deep network is proposed. This method can obtain the image description and the location information of the described region simply by inputting an image, thus providing a better description of image information.

[0007] According to some embodiments, the present disclosure adopts the following technical solutions:

[0008] Image aesthetic description and localization methods based on multimodal mutual attention include:

[0009] Acquire various image data, including images, image comment data, image description data, and location information, and preprocess the images.

[0010] Text and image information from image data are input into a neural network model. For text information, high-level semantic vector features are extracted; for image information, deep semantic features are extracted through a convolutional neural network. An attention fusion module is introduced to fuse the high-level semantic vector features and deep semantic features to generate image descriptions and image region localization.

[0011] According to some embodiments, the present disclosure adopts the following technical solutions:

[0012] Image aesthetic description and localization systems based on multimodal mutual attention include:

[0013] The image data acquisition module is used to acquire various image data, including images, image comment data, image description data, and location information, and to preprocess the images.

[0014] The text learning module acquires text and image information from image data and inputs them into the neural network model. For the text information, it extracts high-level semantic vector features of the text.

[0015] The image learning module extracts deep semantic features from image information using a convolutional neural network.

[0016] The attention fusion output module is introduced to fuse the high-level semantic vector features with the deep semantic features to generate image descriptions and image region localization.

[0017] According to some embodiments, the present disclosure adopts the following technical solutions:

[0018] A non-transitory computer-readable storage medium is provided for storing computer instructions, which, when executed by a processor, implement the image aesthetic description and localization method based on multimodal mutual attention.

[0019] According to some embodiments, the present disclosure adopts the following technical solutions:

[0020] An electronic device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to implement the image aesthetic description and localization method based on multimodal mutual attention.

[0021] Compared with the prior art, the beneficial effects of this disclosure are as follows:

[0022] This disclosure proposes an aesthetic description and localization method based on multimodal mutual attention, which fully considers the combination of textual and image features and can effectively reflect subtle aesthetic changes. To enhance multimodal relevance, an attention fusion module is proposed, including a mutual attention model and a self-attention model to construct the interrelationship between textual and image features, resulting in a more comprehensive image representation. Finally, an image description and localization network is proposed to further enrich the image's content expression capabilities. Attached Figure Description

[0023] The accompanying drawings, which form part of this disclosure, are used to provide a further understanding of this disclosure. The illustrative embodiments of this disclosure and their descriptions are used to explain this disclosure and do not constitute an undue limitation of this disclosure.

[0024] Figure 1 This is a flowchart of the aesthetic description and localization method based on multimodal mutual attention provided in this disclosure.

[0025] Figure 2 This is a model framework diagram of the aesthetic description and localization method based on multimodal mutual attention provided in this disclosure. Detailed implementation method:

[0026] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0027] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0028] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0029] Terminology Explanation:

[0030] Co-Attn: Mutual Attention Module

[0031] Self-Attn: Self-Attention Module

[0032] Emd: Embedding operation

[0033] LN: Standardization based on layer dimension

[0034] Ep: Position Code

[0035] Trm: Transformer encoder

[0036] Tp: High-level semantic vector features

[0037] I: Deep semantic features

[0038] Multi-Head Attention: Multi-head attention layer

[0039] Add&Norm: Residual Calculation and Standardization Layer

[0040] Feed Forward: Fully Connected Layer

[0041] Example 1

[0042] One embodiment of this disclosure provides an image aesthetic description and localization method based on multimodal mutual attention, including:

[0043] Step 1: Acquire various image data, including images, image comment data, image description data, and location information, and preprocess the images;

[0044] Step 2: Input the text and image information from the image data into the neural network model. For the text information, extract the high-level semantic vector features of the text; for the image information, extract the deep semantic features of the image through a convolutional neural network.

[0045] Step 3: Introduce an attention fusion module to fuse the high-level semantic vector features with the deep semantic features to generate image descriptions and image region localization.

[0046] As one embodiment, acquiring various image data, including images, image comment data, image description data, and location information, specifically includes: photographic images, textual comments on photographic images, description tags for photographic images, and descriptions of location coordinate ranges. Specifically, it includes the following data:

[0047]

[0048] Where D represents the dataset, x i Represents the i-th image. This represents the image comment data for the i-th image. This represents the image description label for the i-th image. This represents the range of position coordinates corresponding to the description label of the i-th image.

[0049] The image is then preprocessed, including image normalization. The image normalization formula used is as follows:

[0050]

[0051] Where, x s This represents the result of image standardization, where x represents the original image (understood as a specific image in the dataset), μ represents the mean of the image, and ξ represents the standard deviation of the image.

[0052] As one embodiment, using the neural network model used in this disclosure for image description and localization first requires training the neural network model. The specific methods for constructing training data and the model are as follows:

[0053] The neural network model includes a text learning module, an image learning module, and an attention fusion module. The text learning module processes text information, extracting high-level semantic vector features. The network structure is designed with reference to the Transformer architecture, as shown below. Figure 2 The input is text data t p After the embedding operation Emb is performed, the positional code E is obtained by adding the position vector LN. p After passing through multiple layers of transformer encoder Trm, the high-level semantic vector features T of the text are obtained. p The structure of the transformer encoder TRM is as follows: Figure 2 As shown in the lower right corner, it includes a multi-head attention layer, an add&norm layer for residual calculation and normalization, and a fully connected feed forward layer.

[0054] The image learning module, targeting image information, needs to extract deep semantic features from images using a convolutional neural network. For this purpose, a convolutional neural network with VGG-16 as the backbone network is designed as follows: Figure 2 The input is the standardized image data x. s After passing through multiple convolutional layers (conv) and max pooling layers, the deep semantic features I of the image are output. The above adopts the standard VGG-16 network design structure, with a total of 13 layers of 3×3 convolutional operations and 4 max pooling layers. Figure 2 In the text, the part where the photo is input into the network to obtain feature I is the specific implementation details of VGG-16.

[0055] The attention fusion module, including mutual attention and self-attention models, needs to fuse the high-level semantic vector features output by the text learning module with the deep semantic features output by the image learning module to generate image descriptions and image region localization, such as... Figure 2 .like Figure 2In the right center, the Self-Attention module first processes the single input information into three features: query (Q), key (K), and value (V). For each query, the similarity is calculated with all keys. This similarity value is then passed through a Softmax layer to obtain a set of weights. The product of these weights and the corresponding value yields the value under self-attention. Multi-head attention operations process multiple queries in this way to compute multiple pieces of information selected from the input information in parallel. Finally, the value is added to the original input, and after standardization, the module output is obtained.

[0056] First, the input to this module is the high-level semantic vector features T of the text. p And the deep semantic features I of the image, firstly, a mutual attention model is designed as follows: Figure 2 Top right corner. (For example) Figure 2 In the upper right corner, the Co-Attention module first processes the input text information into key (K) and value (V) features, and the input image information into query (Q) features. For each query, a similarity score is calculated with all keys. This similarity score is then passed through a Softmax layer to obtain a set of weights. The sum of the products of these weights and their corresponding values ​​yields the value under self-attention. This value is then added to the original image input and standardized to obtain the module output. Specifically, the high-level semantic vector features of the text are used as the values ​​(V, value) and keys (K, key) in the attention mechanism, and the deep semantic features of the image are used as the queries (Q, query) in the attention mechanism. The output of the Co-Attention model is obtained through a multi-head self-attention layer (Multi-Head Attention) and residual calculation and standardization layers (Add&Norm). The output of the Co-Attention model is then processed by a self-attention model such as... Figure 2 In the right-middle section, specifically, the output of the mutual attention model serves as the value (V), key (K), and query (Q) in the attention mechanism. This output is then processed through a multi-head attention layer (Multi-Head Attention) and residual computation and a normalization layer (Add&Norm) to obtain the output of the self-attention model. Finally, the output of the attention fusion module includes image description and description location.

[0057] As one example, the training method for the neural network model is as follows:

[0058] The final neural network model for image description and localization includes a text learning module, an image learning module, and an attention fusion module. The parameters of the network in the text learning module and the convolutional neural network in the image learning module of the constructed model are transferred to the final neural network model for image description and localization, and then the overall model is trained.

[0059] The loss function selected is the design error loss function, as follows:

[0060] L = L T +L L

[0061] Where L is the total loss, L T For text similarity loss, cosine similarity is used, L L To determine the location loss, the Intersection over Union (IOU) loss is used.

[0062] The optimizer selected is the stochastic gradient descent optimizer, and in the final testing phase, the model output is denormalized to obtain the final image description and localization results.

[0063] As one example, such as Figure 2 As shown, the network input consists of two types of data: textual commentary on photos and images. First, the textual commentary on the photos is broken down word by word, and the resulting words and their positions within the sentences are then embedded separately (see...). Figure 2 The features obtained (from the middle Emd) are added together and then standardized according to the layer dimension (see Emd). Figure 2 The LN is used to obtain the final representation EP for each word. The EP is then fed into a bidirectional Transformer encoder (see LN). Figure 2 The deep bidirectional text representation (TP) is trained using conditional computation common to the context, and output TP is obtained. Simultaneously, image information is fed into the VGG-16 network to obtain image features I. The bidirectional text features TP and image features I are then fed into a multi-layered architecture combining mutual attention modules (Co-Attn) and self-attention modules (Self-Attn) to learn the relationships between cross-modal features. Finally, the output features are obtained, namely the image description and its corresponding location information within the image.

[0064] Among them, the bidirectional Transformer encoder (see...) Figure 2 The first half of the TRM (Self-Attention Module) operates identically to the Self-Attention Module (Self-Attention Module), while the second half adds fully connected layers, residual computation, and normalization layers. Since the input to each encoder is the features of all words, it can effectively integrate the contextual information of the sentence.

[0065] Example 2

[0066] One embodiment of this disclosure provides an image aesthetic description and localization system based on multimodal mutual attention, including:

[0067] The image data acquisition module is used to acquire various image data, including images, image comment data, image description data, and location information, and to preprocess the images.

[0068] The text learning module acquires text and image information from image data and inputs them into the neural network model. For the text information, it extracts high-level semantic vector features of the text.

[0069] The image learning module extracts deep semantic features from image information using a convolutional neural network.

[0070] The attention fusion output module is introduced to fuse the high-level semantic vector features with the deep semantic features to generate image descriptions and image region localization.

[0071] Example 3

[0072] One embodiment of this disclosure provides a non-transitory computer-readable storage medium for storing computer instructions, which, when executed by a processor, implement the image aesthetic description and localization method based on multimodal mutual attention.

[0073] Example 4

[0074] One embodiment of this disclosure provides an electronic device, including: a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to cause the electronic device to perform the image aesthetic description and localization based on multimodal mutual attention.

[0075] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0076] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0077] While the specific embodiments of this disclosure have been described above in conjunction with the accompanying drawings, this is not intended to limit the scope of protection of this disclosure. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art without creative effort based on the technical solutions of this disclosure are still within the scope of protection of this disclosure.

Claims

1. An image aesthetic description and localization method based on multimodal mutual attention, characterized in that, include: Acquire various image data, including images, image comment data, image description data, and location information, and preprocess the images. Text and image information from image data are input into a neural network model. For text information, high-level semantic vector features are extracted; for image information, deep semantic features are extracted through a convolutional neural network. An attention fusion module is introduced to fuse the high-level semantic vector features and deep semantic features to generate image descriptions and image region localization. The attention fusion module includes a mutual attention model and a self-attention model. When the high-level semantic vector features of the text and the deep semantic features of the image are input, the mutual attention model first uses the high-level semantic vector features of the text as the value and key in the attention mechanism, and the deep semantic features of the image as the query in the attention mechanism. After passing through a multi-head self-attention layer, residual calculation, and a normalization layer, the output of the mutual attention model is obtained. The output of the mutual attention model is used as the value, key, and query in the self-attention mechanism. After passing through a multi-head self-attention layer, residual calculation, and a normalization layer, the output of the self-attention model is obtained. Finally, the image description and description location are output.

2. The image aesthetic description and localization method based on multimodal mutual attention as described in claim 1, characterized in that, The various image data acquired, including image images, image comment data, image description data, and location information, specifically include: photographic images, textual comments on photographic images, description tags for photographic images, and descriptions of location coordinate ranges.

3. The image aesthetic description and localization method based on multimodal mutual attention as described in claim 1, characterized in that, The preprocessing includes standardizing the image.

4. The image aesthetic description and localization method based on multimodal mutual attention as described in claim 1, characterized in that, Text information from image data is input into a neural network model. The text information is then input into the Transformer structure of the neural network model. After embedding operations and adding position vectors, positional encoding is obtained. After multiple layers of Transformer encoding, high-level semantic vector features of the text are obtained.

5. The image aesthetic description and localization method based on multimodal mutual attention as described in claim 1, characterized in that, Image information from image data is input into a neural network model. Based on the image information, the deep semantic features of the image are output through multiple convolutional layers and max pooling layers of the convolutional neural network structure.

6. The image aesthetic description and localization method based on multimodal mutual attention as described in claim 1, characterized in that, When training the neural network model, the loss function is selected as the error loss function, the optimizer is selected as the stochastic gradient descent optimizer, and the output of the neural network model is denormalized during the training and testing phase.

7. An image aesthetic description and localization system based on multimodal mutual attention, characterized in that, include: The image data acquisition module is used to acquire various image data, including images, image comment data, image description data, and location information, and to preprocess the images. The text learning module acquires text and image information from image data and inputs them into the neural network model. For the text information, it extracts high-level semantic vector features of the text. The image learning module extracts deep semantic features from image information using a convolutional neural network. An attention fusion output module is introduced to fuse the high-level semantic vector features with the deep semantic features to generate image descriptions and image region localization. The attention fusion module includes a mutual attention model and a self-attention model. When the high-level semantic vector features of the text and the deep semantic features of the image are input, the mutual attention model first uses the high-level semantic vector features of the text as the value and key in the attention mechanism, and the deep semantic features of the image as the query in the attention mechanism. After passing through a multi-head self-attention layer, residual calculation, and a normalization layer, the output of the mutual attention model is obtained. The output of the mutual attention model is used as the value, key, and query in the self-attention mechanism. After passing through a multi-head self-attention layer, residual calculation, and a normalization layer, the output of the self-attention model is obtained. Finally, the image description and description location are output.

8. A non-transitory computer-readable storage medium, characterized in that, The non-transitory computer-readable storage medium is used to store computer instructions, which, when executed by a processor, implement the image aesthetic description and localization method based on multimodal mutual attention as described in any one of claims 1-6.

9. An electronic device, characterized in that, include: The device includes a processor, a memory, and a computer program; wherein the processor is connected to the memory, the computer program is stored in the memory, and when the electronic device is running, the processor executes the computer program stored in the memory to enable the electronic device to perform the image aesthetic description and localization method based on multimodal mutual attention as described in any one of claims 1-6.

Images