Image description generation method and apparatus
By using multimodal, multi-granularity feature fusion and joint embedding semantic space model, the problem of inaccurate image description generation is solved, and more accurate and multi-layered text description generation is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- INDUSTRIAL AND COMMERCIAL BANK OF CHINA
- Filing Date
- 2023-06-07
- Publication Date
- 2026-07-14
AI Technical Summary
Existing image description generation schemes do not take into account enough information, resulting in inaccurate image descriptions.
A multimodal, multi-granularity feature fusion method is adopted to extract data from different modalities in the test image and perform multi-level granularity division. The multimodal joint embedding semantic space model is used for feature extraction and similarity calculation to generate accurate image descriptions.
The generated image descriptions are more accurate, capable of generating corresponding levels of text descriptions based on images of different granularities, thus improving the accuracy and wide applicability of image semantic descriptions.
Smart Images
Figure CN116740713B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of artificial intelligence technology, and in particular to an image description generation method and apparatus. Background Technology
[0002] This section is intended to provide background or context for the embodiments of the invention set forth in the claims. The description herein is not an admission that it is prior art simply because it is included in this section.
[0003] In recent years, with the development of the internet and the improvement of storage media performance, multimedia (such as images, audio, video, and text) has ushered in a period of rapid development. Different forms of media data are also referred to as different modalities. Among them, cross-media solutions combining images and text have attracted the attention of scholars in the fields of natural language processing and machine vision. One of the mainstream tasks is image description generation, or "image captioning," which uses a sentence or a paragraph of text to describe the semantic information in an image, realizing the conversion from image to natural language. For the task of automatic image-to-text generation, humans can easily understand the content of images and express it in different forms of natural language according to specific requirements. However, for computers, it is necessary to comprehensively utilize the achievements of computer vision and natural language processing, two popular fields of deep learning. Therefore, if computers can perform high-level and complex semantic annotation of images, it is very meaningful work. This can further improve the completeness of image semantic annotation, containing richer information than simple keyword and label annotation.
[0004] For application scenarios in bank branches, frame-by-frame image analysis can be performed on the video obtained from surveillance. This patent enables text annotation of surveillance images, such as describing the identity, clothing, actions, and status of specific people in the image, as well as objective objects in the environment, such as cash, passbooks, bank cards, and gold bars, forming a description of the relationship between people, objects, and scenes, thereby achieving the purpose of diverting potential customer business scenarios or identifying key customers.
[0005] Over the past two decades, natural language processing and computer vision have made tremendous progress in generating text and understanding images and videos. Historically, the two fields developed separately. Combining natural language processing and computer vision techniques to form new algorithms and models, and applying them to image description generation, has gradually become a hot topic in industrial applications.
[0006] From a computer vision perspective, text descriptions are not limited to the main entities in an image; they involve scene features or how people and objects within the scene interact. More challenging is that text descriptions can even involve inference information, providing high-level semantic information directly from the image. In short, good image descriptions require a comprehensive understanding of the image; therefore, text description generation is more comprehensive for the field of computer vision than general object detection.
[0007] From a natural language processing perspective, generating descriptions is a natural language generation problem, which involves converting non-verbal representations into human-readable text. In image description, the input is an image, and the natural language generation model must transform it into words, sentences, and even paragraphs. Therefore, this task not only requires recognizing entities in the image but also possessing textual logic to organize words of different natures into understandable paragraph descriptions.
[0008] Current image description generation schemes do not take into account enough information, resulting in inaccurate image descriptions. Summary of the Invention
[0009] This invention provides an image description generation method that considers multimodal and multi-granularity feature fusion, enabling the generation of accurate image descriptions. The method includes:
[0010] Extract data from different modalities in the test image;
[0011] The data for each modality are divided into multiple levels of granularity.
[0012] Feature extraction is performed on data at each level of granularity for each modality;
[0013] The extracted features are input into a multimodal joint embedding semantic space model to obtain multiple probabilistic features;
[0014] The probability features are compared with multiple predefined sets of text description data to calculate similarity, and the preset number of text description data with the highest similarity is output to the user.
[0015] This invention also provides an image description generation apparatus that considers multimodal and multi-granularity feature fusion, enabling the generation of accurate image descriptions. The apparatus includes:
[0016] The modal data extraction module is used to extract data of different modalities from the test image;
[0017] The multi-level granularity partitioning module is used to partition the data of each modality into multi-level granularities.
[0018] The feature extraction module is used to extract features from data at each level of granularity for each modality;
[0019] The probabilistic feature calculation module is used to input the extracted features into the multimodal joint embedding semantic space model to obtain multiple probabilistic features;
[0020] The text description data generation module is used to calculate the similarity between the probability features and multiple predefined sets of text description data, and output the preset number of text description data with the highest similarity to the user.
[0021] This invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described image description generation method.
[0022] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image description generation method.
[0023] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described image description generation method.
[0024] In this embodiment of the invention, data from different modalities are extracted from the test image; multi-level granularity is performed on the data of each modality; features are extracted from the data at each level of granularity for each modality; the extracted features are input into a multimodal joint embedding semantic space model to obtain multiple probabilistic features; the similarity of the probabilistic features with multiple predefined sets of text description data is calculated, and the preset number of text description data with the highest similarity is output to the user. In the above process, data from multiple modalities is fully considered, and multi-level granularity is performed, utilizing the complementarity of multi-level information to compensate for the lack of information at different granularities in previous methods. The above process innovatively proposes a multimodal joint embedding semantic space model, which emphasizes key correlations between data, bridging the heterogeneity gap of multimodal data, resulting in more accurate text description data for the generated images, and can generate corresponding text description data at different levels of granularity when the input test image is an image with different levels of granularity. Attached Figure Description
[0025] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. In the drawings:
[0026] Figure 1This is a flowchart of the image description generation method in an embodiment of the present invention;
[0027] Figure 2 This is an example of performing multi-level granularity division and feature extraction on the data of each modality in an embodiment of the present invention;
[0028] Figure 3 This is a schematic diagram of the multimodal joint embedding semantic space model in an embodiment of the present invention;
[0029] Figure 4 This is a schematic diagram illustrating the principle of training a multimodal joint embedding semantic space model in an embodiment of the present invention;
[0030] Figure 5 In this embodiment of the invention, the image corresponds to the document and the bounding box corresponds to the word.
[0031] Figure 6 This is a schematic diagram of the image description generation device in an embodiment of the present invention;
[0032] Figure 7 This is a schematic diagram of a computer device in an embodiment of the present invention. Detailed Implementation
[0033] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings. Here, the illustrative embodiments of the present invention and their descriptions are used to explain the present invention, but are not intended to limit the present invention.
[0034] Figure 1 The flowchart of the image description generation method in this embodiment of the invention includes:
[0035] Step 101: Extract data from different modalities in the test image;
[0036] Step 102: Perform multi-level granularity division on the data for each modality;
[0037] Step 103: Extract features from the data at each level of granularity for each modality;
[0038] Step 104: Input the extracted features into the multimodal joint embedding semantic space model to obtain multiple probabilistic features;
[0039] Step 105: Calculate the similarity between the probability feature and multiple predefined sets of text description data, and output the preset number of text description data with the highest similarity to the user.
[0040] The embodiments of this invention fully consider data from multiple modalities and perform multi-level granularity division, that is, utilize the complementarity of multi-level information to make up for the defects of previous methods in the lack of information at different granularities; the above process innovatively proposes a multi-modal joint embedding semantic space model, which emphasizes the key correlation information between data, bridges the heterogeneity gap of multi-modal data, and generates more accurate text description data for images.
[0041] Each step is described in detail below.
[0042] In step 101, data of different modalities are extracted from the test image; the modalities include image modalities and / or text modalities, the data of image modalities can be referred to as image data, and the data of text modalities can be referred to as text data.
[0043] In step 102, the data for each modality is divided into multiple levels of granularity. The multi-level granularity for the image modality includes one or any combination of images, image patches, and bounding boxes; the multi-level granularity for the text modality includes one or any combination of documents, sentences, and words. This multi-level granularity division is performed to better mine high-level semantic information, ensuring a reasonable cognitive correspondence between data of the same granularity levels from two modalities, and facilitating a more comprehensive acquisition of information from a single modality.
[0044] In one embodiment, the data for each modality is divided into multiple levels of granularity, including:
[0045] All image modal data are treated as images;
[0046] The image modality data is divided into blocks to obtain image patches.
[0047] Bounding boxes are annotated on the image modal data to obtain the bounding boxes;
[0048] Treat all text modal data as documents;
[0049] Perform sentence segmentation on the text modal data to obtain sentences;
[0050] Perform word segmentation on all sentences to obtain words.
[0051] In step 103, feature extraction is performed on the data at each level of granularity for each modality; in one embodiment, feature extraction on the data at each level of granularity for each modality includes:
[0052] Feature extraction of images and image patches was performed using the VGG16 neural network.
[0053] Use Faster-RCNN to extract bounding box features;
[0054] Use BERT for feature extraction of documents and sentences;
[0055] Use dynamic word vector extraction methods to extract word features.
[0056] Figure 2 This is an example of performing multi-level granular division and feature extraction on the data of each modality in an embodiment of the present invention. The meaning of bounding boxes, image blocks, images, documents, sentences, and words can be clearly seen.
[0057] In step 104, the extracted features are input into the multimodal joint embedding semantic space model to obtain multiple probabilistic features; in one embodiment, the multimodal joint embedding semantic space model includes: a sequentially connected text semantic space, a first joint embedding layer, a similarity fusion layer, a second joint embedding layer, and an image semantic space.
[0058] In one embodiment, the text semantic space includes a sequentially connected fully connected layer, a bidirectional GRU layer, and a Softmax layer;
[0059] The image semantic space includes a sequentially connected fully connected layer, a bidirectional GRU layer, and a Softmax layer.
[0060] Figure 3 This is a schematic diagram of the multimodal joint embedding semantic space model in this embodiment of the invention. The multimodal joint embedding semantic space model selects an appropriate deep neural network model as the basis for the text semantic space and the image semantic space. This can preserve the positional or sequence relationships within a single level of granularity and effectively fuse data at different levels of granularity. Secondly, based on this, independent image semantic space and text semantic space are constructed. Finally, cross-modal data embedding is performed through the semantic space established by this model, laying the foundation for the automatic generation of image semantic descriptions.
[0061] See Figure 3 In this embodiment of the invention, similarity calculation is performed using multimodal data joint embedding based on an attention mechanism. The principle is as follows: First, multi-layered granular data from the text modality is mapped to the image semantic space, and multi-layered granular data from the image modality is mapped to the text semantic space, completing cross-modal joint embedding through data inter-mapping. Second, an attention mechanism is applied to the joint embedding to quickly obtain the data that needs to be focused on in the semantic space and optimize the joint embedding. Finally, based on the joint embedding of the two semantic spaces, they are adaptively fused at multiple granularities, and the similarity between the heterogeneous data is calculated.
[0062] In this embodiment of the invention, joint embedding refers to mapping data from different modalities to the same space.
[0063] Figure 4This is a schematic diagram illustrating the principle of training a multimodal joint embedding semantic space model in an embodiment of the present invention. Both the first and second joint embedding layers are joint embeddings based on an attention mechanism. Figure 4 The similarity fusion layer illustrates two similarity fusion methods: adaptive similarity fusion based on image-document similarity, and adaptive similarity fusion based on bounding box-word similarity.
[0064] In one embodiment, the training steps of the multimodal joint embedding semantic space model are as follows:
[0065] Input the features at each level of granularity of the text modality into the text semantic space to obtain the probabilistic features corresponding to the text modality;
[0066] The probability features corresponding to the text modality and the features at each level of granularity of the image modality are input into the first joint embedding layer to obtain the joint embedding value corresponding to each feature of the image modality.
[0067] Input the features at each level of granularity of the image modality into the image semantic space to obtain the probabilistic features corresponding to the image modality;
[0068] The probability features corresponding to the image modality and the features at each level of granularity of the text modality are input into the second joint embedding layer to obtain the joint embedding value corresponding to each feature of the text modality.
[0069] The joint embedding values corresponding to each feature of the image modality and the joint embedding values corresponding to each feature of the text modality are respectively input into the similarity fusion layer to obtain the image similarity fusion matrix and the text similarity fusion matrix;
[0070] If the eigenvalues of the image similarity fusion matrix do not reach the first threshold, or the eigenvalues of the text similarity fusion matrix do not reach the second threshold, the probability features corresponding to the text modality are used as features at each level of granularity of the text modality, and the probability features corresponding to the image modality are used as features at each level of granularity of the image modality. The above steps are repeated until the eigenvalues of the image similarity fusion matrix reach the first threshold and the eigenvalues of the text similarity fusion matrix reach the second threshold. The trained multimodal joint embedding semantic space model is then output.
[0071] In one embodiment, the first joint embedding layer and the second joint embedding layer are based on an attention mechanism;
[0072] The first joint embedding layer includes a first inner product layer and a first summation layer. The first inner product layer is used to calculate the inner product of the feature at each level of granularity of each image modality with the feature at each level of granularity of the text modality in turn to obtain multiple inner product values corresponding to the feature. The first summation layer is used to calculate all the inner product values of the feature at each level of granularity of each image modality to obtain the joint embedding value of the feature.
[0073] The second joint embedding layer includes a second inner product layer and a second summation layer. The second inner product layer is used to calculate the inner product of the feature at each level of granularity of each text modality with the feature at each level of granularity of the image modality in turn to obtain multiple inner product values corresponding to the feature. The second summation layer is used to calculate all the inner product values of the feature at each level of granularity of each text modality to obtain the joint embedding value of the feature.
[0074] Figure 4 In the first joint embedding layer, each image modality has two features at each level of granularity. These are just examples; normally, features at each level of granularity of all image modalities need to be input. The second joint embedding layer works similarly. Additionally, the probability features output by the Softmax layer are obtained by multiplying the features calculated by Softmax by their probabilities bitwise. If there are eight features at each level of granularity for the text modality, then after passing through the text semantic space, eight probability features are output. Then, after entering the first joint embedding layer, the first inner product layer performs inner product calculations on each feature at each level of granularity for each image modality, obtaining multiple inner product values corresponding to each feature at each level of granularity for each image modality (here, eight inner product values). The first summing layer calculates all the inner product values (here, eight) for each feature at each level of granularity for each image modality, obtaining the joint embedding value of that feature. In other words, each feature at each level of granularity for an image modality has one joint embedding value, so the features corresponding to the eight image modalities have eight joint embedding values. Similarly, the features corresponding to the 8 text modalities have 8 joint embedding values, so the image similarity fusion matrix and the text similarity fusion matrix are both 8×8 matrices, only the rows and columns are reversed.
[0075] In one embodiment, the extracted features are input into a multimodal joint embedding semantic space model to obtain multiple probabilistic features, including:
[0076] Input the features at each level of granularity of the text modality into the text semantic space to obtain the probabilistic features corresponding to the text modality;
[0077] By inputting the features at each level of granularity of the image modality into the image semantic space, the probabilistic features corresponding to the image modality are obtained.
[0078] In step 105, the probability feature is compared with multiple predefined sets of text description data to calculate the similarity, and the preset number of text description data with the highest similarity is output to the user.
[0079] In practice, the predefined multiple sets of text description data are open-source training sets that can be directly applied. After similarity calculation, the similarity is sorted from high to low, and the top few sets of text with the highest similarity are returned as the generated text description output to the user, thus realizing the automatic generation of text descriptions from images.
[0080] In addition, in this embodiment of the invention, fine-grained image patches or bounding boxes can be input, and after obtaining probabilistic features, similarity calculations can be performed with sentences or words in the training set. The similarity scores can be sorted to obtain multiple hierarchical text description data.
[0081] Figure 5 In this embodiment of the invention, the document corresponding to the image and the word corresponding to the bounding box represent multiple levels of text description data.
[0082] In summary, the method proposed in the embodiments of the present invention has the following beneficial effects:
[0083] First, a multimodal, multi-level granular data partitioning strategy is designed, using cutting-edge natural language processing and computer vision granular segmentation and feature extraction techniques (including neural networks VGG16, Faster-RCNN, BERT, and dynamic word vector extraction methods). This strategy integrates a multimodal joint embedding semantic space model for feature extraction fine-tuning. By leveraging the complementarity of multi-level granular data, the shortcomings of previous methods in missing data at different granularities are overcome. This approach preserves global information while capturing local details, effectively fusing global and local features and significantly improving the accuracy of image semantic description.
[0084] Second, attention mechanisms are used for cross-modal joint embedding and adaptive fusion metric learning. Based on the obtained joint embedding space, similarity fusion with dynamic weights across semantic spaces is performed, emphasizing key information related to data and bridging the heterogeneous gap of multimodal data.
[0085] Third, to enable "image description" applications, multi-level text descriptions can be generated as needed, such as generating paragraph text descriptions for complete images, sentence descriptions of the relationship between people and objects in specific image regions, and text labels for image blocks, thereby improving the model's versatility.
[0086] This invention also provides an image description generation apparatus, as described in the following embodiments. Since the principle by which this apparatus solves the problem is similar to that of the image description generation method, the implementation of this apparatus can be referred to the implementation of the image description generation method, and repeated details will not be elaborated further.
[0087] Figure 6 This is a schematic diagram of an image description generation device in an embodiment of the present invention, comprising:
[0088] Modal data extraction module 601 is used to extract data of different modalities from test images;
[0089] The multi-level granularity partitioning module 602 is used to perform multi-level granularity partitioning on the data of each modality.
[0090] The feature extraction module 603 is used to extract features from data at each level of granularity for each modality;
[0091] The probabilistic feature calculation module 604 is used to input the extracted features into the multimodal joint embedding semantic space model to obtain multiple probabilistic features;
[0092] The text description data generation module 605 is used to calculate the similarity between the probability features and multiple predefined sets of text description data, and output the preset number of text description data with the highest similarity to the user.
[0093] In one embodiment, the modality includes an image modality and / or a text modality;
[0094] The multi-level granularity of image modalities includes one or any combination of images, image patches, and bounding boxes;
[0095] The multi-level granularity of text modalities includes one or any combination of documents, sentences, and words.
[0096] In one embodiment, the multi-level granularity division module is specifically used for:
[0097] All image modal data are treated as images;
[0098] The image modality data is divided into blocks to obtain image patches.
[0099] Bounding boxes are annotated on the image modal data to obtain the bounding boxes;
[0100] Treat all text modal data as documents;
[0101] Perform sentence segmentation on the text modal data to obtain sentences;
[0102] Perform word segmentation on all sentences to obtain words.
[0103] In one embodiment, the feature extraction module is specifically used for:
[0104] Feature extraction of images and image patches was performed using the VGG16 neural network.
[0105] Use Faster-RCNN to extract bounding box features;
[0106] Use BERT for feature extraction of documents and sentences;
[0107] Use dynamic word vector extraction methods to extract word features.
[0108] In one embodiment, the multimodal joint embedding semantic space model includes: a sequentially connected text semantic space, a first joint embedding layer, a similarity fusion layer, a second joint embedding layer, and an image semantic space.
[0109] In one embodiment, the text semantic space includes a sequentially connected fully connected layer, a bidirectional GRU layer, and a Softmax layer;
[0110] The image semantic space includes a sequentially connected fully connected layer, a bidirectional GRU layer, and a Softmax layer.
[0111] In one embodiment, the training steps of the multimodal joint embedding semantic space model are as follows:
[0112] Input the features at each level of granularity of the text modality into the text semantic space to obtain the probabilistic features corresponding to the text modality;
[0113] The probability features corresponding to the text modality and the features at each level of granularity of the image modality are input into the first joint embedding layer to obtain the joint embedding value corresponding to each feature of the image modality.
[0114] Input the features at each level of granularity of the image modality into the image semantic space to obtain the probabilistic features corresponding to the image modality;
[0115] The probability features corresponding to the image modality and the features at each level of granularity of the text modality are input into the second joint embedding layer to obtain the joint embedding value corresponding to each feature of the text modality.
[0116] The joint embedding values corresponding to each feature of the image modality and the joint embedding values corresponding to each feature of the text modality are respectively input into the similarity fusion layer to obtain the image similarity fusion matrix and the text similarity fusion matrix;
[0117] If the eigenvalues of the image similarity fusion matrix do not reach the first threshold, or the eigenvalues of the text similarity fusion matrix do not reach the second threshold, the probability features corresponding to the text modality are used as features at each level of granularity of the text modality, and the probability features corresponding to the image modality are used as features at each level of granularity of the image modality. The above steps are repeated until the eigenvalues of the image similarity fusion matrix reach the first threshold and the eigenvalues of the text similarity fusion matrix reach the second threshold. The trained multimodal joint embedding semantic space model is then output.
[0118] In one embodiment, the first joint embedding layer and the second joint embedding layer are based on an attention mechanism;
[0119] The first joint embedding layer includes a first inner product layer and a first summation layer. The first inner product layer is used to calculate the inner product of the feature at each level of granularity of each image modality with the feature at each level of granularity of the text modality in turn to obtain multiple inner product values corresponding to the feature. The first summation layer is used to calculate all the inner product values of the feature at each level of granularity of each image modality to obtain the joint embedding value of the feature.
[0120] The second joint embedding layer includes a second inner product layer and a second summation layer. The second inner product layer is used to calculate the inner product of the feature at each level of granularity of each text modality with the feature at each level of granularity of the image modality in turn to obtain multiple inner product values corresponding to the feature. The second summation layer is used to calculate all the inner product values of the feature at each level of granularity of each text modality to obtain the joint embedding value of the feature.
[0121] In one embodiment, the probability feature calculation module is specifically used for:
[0122] Input the features at each level of granularity of the text modality into the text semantic space to obtain the probabilistic features corresponding to the text modality;
[0123] By inputting the features at each level of granularity of the image modality into the image semantic space, the probabilistic features corresponding to the image modality are obtained.
[0124] In summary, the device proposed in the embodiments of the present invention has the following beneficial effects:
[0125] First, a multimodal, multi-level granular data partitioning strategy is designed, using cutting-edge natural language processing and computer vision granular segmentation and feature extraction techniques (including neural networks VGG16, Faster-RCNN, BERT, and dynamic word vector extraction methods). This strategy integrates a multimodal joint embedding semantic space model for feature extraction fine-tuning. By leveraging the complementarity of multi-level granular data, the shortcomings of previous methods in missing data at different granularities are overcome. This approach preserves global information while capturing local details, effectively fusing global and local features and significantly improving the accuracy of image semantic description.
[0126] Second, attention mechanisms are used for cross-modal joint embedding and adaptive fusion metric learning. Based on the obtained joint embedding space, similarity fusion with dynamic weights across semantic spaces is performed, emphasizing key information related to data and bridging the heterogeneous gap of multimodal data.
[0127] Third, to enable "image description" applications, multi-level text descriptions can be generated as needed, such as generating paragraph text descriptions for complete images, sentence descriptions of the relationship between people and objects in specific image regions, and text labels for image blocks, thereby improving the model's versatility.
[0128] This invention also provides a computer device. Figure 7 This is a schematic diagram of a computer device in an embodiment of the present invention. The computer device 700 includes a memory 710, a processor 720, and a computer program 730 stored in the memory 710 and executable on the processor 720. When the processor 720 executes the computer program 730, it implements the above-described image description generation method.
[0129] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described image description generation method.
[0130] This invention also provides a computer program product, which includes a computer program that, when executed by a processor, implements the above-described image description generation method.
[0131] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0132] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0133] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0134] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0135] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above descriptions are merely specific embodiments of the present invention and are not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.
Claims
1. An image description generation method, characterized in that, include: Extract data from different modalities in the test image; The data for each modality are divided into multiple levels of granularity. Feature extraction is performed on data at each level of granularity for each modality; The extracted features are input into a multimodal joint embedding semantic space model to obtain multiple probabilistic features. The multimodal joint embedding semantic space model includes: a sequentially connected text semantic space, a first joint embedding layer, a similarity fusion layer, a second joint embedding layer, and an image semantic space. The probability features are compared with multiple predefined sets of text description data to calculate similarity, and the preset number of text description data with the highest similarity is output to the user. The training steps for the multimodal joint embedding semantic space model are as follows: Features at each granularity level of the text modality are input into the text semantic space to obtain the probabilistic features corresponding to the text modality. The probabilistic features corresponding to the text modality and the features at each granularity level of the image modality are input into the first joint embedding layer to obtain the joint embedding value corresponding to each feature of the image modality. The features at each granularity level of the image modality are input into the image semantic space to obtain the probabilistic features corresponding to the image modality. The probabilistic features corresponding to the image modality and the features at each granularity level of the text modality are input into the second joint embedding layer to obtain the joint embedding value corresponding to each feature of the text modality. The joint embedding values corresponding to each feature of the image modality and the text modality are then respectively... The joint embedding value corresponding to each feature is input into the similarity fusion layer to obtain the image similarity fusion matrix and the text similarity fusion matrix. If the feature value of the image similarity fusion matrix does not reach the first threshold, or the feature value of the text similarity fusion matrix does not reach the second threshold, the probability feature corresponding to the text modality is used as the feature at each level of granularity of the text modality, and the probability feature corresponding to the image modality is used as the feature at each level of granularity of the image modality. The above steps are repeated until the feature value of the image similarity fusion matrix reaches the first threshold and the feature value of the text similarity fusion matrix reaches the second threshold. The trained multimodal joint embedding semantic space model is then output.
2. The method as described in claim 1, characterized in that, The modalities include image modalities and / or text modalities; The multi-level granularity of image modalities includes one or any combination of images, image patches, and bounding boxes; The multi-level granularity of text modalities includes one or any combination of documents, sentences, and words.
3. The method as described in claim 2, characterized in that, The data for each modality is divided into multiple levels of granularity, including: All image modal data are treated as images; The image modality data is divided into blocks to obtain image patches. Bounding boxes are annotated on the image modal data to obtain the bounding boxes; Treat all text modal data as documents; Perform sentence segmentation on the text modal data to obtain sentences; Perform word segmentation on all sentences to obtain words.
4. The method as described in claim 2, characterized in that, Feature extraction is performed on data at each level of granularity for each modality, including: Feature extraction of images and image patches was performed using the VGG16 neural network. Use Faster-RCNN to extract bounding box features; Use BERT for feature extraction of documents and sentences; Use dynamic word vector extraction methods to extract word features.
5. The method as described in claim 1, characterized in that, The text semantic space includes a sequentially connected fully connected layer, a bidirectional GRU layer, and a Softmax layer; The image semantic space includes a sequentially connected fully connected layer, a bidirectional GRU layer, and a Softmax layer.
6. The method as described in claim 1, characterized in that, The first and second joint embedding layers are based on an attention mechanism; The first joint embedding layer includes a first inner product layer and a first summation layer. The first inner product layer is used to calculate the inner product of the feature at each level of granularity of each image modality with the feature at each level of granularity of the text modality in turn to obtain multiple inner product values corresponding to the feature. The first summation layer is used to calculate all the inner product values of the feature at each level of granularity of each image modality to obtain the joint embedding value of the feature. The second joint embedding layer includes a second inner product layer and a second summation layer. The second inner product layer is used to calculate the inner product of the feature at each level of granularity of each text modality with the feature at each level of granularity of the image modality in turn to obtain multiple inner product values corresponding to the feature. The second summation layer is used to calculate all the inner product values of the feature at each level of granularity of each text modality to obtain the joint embedding value of the feature.
7. The method as described in claim 6, characterized in that, The extracted features are input into a multimodal joint embedding semantic space model to obtain multiple probabilistic features, including: Input the features at each level of granularity of the text modality into the text semantic space to obtain the probabilistic features corresponding to the text modality; By inputting the features at each level of granularity of the image modality into the image semantic space, the probabilistic features corresponding to the image modality are obtained.
8. An image description generation apparatus, characterized in that, include: The modal data extraction module is used to extract data of different modalities from the test image; The multi-level granularity partitioning module is used to partition the data of each modality into multi-level granularities. The feature extraction module is used to extract features from data at each level of granularity for each modality; The probability feature calculation module is used to input the extracted features into the multimodal joint embedding semantic space model to obtain multiple probability features. The multimodal joint embedding semantic space model includes: a sequentially connected text semantic space, a first joint embedding layer, a similarity fusion layer, a second joint embedding layer, and an image semantic space. The text description data generation module is used to calculate the similarity between the probability features and multiple predefined sets of text description data, and output the preset number of text description data with the highest similarity to the user. The training steps for the multimodal joint embedding semantic space model are as follows: Features at each granularity level of the text modality are input into the text semantic space to obtain the probabilistic features corresponding to the text modality. The probabilistic features corresponding to the text modality and the features at each granularity level of the image modality are input into the first joint embedding layer to obtain the joint embedding value corresponding to each feature of the image modality. The features at each granularity level of the image modality are input into the image semantic space to obtain the probabilistic features corresponding to the image modality. The probabilistic features corresponding to the image modality and the features at each granularity level of the text modality are input into the second joint embedding layer to obtain the joint embedding value corresponding to each feature of the text modality. The joint embedding values corresponding to each feature of the image modality and the text modality are then respectively... The joint embedding value corresponding to each feature is input into the similarity fusion layer to obtain the image similarity fusion matrix and the text similarity fusion matrix. If the feature value of the image similarity fusion matrix does not reach the first threshold, or the feature value of the text similarity fusion matrix does not reach the second threshold, the probability feature corresponding to the text modality is used as the feature at each level of granularity of the text modality, and the probability feature corresponding to the image modality is used as the feature at each level of granularity of the image modality. The above steps are repeated until the feature value of the image similarity fusion matrix reaches the first threshold and the feature value of the text similarity fusion matrix reaches the second threshold. The trained multimodal joint embedding semantic space model is then output.
9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.
11. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the method of any one of claims 1 to 7.