Image description method, apparatus, device, and medium
By fusing regional and grid features of the image and introducing textual adaptive features and historical words, the problem of insufficient information coverage and ambiguity in image description methods is solved, resulting in more accurate image descriptions.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2022-07-12
- Publication Date
- 2026-06-26
AI Technical Summary
Existing image description methods can only cover limited image information and the generated descriptions may be ambiguous.
By fusing regional and grid features of the input image, combining text adaptive features and preprocessed historical words, and using a memory self-attention layer and multi-head attention mechanism for feature adjustment and fusion, an accurate image description is finally generated.
It improves the coverage of image descriptions and reduces ambiguity, resulting in more accurate and coherent image descriptions.
Smart Images
Figure CN115186061B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of image description technology, including image description methods, apparatus, devices, and media. Background Technology
[0002] Image caption generation, as a comprehensive research direction combining natural language processing and computer vision, has made significant progress and has been widely applied. Current methods for image caption generation mainly fall into two categories: one is to incorporate various improved attention mechanisms into traditional encoder-decoder models, and the other is to improve the encoding and decoding processes.
[0003] Object detection and the Transformer model show promise for image caption generation. In object detection, bounding boxes can extract the features of the region where the object is located. However, the details extracted from these region features lack contextual information and cover limited image information. Furthermore, the connection between some generated descriptive words and the image is weak, potentially leading to ambiguity in the resulting image descriptions. Summary of the Invention
[0004] This application provides an image description method, apparatus, device, and storage medium, aiming to solve the problems that existing image description methods can only cover limited image information and that the resulting image descriptions may be ambiguous.
[0005] To solve the above problems, this application adopts the following technical solution:
[0006] This article provides image description methods, including:
[0007] Acquire an input image; encode the regional features of the input image to obtain regional feature codes;
[0008] The grid features of the input image are encoded to obtain the grid feature code;
[0009] The region features and the grid features are fused according to the region feature encoding and the grid feature encoding to obtain image features;
[0010] The image features and preprocessed historical words are fused based on the text adaptation features to obtain the image-text fusion result;
[0011] The image-text fusion results are classified to obtain image description results.
[0012] The step of fusing the region features and the grid features according to the region feature encoding and the grid feature encoding to obtain image features includes:
[0013] The region features and their encodings are input into a memory self-attention layer, and the weights of the region features are adjusted to obtain weight-adjusted region features.
[0014] The grid features and their encodings are input into the memory self-attention layer, and the grid features are weighted to obtain weighted grid features.
[0015] The weight adjustment region features are input into the feedforward neural network layer for correction to obtain the corrected region features.
[0016] The weighted grid features are input into the feedforward neural network layer for correction to obtain the corrected grid features.
[0017] The modified region features and the modified mesh features are interactively adjusted to obtain interactive region features and interactive mesh features;
[0018] The image features are obtained based on the interactive region features and the interactive grid features.
[0019] The step of interactively adjusting the modified region features and the modified mesh features to obtain interactive region features and interactive mesh features includes:
[0020] The interaction region features are obtained by adjusting the weights of the modified region features based on the modified grid features.
[0021] The interactive mesh features are obtained by adjusting the weights of the modified mesh features based on the modified region features.
[0022] The step of obtaining the image features based on the interaction region features and the interaction grid features includes:
[0023] The interaction region features are input into a feedforward neural network layer for correction to obtain the corrected interaction region features.
[0024] The interactive grid features are input into a feedforward neural network layer for correction to obtain the corrected interactive grid features.
[0025] The modified interaction region features and the interaction region features are input into the concatenation and regularization layer for concatenation and regularization to obtain regularized concatenated region features.
[0026] The modified grid region features and the grid region features are input into the splicing and regularization layer for splicing and regularization to obtain regularized spliced grid features;
[0027] The image features are obtained by combining the regularized stitched region features and the regularized stitched grid features.
[0028] Furthermore, before fusing the image features and preprocessed historical words based on text adaptation features, the method further includes:
[0029] The set of historical image description results is subjected to word encoding and position encoding to obtain a set of historical word vectors;
[0030] The historical word vector set is input into the mask self-attention layer, and the historical word vector set is filtered according to the time order to obtain the filtered word vector set.
[0031] The filtered word vector set and the historical word vector set are input into the concatenation and regularization layer for concatenation and regularization to obtain preprocessed historical words.
[0032] The process of classifying the image-text fusion result to obtain the image description result includes:
[0033] The image-text fusion result is used to extract features based on a multi-head attention mechanism to obtain image-text fusion features;
[0034] The image text fusion features are sequentially classified through the feedforward neural network layer, the splicing and regularization layer, and the normalized exponential function layer to obtain the image description result.
[0035] The feature extraction of the image-text fusion result based on the multi-head attention mechanism to obtain image-text fusion features includes:
[0036] The text features and image features in the image-text fusion result are extracted based on the multi-head attention mechanism.
[0037] The extracted text features and image features are fused together to obtain the image-text fusion features.
[0038] This application also provides an image description device, including:
[0039] The input image acquisition module is used to acquire the input image;
[0040] The region feature encoding module is used to encode the region features of the input image to obtain the region feature code;
[0041] The grid feature encoding module is used to encode the grid features of the input image to obtain the grid feature code;
[0042] The image feature extraction module is used to fuse the region features and the grid features according to the region feature encoding and the grid feature encoding to obtain image features;
[0043] The image-text fusion result calculation module is used to fuse the image features and preprocessed historical words according to the text adaptation features to obtain the image-text fusion result;
[0044] The description result generation module is used to classify the image-text fusion result to obtain the image description result.
[0045] This application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the image description method described in any of the above claims.
[0046] This application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the image description method described in any of the preceding claims.
[0047] The image description method of this application involves acquiring an input image; encoding the regional features of the input image to obtain regional feature codes; fusing the regional features and the grid features based on the regional feature codes and the grid features to obtain image features. By complementing the regional features and the grid features, the image features can cover more image content. The image features are then fused with preprocessed historical words based on text-adaptive features to obtain an image-text fusion result. The image-text fusion result is then classified to obtain an image description result. Guiding the fusion process of image features and preprocessed historical words with text-adaptive features makes the image-text fusion result more accurate, thereby reducing ambiguity in the image description result. Attached Figure Description
[0048] Figure 1 This is a flowchart illustrating an embodiment of an image description method;
[0049] Figure 2 This is a schematic diagram illustrating the process of fusing region features and grid features to obtain image features according to one embodiment.
[0050] Figure 3 This is a flowchart illustrating the interactive adjustment of modified region features and modified mesh features according to one embodiment.
[0051] Figure 4 This is a schematic diagram of the process for obtaining preprocessed historical words according to one embodiment;
[0052] Figure 5 This is a schematic block diagram of the structure of an image description device according to an embodiment;
[0053] Figure 6 This is a schematic block diagram of the structure of a computer device according to one embodiment.
[0054] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0055] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0056] Those skilled in the art will understand that, unless explicitly stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in the specification of this application means the presence of features, integers, steps, operations, elements, units, cells, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, units, cells, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or wireless couplings. The term “and / or” as used herein includes all or any of the units and all combinations thereof of one or more associated listed items.
[0057] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless specifically defined as herein.
[0058] Reference Figure 1 This is a flowchart illustrating the image description method applied for in this solution, including:
[0059] S1: Obtain the input image.
[0060] Using the MS COCO dataset, a certain number of images are randomly selected from the MS COCO dataset as an image set, and images are selected as input images according to the order of the images in the image set.
[0061] For example, 100 images are randomly selected from the MS COCO dataset, and sorted according to their order in the MS COCO dataset to obtain an image set. The first to the 100th images in the image set are then selected as input images.
[0062] S2: Encode the regional features of the input image to obtain the regional feature code.
[0063] Extract the regional features from the input image, encode the regional features, and obtain the regional feature code.
[0064] The Faster-RCNN object detection model is used to extract region features, and the bounding boxes of the region features are encoded to obtain the region feature codes.
[0065] Four coordinates are used for image features It represents the position information of the positive anchor point and the ground truth box. The four parameters represent the x-coordinate of the center point of the anchor point, the y-coordinate of the center point, the width and the height, respectively. The four parameters are learned by linear regression so that the positive anchor point can continuously approach the ground truth box and obtain the bounding box of the accurate regional features.
[0066] The region features of the input image are adjusted to a standard size using bilinear interpolation.
[0067] S3: Encode the grid features of the input image to obtain the grid feature code.
[0068] The grid features of the input image are encoded by the trigonometric function position encoding in the transformer model, resulting in grid feature encoding.
[0069] Image grid features can effectively compensate for the lack of contextual information and fine-grained details in region features.
[0070] S4: The region features and the grid features are fused according to the region feature encoding and the grid feature encoding to obtain image features.
[0071] The region features and their encodings are input into a memory self-attention layer, and the weights of the region features are adjusted to obtain weight-adjusted region features.
[0072] The grid features and their encodings are input into the memory self-attention layer, and the grid features are weighted to obtain weighted grid features.
[0073] The weight adjustment region features are input into the feedforward neural network layer for correction to obtain the corrected region features.
[0074] The weighted grid features are input into the feedforward neural network layer for correction to obtain the corrected grid features.
[0075] The modified region features and the modified mesh features are interactively adjusted to obtain interactive region features and interactive mesh features.
[0076] The interaction region features are input into a feedforward neural network layer for correction to obtain the corrected interaction region features.
[0077] The interactive grid features are input into a feedforward neural network layer for correction to obtain the corrected interactive grid features.
[0078] The modified interaction region features and the interaction region features are input into the concatenation and regularization layer for concatenation and regularization to obtain regularized concatenated region features.
[0079] The modified mesh region features and the mesh region features are input into the splicing and regularization layer for splicing and regularization to obtain regularized spliced mesh features.
[0080] The image features are obtained by combining the regularized stitched region features and the regularized stitched grid features.
[0081] The memory self-attention layer extends the encoding of prior knowledge, which helps generate image description results.
[0082] S5: The image features and preprocessed historical words are fused according to the text adaptation features to obtain the image-text fusion result.
[0083] Introducing text-adaptive features can make image-text fusion results more accurate, thereby reducing ambiguity in image descriptions.
[0084] Text-adaptive features contain guiding information for generating image descriptions, directing the process and ensuring that the descriptions don't solely depend on image features. Since the final image descriptions may contain content unrelated to image features—content that image features themselves cannot guide—text-adaptive features enhance the coherence of the descriptions. For example, a preposition connecting a subject and object can be used to create a description. All the image descriptions can then be combined into a single sentence.
[0085] The preprocessed historical words are obtained by integrating the image description results corresponding to the previous input images. By using the preprocessed historical words, the image-text fusion result takes into account the image description results corresponding to the previous input images. Referencing historical information can improve the accuracy of the image-text fusion result.
[0086] S6: Classify the image-text fusion results to obtain image description results.
[0087] Based on the multi-head attention mechanism, feature extraction is performed on the image-text fusion result to obtain image-text fusion features;
[0088] The image text fusion features are sequentially classified through the feedforward neural network layer, the splicing and regularization layer, and the normalized exponential function layer to obtain the image description result.
[0089] The image description is a single word.
[0090] The image description method of this application embodiment acquires an input image; encodes the regional features of the input image to obtain regional feature codes; fuses the regional features and the grid features according to the regional feature codes and the grid features codes to obtain image features. By complementing the regional features and the grid features, the image features can cover more image content. The image features are fused with preprocessed historical words according to text adaptation features to obtain an image-text fusion result. The image-text fusion result is classified to obtain an image description result. By using text adaptation features to guide the fusion process of image features and preprocessed historical words, the image-text fusion result can be made more accurate, thereby reducing ambiguity in the image description result.
[0091] Reference Figure 2 This is a flowchart illustrating the process of fusing region features and grid features to obtain image features according to this application, including the following steps:
[0092] S41: Input the region features and the region feature encoding into the memory self-attention layer, and adjust the weights of the region features to obtain weight-adjusted region features.
[0093] The memory-based self-attention layer adds extra memory slots to the classic attention function. Based on the region features or grid features of the input image, the extra memory slots are used to expand the derivation key and value matrices of the self-attention layer, encoding prior information and transforming the memory slots into ordinary learnable vectors.
[0094] The formula for calculating the memory self-attention layer is as follows:
[0095]
[0096] K = [(W k X+pos k ),M k ];
[0097] V = [(W v X+pos v ),M v ];
[0098] Where Attention is the classic attention function, M k To derive the key learning matrix, M v For the value learning matrix, Wq To query the weight matrix, W k To derive the bond weight matrix, W v Let K be the value weight matrix, [.,.] represent the merge operation, K be the derivation key matrix, V be the value matrix, and pos be the value weight matrix. k To derive the absolute position encoding of the key matrix, pos v Encoding the absolute position of the value matrix. This represents the i-th output of the memory self-attention layer.
[0099] This is a separate scaled dot product attention function.
[0100] When the value matrix V is a set of region features of the input image, the formula for calculating the value matrix is as follows:
[0101] V = {V i} NG ;
[0102] Among them, V i Let be the i-th regional feature, and the total number of regional features is NG.
[0103] When the value matrix V is a set of grid features of the input image, the formula for calculating the value matrix is as follows:
[0104] V = {V i} NR ;
[0105] Among them, V i Let be the i-th grid feature, and let NR be the total number of grid features.
[0106] Output all regional features of the memory self-attention layer Combining these elements, we obtain the set of regional feature outputs H. R ,in, Output the first region feature. Output the features of the NG-th region.
[0107] Output all grid features of the self-attention layer. Combining these elements, we obtain the grid feature output set H. G ,in, Output the first grid feature. This is the output of the NRth grid feature.
[0108] A total of N weight adjustments need to be made to the regional features and grid features respectively.
[0109] The formula for weighting regional features is as follows:
[0110]
[0111] MHCA stands for Multi-Head Self-Attention Mechanism. This is the set of region features in the l-th layer after attention weight adjustment. Let RPE be the set of region feature outputs for layer l, and Ω be the region feature encoding. rr This is the relative position matrix of regional features.
[0112] The multi-head self-attention mechanism combines multiple scaled dot-product attention functions output from the memory self-attention layer to obtain a combined attention function. This combined attention function is then used to adjust the weights of regional features, resulting in weighted regional features.
[0113] S42: Input the grid features and the grid feature encoding into the memory self-attention layer, and adjust the weights of the grid features to obtain weight-adjusted grid features.
[0114] The weights of grid features are adjusted through a multi-head self-attention mechanism using a memory self-attention layer.
[0115] The formula for weight adjustment of grid features is as follows:
[0116]
[0117] MHCA stands for Multi-Head Self-Attention Mechanism. This is the set of grid features in the l-th layer after attention weight adjustment. Let GPE be the set of grid feature outputs for layer l, where GPE is the grid feature encoding, and Ω is the grid feature output set for layer l. gg This is the relative position matrix of the grid features.
[0118] The multi-head self-attention mechanism combines multiple scaled dot-product attention functions output from the memory self-attention layer to obtain a combined attention function. This combined attention function is then used to adjust the weights of the grid features, resulting in grid-adjusted region features.
[0119] S43: Input the weight adjustment region features into the feedforward neural network layer for correction to obtain the corrected region features.
[0120] The formula for obtaining the corrected region features by inputting the weight adjustment region features into the feedforward neural network layer is as follows:
[0121]
[0122] Where FNN represents a feedforward neural network layer, For the weighted adjustment region features, To correct regional features.
[0123] S44: Input the weighted grid features into the feedforward neural network layer for correction to obtain the corrected grid features.
[0124] The formula for obtaining the corrected grid features by inputting the weighted grid features into the feedforward neural network layer is as follows:
[0125]
[0126] Where FNN represents a feedforward neural network layer, For weighted grid features, To correct the mesh features.
[0127] S45: The modified region features and the modified mesh features are interactively adjusted to obtain interactive region features and interactive mesh features.
[0128] Adding the modified region features to the modified mesh features to supplement contextual information results in the interactive region features. The modified mesh features are added to the modified region features to obtain high-level object information, resulting in interactive mesh features.
[0129] S46: The interactive region features are input into the feedforward neural network layer for correction to obtain the corrected interactive region features; the interactive grid features are input into the feedforward neural network layer for correction to obtain the corrected interactive grid features.
[0130] After supplementing the modified region features with modified grid features, the interactive region features are obtained. These interactive region features are then input into a feedforward network layer, which undergoes two linear transformations and is connected via the GELU activation function to obtain the modified interactive region features. The formula for the modified interactive region features is as follows:
[0131]
[0132] In this layer, FC1 is the first fully connected layer, FC2 is the second fully connected layer, GELU represents the GELU activation function, and Dropout is a temporary dropout layer. Features of the interactive area To correct the characteristics of the interactive area.
[0133] Temporary dropout layers reduce overfitting by randomly and temporarily dropping neurons with a fixed probability.
[0134] After supplementing the corrected mesh features with the corrected region features, the interactive mesh features are obtained. These interactive mesh features are then input into a feedforward network layer, which undergoes two linear transformations and is connected via the GELU activation function to obtain the corrected interactive mesh features. The formula for the corrected interactive mesh features is as follows:
[0135]
[0136] In this layer, FC1 is the first fully connected layer, FC2 is the second fully connected layer, GELU represents the GELU activation function, and Dropout is a temporary dropout layer. For interactive mesh features, To correct the interactive mesh features.
[0137] S47: Input the modified interaction region features and the interaction region features into the concatenation and regularization layer for concatenation and regularization to obtain regularized concatenation region features.
[0138] Different modified interactive region features are concatenated, and the concatenated modified interactive region features are normalized to obtain regularized concatenated region features with a mean of 0 and a variance of 1.
[0139] S48: Input the modified grid region features and the grid region features into the splicing and regularization layer for splicing and regularization to obtain regularized spliced grid features.
[0140] Different modified interactive grid features are concatenated, and the concatenated modified interactive grid features are normalized to obtain regularized concatenated grid features with a mean of 0 and a variance of 1.
[0141] S49: Combine the regularized stitching region features and the regularized stitching grid features to obtain the image features.
[0142] Image features are formed by combining regularized stitched region features and regularized stitched grid features using a connection function.
[0143] This embodiment of the application fuses region features and grid features to obtain image features. The region features and their encodings are input into a memory self-attention layer, and weights are adjusted to obtain weighted region features. Grid features and their encodings are input into the memory self-attention layer, and weights are adjusted to obtain weighted grid features. The weighted region features are then input into a feedforward neural network layer for further correction, resulting in corrected region features; the weighted grid features are also input into a feedforward neural network layer for further correction, resulting in corrected grid features. The corrected region features and corrected grid features are then interactively adjusted to obtain interactive region features and interactive grid features. The interactive region features are then input into a feedforward neural network layer to obtain corrected interactive region features; the interactive grid features are also input into a feedforward neural network layer to obtain corrected interactive grid features. The corrected interactive region features and interactive region features are then input into a concatenation and regularization layer to obtain regularized concatenated region features. The corrected grid region features and grid region features are then input into a concatenation and regularization layer to obtain regularized concatenated grid features. Finally, the regularized concatenated region features and regularized concatenated grid features are combined to obtain image features. The first optimization involves adjusting the weights of region features and grid features using a self-attention layer. A second optimization, using region features to adjust the grid feature weights, is then performed. This two-stage optimization results in an image with a greater number of features obtained by fusing region and grid features.
[0144] Reference Figure 3 This is a flowchart illustrating the interactive adjustment process of the modified region features and modified mesh features in this application.
[0145] S451: Adjust the weights of the modified region features according to the modified mesh features to obtain the interactive region features;
[0146] The formula for calculating the interactive region features is as follows:
[0147]
[0148] MHRGCA is the modified region feature weight adjustment function, which means that the modified region features obtained after attention weight adjustment are combined with the modified mesh features for weight adjustment. To correct regional features, To correct grid features, RPE is region feature encoding, GPE is grid feature encoding, and Ω... rg This is the relative position matrix between region features and grid features. Features of the interactive area.
[0149] The relative position matrix of region features and grid features can be determined by the positional variations between the bounding boxes of the region features and the grid features. Since region features are defined by bounding boxes containing four parameters, and grid features can also be considered special region features, they are defined by bounding boxes with equal length and width. Each grid cell in the grid feature can also be represented by coordinates. This allows for a unified representation of region feature bounding boxes and grid feature bounding boxes.
[0150] The formula for the relative positional information between the region feature bounding box and the grid feature bounding box is as follows:
[0151]
[0152] Where i is the number of the region feature bounding box, j is the number of the grid feature bounding box, and x i The x-coordinate of the center point of the region feature bounding box is y. i w is the ordinate of the center point of the region feature bounding box. i h is the width of the region feature bounding box. i x represents the height of the region feature bounding box. j y is the x-coordinate of the center point of the grid feature bounding box. j y is the ordinate of the center point of the grid feature bounding box. j h is the width of the bounding box of the mesh feature. j Ω(i,j) represents the height of the grid feature bounding box, and Ω(i,j) represents the relative position information between the i-th region feature bounding box and the j-th grid feature bounding box.
[0153] By traversing the relative position information of each region feature bounding box and each grid feature bounding box, the relative position matrix of region features and grid features can be obtained.
[0154] S452: Adjust the weights of the modified mesh features according to the modified region features to obtain the interactive mesh features.
[0155] The formula for calculating interactive mesh features is as follows:
[0156]
[0157] MHGRCA is a modified mesh feature adjustment function, which means that the modified mesh features obtained after attention weight adjustment are combined with the modified region features for weight adjustment. To correct the mesh features, To correct region features, GPE is the grid feature code, RPE is the region feature code, and Ω... gr This is the relative position matrix between grid features and region features. This is an interactive grid feature.
[0158] This application embodiment interactively adjusts the modified region features and modified mesh features. The modified region features are weighted according to the modified mesh features to obtain the interactive region features. Similarly, the modified mesh features are weighted according to the modified region features to obtain the interactive mesh features. The modified region features are the region features after the first weight adjustment, and the modified mesh features are the mesh features after the first weight adjustment. By fusing the modified region features and modified mesh features, and performing a second weight adjustment on the region features and mesh features, the connection between the region features and mesh features can be better established, making them complementary to each other.
[0159] Reference Figure 4 This is a flowchart illustrating the process of obtaining preprocessed historical words in this application.
[0160] Steps S2-S4 are included before step S5.
[0161] S2”: Perform word encoding and position encoding on the historical image description result set to obtain a historical word vector set.
[0162] Since the generation of image description results is based on all previous image description results, each image description result is represented by one word. The set of previously generated image description results is defined as Y = [y1, y2, y3, ..., y...]. t ,y t+1 ,...,y T ], where y1 is the first word in the historical image description result set, y T This is the last word in the historical image description result set. To ensure that the image description results and the region features and grid features of the input image are on the same dimension for easier subsequent calculations, each image description result in the historical image description result set is represented as a historical word vector through word encoding and position encoding. For example, the t-th image description result y is... t Represented as historical word vector F t ∈R d×t , and the historical word vector F t Encode as input.
[0163] For the m-th encoding, it is also necessary to train the text adaptive feature vector. Where m > 1.
[0164] S3”: Input the set of historical word vectors into the mask self-attention layer, and filter the set of historical word vectors according to the time order to obtain the filtered set of word vectors.
[0165] Since the historical word vector set is input once, the sequence information of the output at all times can be obtained. However, in practical applications, only the sequence information of the output before the current time can be seen. Therefore, a masked self-attention layer is introduced to process the input historical word vector.
[0166] The input mask self-attention layer contains the historical image description results for all time periods. However, during the generation of the image description result at time t, only the historical image description results generated up to time t-1 are visible; those generated at time t+1 and later are not. Therefore, the mask self-attention layer is used to filter the historical image description result set, resulting in a filtered set of word vectors Y' = [y1, y2, y3, ..., y t ].
[0167] The masked self-attention layer is used to filter historical word vectors according to time order.
[0168] S4”: Input the filtered word vector set and the historical word vector set into the concatenation and regularization layer for concatenation and regularization to obtain preprocessed historical words.
[0169] The concatenation and regularization layer concatenates and regularizes the filtered word vector set and the historical word vector set to obtain preprocessed historical words with a mean of 0 and a variance of 1.
[0170] This embodiment of the application obtains preprocessed historical words by performing word encoding and position encoding on the historical image description result set to obtain a historical word vector set. The historical word vector set is input into a masked self-attention layer, and the historical word vector set is filtered according to chronological order to obtain a filtered word vector set. The filtered word vector set and the historical word vector set are input into a concatenation and regularization layer for concatenation and regularization to obtain the preprocessed historical words. By using a masked self-attention layer to filter the historical word vector set, retaining historical word vectors that meet the chronological conditions, the historical image description result set can be utilized more reasonably, resulting in more accurate preprocessed historical words.
[0171] Reference Figure 5 This is a schematic block diagram of an image description device according to this application. The device includes:
[0172] Input image acquisition module 10 is used to acquire input images;
[0173] The region feature encoding module 20 is used to encode the region features of the input image to obtain the region feature code;
[0174] The grid feature encoding module 30 is used to encode the grid features of the input image to obtain grid feature codes;
[0175] Image feature extraction module 40 is used to fuse the region features and the grid features according to the region feature code and the grid feature code to obtain image features;
[0176] The image-text fusion result calculation module 50 is used to fuse the image features and preprocessed historical words according to the text adaptation features to obtain the image-text fusion result;
[0177] The description result generation module 60 is used to classify the image-text fusion result to obtain the image description result.
[0178] The image description apparatus of this application embodiment is used to implement the image description method.
[0179] Reference Figure 6 This application also provides a computer device, which may be a server, and its internal structure may be as follows: Figure 6 As shown. This computer device includes a processor, memory, network interface, and database connected via a system bus. The processor in this computer design provides computing and control capabilities. The memory of this computer device includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of this computer device is used to store area feature codes and grid feature codes, etc. The network interface of this computer device is used for communication with external terminals via network connection. When the computer program is executed by the processor, it implements an image description method.
[0180] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer equipment on which the present application is applied.
[0181] One embodiment of this application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements an image description method. It is understood that the computer-readable storage medium in this embodiment can be a volatile readable storage medium or a non-volatile readable storage medium.
[0182] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media provided in this application and in the embodiments may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual-speed SDRAM (SSRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0183] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, apparatus, article, or method. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, apparatus, article, or method that includes that element.
[0184] The above description is only a preferred embodiment of this application and does not limit the patent scope of this application. Any equivalent structural or procedural changes made based on the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.
Claims
1. An image description method, characterized in that, include: Acquire an input image; encode the regional features of the input image to obtain regional feature codes; The grid features of the input image are encoded to obtain the grid feature code; The region features and the grid features are fused according to the region feature encoding and the grid feature encoding to obtain image features; The image features and preprocessed historical words are fused based on the text adaptation features to obtain the image-text fusion result; The image-text fusion results are classified to obtain image description results; The step of fusing the region features and the grid features according to the region feature encoding and the grid feature encoding to obtain image features includes: The region features and their encodings are input into a memory self-attention layer, and the weights of the region features are adjusted to obtain weight-adjusted region features. The grid features and their encodings are input into the memory self-attention layer, and the grid features are weighted to obtain weighted grid features. The weight adjustment region features are input into the feedforward neural network layer for correction to obtain the corrected region features. The weighted grid features are input into the feedforward neural network layer for correction to obtain the corrected grid features. The modified region features and the modified mesh features are interactively adjusted to obtain interactive region features and interactive mesh features; The image features are obtained based on the interactive region features and the interactive grid features.
2. The image description method according to claim 1, characterized in that, The step of interactively adjusting the modified region features and the modified mesh features to obtain interactive region features and interactive mesh features includes: The interaction region features are obtained by adjusting the weights of the modified region features based on the modified grid features. The interactive mesh features are obtained by adjusting the weights of the modified mesh features based on the modified region features.
3. The image description method according to claim 2, characterized in that, The step of obtaining the image features based on the interaction region features and the interaction grid features includes: The interaction region features are input into a feedforward neural network layer for correction to obtain the corrected interaction region features. The interactive grid features are input into a feedforward neural network layer for correction to obtain the corrected interactive grid features. The modified interaction region features and the interaction region features are input into the concatenation and regularization layer for concatenation and regularization to obtain regularized concatenated region features. The modified interactive mesh features and the interactive mesh features are input into the splicing and regularization layer for splicing and regularization to obtain regularized spliced mesh features; The image features are obtained by combining the regularized stitched region features and the regularized stitched grid features.
4. The image description method according to claim 3, characterized in that, Before fusing the image features and preprocessed historical words based on text adaptation features, the method further includes: The set of historical image description results is subjected to word encoding and position encoding to obtain a set of historical word vectors; The historical word vector set is input into the mask self-attention layer, and the historical word vector set is filtered according to the time order to obtain the filtered word vector set. The filtered word vector set and the historical word vector set are input into the concatenation and regularization layer for concatenation and regularization to obtain preprocessed historical words.
5. The image description method according to claim 3, characterized in that, The process of classifying the image-text fusion result to obtain the image description result includes: The image-text fusion result is used to extract features based on a multi-head attention mechanism to obtain image-text fusion features; The image text fusion features are sequentially classified through the feedforward neural network layer, the splicing and regularization layer, and the normalized exponential function layer to obtain the image description result.
6. The image description method according to claim 5, characterized in that, The feature extraction of the image-text fusion result based on the multi-head attention mechanism to obtain image-text fusion features includes: The text features and image features in the image-text fusion result are extracted based on the multi-head attention mechanism. The extracted text features and image features are fused together to obtain the image-text fusion features.
7. An image description apparatus for implementing the method according to any one of claims 1-6, characterized in that, include: The input image acquisition module is used to acquire the input image; The region feature encoding module is used to encode the region features of the input image to obtain the region feature code; The grid feature encoding module is used to encode the grid features of the input image to obtain the grid feature code; An image feature extraction module is used to fuse the region features and the grid features according to the region feature encoding and the grid feature encoding to obtain image features; The image-text fusion result calculation module is used to fuse the image features and preprocessed historical words according to the text adaptation features to obtain the image-text fusion result; The description result generation module is used to classify the image-text fusion result to obtain the image description result.
8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the image description method according to any one of claims 1 to 6.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the image description method according to any one of claims 1 to 6.