Controllable image description methods, systems, devices, and media based on multimodal fusion

Abstract

This invention provides a controllable image description method, system, device, and medium based on multimodal fusion. The method includes the following steps: based on the original image features X... ν * and object regression box features X ν b The images are fused to generate sample image features X. ν Extract each word from the labeled language text, and label and positionally encode the extracted words to generate sample text description features X. w Generate a trajectory regression box based on the mouse trajectory segment, and generate sample trajectory features X based on the trajectory regression box. r Using a pre-built TO-Transformer feature fusion model, the sample image features X are processed. ν The sample text description feature X w and the sample trajectory feature X r Feature fusion is performed to generate sample fusion features; based on the sample fusion features, a controllable image description model is generated using a pre-built output model, thereby effectively improving the controllability and efficiency of image description.

Description

Technical Field

[0001] This invention relates to the field of image description technology, and more specifically, to a controllable image description method, system, device, and medium based on multimodal fusion. Background Technology

[0002] The data processing flow of image description models typically involves taking image features as input and outputting natural language text corresponding to those features. Since the input to an image description model is only image features, it can only output text descriptions based on pre-trained weights and the input image. This text description is generated by the image description model in a targeted manner, and its output often does not meet human expectations (an image generates a text description that may not be what the user wants). Therefore, the output of an image description model often differs from the information a user receives from seeing the image.

[0003] To enhance the ability of image description models to capture key semantics of images, the main methods currently used are to improve the model's image feature processing algorithms or enhance its natural language output capabilities. However, these methods only improve the model's algorithms without considering other sources of information from the input, resulting in inaccurate model outputs.

[0004] While some existing image description methods can enhance their ability to capture image semantics through model improvements, the underlying algorithmic flow of these models remains essentially the same: inputting an image and outputting corresponding natural language. The information source for these image description models is solely the image, lacking other information sources. Because the input to these models lacks semantic information describing the image, inaccurate image descriptions result from this information scarcity. Consequently, these models cannot verify whether the output natural language contains important information from the image, thus hindering their ability to capture key semantic meanings and leading to low-quality image descriptions.

[0005] Furthermore, image description models typically employ generative models, taking image features as input and outputting natural language text. Inspired by machine translation, most image description models generally use an encoder-decoder framework, where the input image is encoded into an image feature vector, and then a descriptive text sequence is generated by a decoder. To improve the model and make image descriptions more accurate, a common approach is to make improvements in computer vision, such as by refining the structure of convolutional neural networks and object detection networks to help improve the image description model.

[0006] On the one hand, many earlier image description models typically processed image features extracted using CNNs, resulting in a single feature vector representing the entire image information. This feature vector, obtained by extracting the entire image, often lacks semantic information, and the image description fails to focus on local targets within the image. Consequently, the output natural language text lacks information about target objects in specific locations within the image. To address this issue, Anderson et al. proposed the UpDown model, combining a top-down attention model with a bottom-up LSTM (Long Short-Term Memory) approach, based on a Faster-RCNN object detection-based image description model. The bottom-up attention model operates on mean-pooled convolutional features obtained from the region of interest (ROI) of the Faster-RCNN object detection network, using these features instead of overall image features.

[0007] However, while the aforementioned methods utilize image features extracted by object detection networks, these features do not include object location information, resulting in a lack of description of object positions in the image description output. To address this issue, Herdade et al. proposed the Object Relation Transformer (ORT) model, using the region of interest (ROI) of the object detection network as bounding boxes. The coordinates and size of the bounding boxes, along with the spatial location of the objects, are used to infer the positional relationships between objects. The closer the coordinates of the bounding boxes, the stronger their relationship. However, this method only utilizes the coordinate information of objects in the image, and the quality of the image description remains relatively low.

[0008] On the other hand, many early image description models used LSTM (Long Short-Term Memory) as their text decoder. Similar to early image feature processing, although LSTM can generate text describing images, its output often fails to represent the key points of the image and is prone to missing certain content in the image. Common measures to improve the model are: (1) Using an attention mechanism, which usually generates an image map to locate the specific image region most relevant to the currently predicted word. In this way, the decoder cannot calculate the relationship or degree of correlation between the input vector and the attention query, which will cause the decoder to produce incorrect results. (2) Using the AOA (Attention on Attention) module, another attention is added by performing element-wise multiplication on the attention to obtain the final "attention information". By using this attention, the relationship between the input vector and the attention query is increased, making the output structure more accurate. However, this method increases the computational cost of the image description model by adding additional attention and does not bridge the gap between the image and text modalities.

[0009] With the development of natural language processing algorithms, the best language generation model currently available is the Transformer architecture. Transformer replaces LSTM with a fully attention-based structure, significantly improving results for various natural language tasks (such as translation, text generation, and language understanding). In existing image description models, applying the Transformer framework to image description tasks involves inputting image feature vectors as a sequence into the Transformer encoder, and then outputting natural language text through the Transformer decoder. However, this approach does not consider the low-level and high-level relationships between regions and language.

[0010] In order to solve the above problems, people have been seeking an ideal technological solution. Summary of the Invention

[0011] The purpose of this invention is to address the shortcomings of existing technologies by providing a controllable image description method, system, device, and medium based on multimodal fusion.

[0012] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0013] The first aspect of this invention provides a controllable image description method based on multimodal fusion, the method comprising the following steps:

[0014] Obtain sample images from the sample dataset, and use ResNet-101 to extract image features from the sample images to generate the original image features X. ν * The sample images are processed using a Faster R-CNN-based object detection model to generate object regression boxes, and object regression box features X are generated based on these object regression boxes. ν b ; for the original image feature X ν * and the object regression box feature X ν b The images are fused to generate sample image features X. ν ;

[0015] Obtain the labeled language text associated with the sample image from the sample dataset, extract each word from the labeled language text, and label and positionally encode the extracted words to generate sample text description features X. w ;

[0016] Obtain mouse trajectories associated with the sample image from the sample dataset, segment the mouse trajectories according to time sequence to obtain multiple mouse trajectory segments, generate trajectory regression boxes based on the mouse trajectory segments, and generate sample trajectory features X based on the trajectory regression boxes. r ;

[0017] Using a pre-built TO-Transformer feature fusion model, the sample image features X are processed. ν The sample text description feature X w and the sample trajectory feature X r Feature fusion is performed to generate sample fusion features; wherein, the TO-Transformer feature fusion model is used to respectively integrate the sample image features X ν The sample text description feature X w The sample trajectory feature X r The data is processed into serialized data, and then the feature vector positions are aligned.

[0018] Based on the sample fusion features, a controllable image description model is generated using a pre-built output model.

[0019] A second aspect of the present invention provides a controllable image description system based on multimodal fusion, comprising:

[0020] The image feature generation module is used to acquire sample images from the sample dataset, extract image features from the sample images using ResNet-101, and generate original image features X. ν * The sample images are processed using a Faster R-CNN-based object detection model to generate object regression boxes, and object regression box features X are generated based on these object regression boxes. ν b ; for the original image feature X ν * and the object regression box feature X ν b The images are fused to generate sample image features X. ν ;

[0021] The text description feature generation module is used to obtain the annotated language text associated with the sample image in the sample dataset, extract each word from the annotated language text, and mark and positionally encode the extracted words to generate sample text description features X. w ;

[0022] The trajectory feature generation module is used to acquire mouse trajectories associated with the sample image in the sample dataset, segment the mouse trajectories according to time sequence to obtain multiple mouse trajectory segments, generate trajectory regression boxes based on the mouse trajectory segments, and generate sample trajectory features X based on the trajectory regression boxes. r ;

[0023] The feature fusion module is used to perform feature fusion on the sample image features X using a pre-built TO-Transformer feature fusion model. ν The sample text description feature X w and the sample trajectory feature X r Feature fusion is performed to generate sample fusion features; wherein, the TO-Transformer feature fusion model is used to respectively integrate the sample image features X ν The sample text description feature X w The sample trajectory feature X r The data is processed into serialized data, and then the feature vector positions are aligned.

[0024] And an image description model generation module, which is used to generate a controllable image description model based on the sample fusion features and using a pre-built output model.

[0025] A third aspect of the present invention provides a controllable image description device based on multimodal fusion, comprising a memory, a processor, and a controllable image description program based on multimodal fusion stored in the memory and executable on the processor. When the controllable image description program based on multimodal fusion is executed by the processor, it implements the steps of the controllable image description method based on multimodal fusion as described above.

[0026] A fourth aspect of the present invention provides a readable storage medium having instructions stored thereon, which, when executed by a processor, implement the steps of the controllable image description method based on multimodal fusion as described above.

[0027] This invention has outstanding substantive features and significant progress compared to the prior art, specifically:

[0028] 1) This invention proposes a controllable image description method and system based on multimodal fusion, which preprocesses sample images in a sample dataset to generate sample image features X. ν The annotated language text of the sample images is preprocessed to generate sample text description features X. w The mouse trajectory of the sample image is preprocessed to generate sample trajectory features X. rUsing a pre-built TO-Transformer feature fusion model, these feature vectors are fused to generate sample fusion features; based on the sample fusion features, a controllable image description model is generated using a pre-built output model, thereby effectively improving the controllability and efficiency of image description.

[0029] 2) The mouse trajectory in this invention is embedded in the form of a truncated trajectory regression box, which can avoid some noise (useless points) from participating in the calculation, greatly reducing the amount of calculation when describing controllable images, and effectively bridging the gap between image and text modalities.

[0030] 3) The pre-constructed TO-Transformer feature fusion model in this invention uses a feature vector position alignment method to perform feature fusion on the sample image feature X. ν The sample text description feature X w and the sample trajectory feature X r Feature fusion is performed to make the feature fusion more complete, thereby improving the accuracy of the controllable image description model;

[0031] 4) This invention incorporates mouse trajectory into the image description model, making the subsequent image description results controllable and ensuring that the output image description results are more closely aligned with the user's actual needs;

[0032] 5) The controllable image description model of the present invention also uses the object regression box and mouse browsing trajectory as input data for image description, thereby improving the ability of image description to capture key semantic information;

[0033] 6) This invention generates sample image features X ν At the same time, by using object regression boxes to fuse the original image feature vectors, the amount of information contained in the image features is increased, and the model is provided with the location of the information, which further improves the accuracy of controllable image description. Attached Figure Description

[0034] Figure 1 This is a flowchart comparing the present invention with traditional image description methods;

[0035] Figure 2 This is a flowchart illustrating the controllable image description method based on multimodal fusion of the present invention.

[0036] Figure 3 This is a schematic diagram of the structure of the TO-Transformer feature fusion model and output model of the present invention;

[0037] Figure 4 This is a schematic diagram of the structure of the MHA model of the present invention;

[0038] Figure 5This is a schematic diagram of the HA model of the present invention;

[0039] Figure 6 This is a schematic diagram of the process for generating sample image features according to the present invention;

[0040] Figure 7 This is a schematic diagram of the Emd feature embedding process of the present invention;

[0041] Figure 8 This is an example diagram illustrating the image description of the image under test according to the present invention. Detailed Implementation

[0042] The technical solution of the present invention will be further described in detail below through specific embodiments.

[0043] Example 1

[0044] Appendix Figure 2 and 3 A controllable image description method based on multimodal fusion is shown, which includes the following steps:

[0045] Obtain sample images from the sample dataset, and use ResNet-101 to extract image features from the sample images to generate the original image features X. ν * The sample images are processed using a Faster R-CNN-based object detection model to generate object regression boxes, and object regression box features X are generated based on these object regression boxes. ν b ; for the original image feature X ν * and the object regression box feature X ν b The images are fused to generate sample image features X. ν ;

[0046] Obtain the labeled language text associated with the sample image from the sample dataset, extract each word from the labeled language text, and label and positionally encode the extracted words to generate sample text description features X. w ;

[0047] Obtain mouse trajectories associated with the sample image from the sample dataset, segment the mouse trajectories according to time sequence to obtain multiple mouse trajectory segments, generate trajectory regression boxes based on the mouse trajectory segments, and generate sample trajectory features X based on the trajectory regression boxes. r ;

[0048] Using a pre-built TO-Transformer feature fusion model, the sample image features X are processed. νThe sample text description feature X w and the sample trajectory feature X r Feature fusion is performed to generate sample fusion features; wherein, the TO-Transformer feature fusion model is used to respectively integrate the sample image features X ν The sample text description feature X w The sample trajectory feature X r The data is processed into serialized data, and then the feature vector positions are aligned.

[0049] Based on the sample fusion features, a controllable image description model is generated using a pre-built output model.

[0050] It should be noted that the feature vector alignment method used in the TO-Transformer feature fusion model refers to the alignment of the sample image features X. ν The sample text description feature X w The sample trajectory feature X r The data alignment corresponding to each position in these three vectors is encoded.

[0051] like Figure 2 As shown, this invention uses mouse trajectory and object bounding box information as additional information input to the image description model, enabling the sample text description features and the sample image features X to... ν Aligning with the sample trajectory features, a controllable image description model is formed, thereby solving the problem of inaccurate description caused by the lack of information sources in the image description model;

[0052] Figure 1 In the method shown, the present invention integrates mouse trajectory and object bounding box information. The mouse trajectory and object bounding box provide the model with information on the region of interest and the position of objects in the image, enabling the controllable image description model to output text descriptions along the mouse trajectory. This improves the quality of image description while making the image description (human) controllable.

[0053] Furthermore, using a pre-built TO-Transformer feature fusion model, the sample image features X are... ν The sample text description feature X w and the sample trajectory feature X r When performing feature fusion to generate sample fusion features, the following steps are executed:

[0054] The pre-built TO-Transformer feature fusion model includes image feature processing structures. First feedforward neural network (FNN), text description feature processing structure Trajectory feature processing structure First feature fusion structure Second feature fusion structure and third feature fusion structure ;

[0055] Using the image feature processing structure For the sample image features X ν After image feature extraction, the first feedforward neural network (FNN) is used to flatten the extracted image features to obtain the image features to be fused.

[0056] Using the text description feature processing structure The sample text description feature X w Perform text feature extraction to obtain the features of the text to be fused;

[0057] Based on the first feature fusion structure The image features and text features to be fused are fused to obtain image-text fused features;

[0058] Through the trajectory feature processing structure For the sample trajectory feature X r Feature extraction is performed to obtain the trajectory features to be fused;

[0059] The second feature fusion structure is adopted The image features to be fused and the trajectory features to be fused are fused to obtain image trajectory fusion features;

[0060] Based on the third feature fusion structure The image text fusion features and the image trajectory fusion features are fused to obtain sample fusion features.

[0061] Specifically, the image feature processing structure The text description feature processing structure The trajectory feature processing structure The first feature fusion structure The second feature fusion structure and the third feature fusion structure All employ a multi-head attention (MHA) mechanism, the structure of which is shown in the attached figure. Figure 4 As shown; the MHA consists of h head attention points (HAs), and its structure is as shown in the attached figure. Figure 5 As shown, MHA is calculated as follows:

[0062]

[0063] Where Concat represents concatenating the calculated attention results for each head. for , represents a linear learnable matrix. This represents the size of the hidden layer in the Transformer;

[0064] In the multi-head attention mechanism, vector multiplication is mainly performed. The TO-Transformer feature fusion model is used for the sample image features X. ν The sample text description feature X w and the sample trajectory feature X r These three vectors are multiplied pairwise, and the results are multiplied again to achieve the alignment and fusion of the three feature vectors.

[0065] Specifically, the expression for the features of the image to be fused is:

[0066]

[0067] in, Indicates the features of the images to be fused. With sample image features X ν same;

[0068] The expression for the image-text fusion feature is:

[0069] ;

[0070] in, This represents the image-text fusion feature. Indicates the text features to be merged. With the sample text description feature X w same;

[0071] The expression for the image trajectory fusion feature is:

[0072]

[0073] in, This represents the image trajectory fusion feature. Indicates the trajectory features to be fused. With the sample trajectory feature X r same;

[0074] The expression for the sample fusion feature is:

[0075]

[0076] in, This represents the sample fusion feature. Indicates the second feature fusion structure Output image-text fusion features (vectors). Represents the first feature fusion structure Output image trajectory fusion features (vector).

[0077] It should be noted that the function of the first feedforward neural network (FNN) is to stretch the image features to match the vector dimension of the text features, thus facilitating the first feature fusion structure. Perform feature fusion;

[0078] Specifically, the first feedforward neural network (FNN) is defined as follows:

[0079]

[0080] in, Represents the ReLU activation function. and Weight matrices and , where f is the size of the hidden layers in the feedforward neural network. and The preset deviation is d, where d represents the hidden layer size; f, , d and d can be chosen by the user, and are usually preset constants.

[0081] As attached Figure 3 As shown, the image feature processing structure in the TO-Transformer feature fusion model Text description feature processing structure Trajectory feature processing structure The first feature fusion structure acts as an encoder. and the second feature fusion structure The task of multimodal feature fusion is performed, fusing image features and text features, and image features and trajectory features respectively; while the third feature fusion structure Acting as a decoder, processing data from the first feature fusion structure and the second feature fusion structure The output information is then fused a second time to achieve the alignment and fusion of the three feature vectors.

[0082] It should be noted that after obtaining the annotated language text in the sample dataset, a vocabulary W is also generated based on the annotated language text, and the vocabulary W includes each word annotated in the dataset;

[0083] As attached Figure 3As shown, the pre-built output model includes a second feedforward neural network (FNN), a fully connected layer, and a normalization layer (softmax). The sample fusion features output by the TO-Transformer feature fusion model are processed by the second feedforward neural network (FNN) and the fully connected layer (to flatten the dimensionality of the output of the second feedforward neural network (FNN) so that the output dimension is the same as the vocabulary length), and finally by the normalization layer (to calculate the probability of each word in the vocabulary, and the word with the highest probability is the word to be output). The output image description is then generated. , The expression is:

[0084]

[0085] in, This represents the output vector of the second feedforward neural network (FNN), where Linear represents a fully connected layer (linear layer). For learnable linear transformation matrix , For deviation, You can choose the option yourself; it's usually a preset constant. Softmax represents the normalization layer.

[0086] Example 2

[0087] Based on Example 1, this example provides a method for generating sample image features X. ν The specific implementation method is shown in the appendix. Figure 6 As shown;

[0088] Specifically, for the original image feature X ν * and the object regression box feature X ν b The images are fused to generate sample image features X. ν At that time, execute:

[0089] Step A1, for the original image features X ν * Feature embedding and layer normalization are performed to obtain the intermediate image feature vector I;

[0090] The regression box feature X of the object ν b Feature embedding and layer normalization are performed to obtain the intermediate image feature vector II;

[0091] Step A2: After superimposing the intermediate image feature vector I and the intermediate image feature vector II, layer normalization is performed to obtain the sample image feature X. ν The sample image feature X ν The expression is as follows:

[0092]

[0093] in, ( ) represents feature embedding processing, and N[ ] represents layer normalization processing;

[0094] The object regression box feature X in the above formula ν b The expression is:

[0095]

[0096] Where (x1, y1) are the coordinates of the upper left corner of the object regression box, (x2, y2) are the coordinates of the lower right corner of the object regression box, w is the width of the sample image, and h is the height of the sample image.

[0097] Specifically, the intermediate feature map vectors of ResNet-101 are used as the original image features. (vector dimension is) The region proposal network (RPN) is used to generate bounding boxes for object proposals. The bounding boxes are represented as 5-dimensional feature vectors, thus obtaining the object bounding box features X in the above formula. ν b .

[0098] Specifically, the feature embedding process Emd mentioned in steps A1 and A2 employs an appended... Figure 7 The algorithm process is shown below; Figure 7 middle The feature vector to be embedded, This is the embedded feature vector. Wherein, the appended... Figure 7 The input size of the linear layer is the feature vector to be embedded. The size of the hidden layer is d, and the output is the size of the hidden layer (usually 512 or 2048).

[0099] It should be noted that this embodiment utilizes object regression boxes as additional supplementary information and fuses the object regression boxes with image features to provide the image feature vector with its relative position in the image (corresponding to...). (the first four dimensions), thereby improving the quality of image description.

[0100] Example 3

[0101] Based on the above embodiments, this embodiment provides a method for generating sample text description features X. w Specific implementation methods;

[0102] Furthermore, each word in the annotated language text is extracted, and the extracted words are labeled and coded at their positions to generate sample text description features X. w At that time, execute:

[0103] Step B1: Tag the extracted words to generate the first tagging information w of the annotated language text; wherein, the first tagging information w is represented as... L represents the length of the annotated language text. The marker symbol represents the first word in the labeled language text, and so on. The marker symbol for the Lth word in the annotated language text;

[0104] Step B2: Compare the length L of the tagged language text with the pre-stored maximum sentence length threshold LMAX.

[0105] If the length L of the annotated language text is greater than or equal to the preset maximum sentence length threshold LMAX, then the first tagging information w will be... LMAX The subsequent deletion of the marker symbols yields the second marker information. At this time, the second tag information Represented as ;

[0106] If the length L of the annotated language text is less than the maximum sentence length threshold LMAX, then the first tagging information w is padded to obtain the second tagging information. At this time, the second tag information Represented as ,in, and The marker between them is a zero marker;

[0107] Step B3, for the second tag information Each marker symbol in the array is positionally encoded to obtain the encoded symbol PWn for each marker symbol;

[0108] Step B4, based on the second tag information And the encoded symbol PWn of each marker symbol, generate the word vector for each word in the annotated language text. The word vectors The expression is:

[0109]

[0110] in, ( ) indicates feature embedding processing. Indicates the second tag information The marker symbol for the nth word, PWn represents the second marker information. The encoded symbol of the nth marker symbol in the code;

[0111] Based on the word vector of each word in the annotated language text Generate the sample text description feature X w Wherein, the sample text description feature X w Represented as , This represents the word vector of the first word, and so on. This represents the word vector of the LMAX-th word.

[0112] The method for generating the encoded symbol PWn includes generating two calculation formulas: one for calculating the encoded symbol of words (or Chinese characters) at odd positions, and the other for calculating the encoded symbol of characters at even positions.

[0113] Encoding symbols for characters at even positions The calculation formula is:

[0114]

[0115] Encoding symbols for characters in odd positions The calculation formula is:

[0116]

[0117] Where n represents the position number of the current word (or Chinese character) in the labeled language text, 0≤n≤L / 2; i represents the dimension corresponding to the current word (or Chinese character); d represents the size of the hidden layer for attention, such as 512.

[0118] It is understandable that by discarding the annotated language text portion exceeding the maximum sentence length threshold LMAX, and by using preset zero-marking (such as...), <unk>The annotation language text is supplemented to make the sample text describe feature X. w It is a preset fixed length to facilitate subsequent alignment.

[0119] Specifically, the feature embedding process Emb mentioned in steps B4 and B5 employs an append-only approach. Figure 7 The algorithm process is shown, word vectors After embedding, the sample text description features X are obtained. w : .

[0120] Example 4

[0121] This embodiment provides a method for generating sample trajectory features X. r In a specific implementation, the mouse trajectory is converted into a trajectory regression box that corresponds one-to-one with the words in the sample text description features, so that each trajectory regression box is aligned with the labeled word.

[0122] Specifically, the sample trajectory feature X r The expression is:

[0123]

[0124] in, This refers to the trajectory regression box generated based on the mouse trajectory segment (by pre-annotating a word vector (word) and associating it with a trajectory regression box). The coded symbol representing the trajectory regression box, () represents feature embedding processing, N[ ] represents layer normalization processing; make the encoding symbols of the trajectory regression boxes with correlations consistent with the encoding symbols of the word vectors, and make the trajectory regression boxes aligned with the word vectors.

[0125] It should be noted that by performing feature embedding on the trajectory regression bounding box, the trajectory regression bounding box after feature embedding is assigned a positional encoding symbol. Therefore, this embodiment not only uses the mouse trajectory as an additional input for image description, providing the model with the region of interest in the image, but also makes the sample trajectory feature X... r and sample text description features X w Alignment facilitates subsequent feature fusion.

[0126] Example 5

[0127] It is understood that the sample dataset includes sample images, labeled language text, and mouse trajectories. A sample image, a labeled language text, and a mouse trajectory are associated. The labeled language text and mouse trajectory are used to teach the controllable image description model to speak. After training, the model no longer needs to input text information when describing images.

[0128] In practical applications, different mouse (browsing) trajectories result in different image description information output by the controllable image description model, achieving controllability. Therefore, users can configure the controllable image description model to output different texts on the same image according to their needs. The target detection bounding box provides additional information to the output, making it more accurate.

[0129] Therefore, based on the above embodiments, the controllable image description method based on multimodal fusion further includes the step of describing the image to be tested:

[0130] Read the image to be tested and generate the object regression bounding box and initial image features of the image to be tested;

[0131] Obtain the mouse trajectory input of the image to be tested, and generate the trajectory features of the image to be tested;

[0132] Based on the object regression bounding boxes and initial image features of the image to be tested, generate image features of the image to be tested;

[0133] The trajectory features and image features of the image to be tested are input into the controllable image description model to obtain the image description of the image to be tested.

[0134] In one specific implementation, this embodiment uses the following parameters: maximum sentence length threshold LMAX is 200, the hidden size d of the attention layer is 512, the hidden size f of the feedforward layer is 2048, the beam search size is set to 2, the initial learning rate is 5e-4, decays once every 3 rounds with a decay rate of 0.8, the training batch size is 100, and there are a total of 120 rounds. The cross-entropy loss function is used for training. Experimental results demonstrate the effectiveness of the controllable image description model, as shown in the attached figure. Figure 1 and attached Figure 8 As shown.

[0135] Example 6

[0136] Based on the above embodiments, this embodiment provides a specific implementation of a controllable image description system based on multimodal fusion;

[0137] Furthermore, the controllable image description system based on multimodal fusion includes:

[0138] The image feature generation module is used to acquire sample images from the sample dataset, extract image features from the sample images using ResNet-101, and generate original image features X. ν * The sample images are processed using a Faster R-CNN-based object detection model to generate object regression boxes, and object regression box features X are generated based on these object regression boxes. ν b ; for the original image feature X ν * and the object regression box feature X ν b The images are fused to generate sample image features X. ν ;

[0139] The text description feature generation module is used to obtain the annotated language text associated with the sample image in the sample dataset, extract each word from the annotated language text, and mark and positionally encode the extracted words to generate sample text description features X. w ;

[0140] The trajectory feature generation module is used to acquire mouse trajectories associated with the sample image in the sample dataset, segment the mouse trajectories according to time sequence to obtain multiple mouse trajectory segments, generate trajectory regression boxes based on the mouse trajectory segments, and generate sample trajectory features X based on the trajectory regression boxes. r ;

[0141] The feature fusion module is used to perform feature fusion on the sample image features X using a pre-built TO-Transformer feature fusion model. ν The sample text description feature X w and the sample trajectory feature X r Feature fusion is performed to generate sample fusion features; wherein, the TO-Transformer feature fusion model is used to respectively integrate the sample image features X ν The sample text description feature X w The sample trajectory feature X r The data is processed into serialized data, and then the feature vector positions are aligned.

[0142] And an image description model generation module, which is used to generate a controllable image description model based on the sample fusion features and using a pre-built output model.

[0143] Furthermore, the image feature generation module includes an original image feature processing unit, an object regression box feature processing unit, and an image feature fusion unit, wherein,

[0144] The original image feature processing unit is used to process the original image feature X. ν * Feature embedding and layer normalization are performed to obtain the intermediate image feature vector I;

[0145] The object regression box feature processing unit is used to process the object regression box feature X. ν b Feature embedding and layer normalization are performed to obtain the intermediate image feature vector II;

[0146] The image feature fusion unit superimposes the intermediate image feature vector I and the intermediate image feature vector II, and then performs layer normalization processing to obtain the sample image feature X. ν The sample image feature X ν The expression is as follows:

[0147]

[0148] in, ( ) represents feature embedding processing, and N[ ] represents layer normalization processing;

[0149] The object regression box feature X in the above formula ν b The expression is:

[0150]

[0151] Where (x1, y1) are the coordinates of the upper left corner of the object regression box, (x2, y2) are the coordinates of the lower right corner of the object regression box, w is the width of the sample image, and h is the height of the sample image.

[0152] Furthermore, the text description feature generation module includes a word tagging unit, a tag information processing unit, a position encoding unit, a word vector generation unit, and a text description feature output unit, wherein,

[0153] The word tagging unit is used to tag the extracted words and generate first tagging information w of the annotated language text; wherein, the first tagging information w is represented as L represents the length of the annotation language text;

[0154] The tagging information processing unit is used to compare the length L of the tagged language text with a pre-stored maximum sentence length threshold LMAX.

[0155] If the length L of the annotated language text is greater than or equal to the maximum sentence length threshold LMAX, then the first tagging information w will be... LMAX The subsequent deletion of the marker symbols yields the second marker information. At this time, the second tag information Represented as ;

[0156] If the length L of the annotated language text is less than the maximum sentence length threshold LMAX, then the first tagging information w is padded to obtain the second tagging information. At this time, the second tag information Represented as ,in, and The marker between them is a zero marker;

[0157] The location encoding unit is used for the second marker information. Each marker symbol in the array is positionally encoded to obtain the encoded symbol PWn for each marker symbol;

[0158] The word vector generation unit is used to generate words based on the second tag information. And the encoded symbol PWn of each marker symbol, generate the word vector for each word in the annotated language text. The word vectors The expression is:

[0159]

[0160] in, Indicates the second tag information The marker for the nth word in the text. ( ) indicates feature embedding processing, and PWn represents the second labeling information. The encoded symbol of the nth marker symbol in the code;

[0161] The text description feature output unit is used to output features based on the word vector of each word in the annotated language text. Generate the sample text description feature X w Wherein, the sample text description feature X w Represented as , This represents the word vector of the first word, and so on. This represents the word vector of the LMAX-th word.

[0162] Furthermore, the pre-built TO-Transformer feature fusion model includes an image feature processing structure. First feedforward neural network (FNN), text description feature processing structure Trajectory feature processing structure First feature fusion structure Second feature fusion structure and third feature fusion structure ;

[0163] The pre-built output model includes a second feedforward neural network (FNN), a fully connected layer, and a normalized softmax layer.

[0164] Example 7

[0165] Based on the above embodiments, this embodiment provides a specific implementation of a controllable image description device based on multimodal fusion, which includes a memory, a processor, and a controllable image description program based on multimodal fusion stored in the memory and executable on the processor. When the controllable image description program based on multimodal fusion is executed by the processor, it implements the steps of the controllable image description method based on multimodal fusion as described above.

[0166] Based on the above embodiments, this embodiment also provides a specific implementation of a readable storage medium storing instructions that, when executed by a processor, implement the steps of the controllable image description method based on multimodal fusion as described above.

[0167] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0168] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0169] In the embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of the modules described above is only a logical functional division, and there may be other division methods in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed.

[0170] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0171] If the integrated modules described above are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms.

[0172] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit them; although the present invention has been described in detail with reference to preferred embodiments, those skilled in the art should understand that modifications can still be made to the specific implementation of the present invention or equivalent substitutions can be made to some technical features without departing from the spirit of the technical solutions of the present invention, and all such modifications and substitutions should be covered within the scope of the technical solutions claimed in the present invention.< / unk>

Claims

1. A controllable image description method based on multimodal fusion, characterized in that, Includes the following steps: Obtain sample images from the sample dataset, and use ResNet-101 to extract image features from the sample images to generate the original image features X. ν * The sample images are processed using a Faster R-CNN-based object detection model to generate object regression boxes, and object regression box features X are generated based on these object regression boxes. ν b ; for the original image feature X ν * and the object regression box feature X ν b The images are fused to generate sample image features X. ν ; Obtain the labeled language text associated with the sample image from the sample dataset, extract each word from the labeled language text, and label and positionally encode the extracted words to generate sample text description features X. w ; Obtain mouse trajectories associated with the sample image from the sample dataset, segment the mouse trajectories according to time sequence to obtain multiple mouse trajectory segments, generate trajectory regression boxes based on the mouse trajectory segments, and generate sample trajectory features X based on the trajectory regression boxes. r ; Using a pre-built TO-Transformer feature fusion model, the sample image features X are processed. ν The sample text description feature X w and the sample trajectory feature X r Feature fusion is performed to generate sample fusion features; wherein, the TO-Transformer feature fusion model is used to respectively fuse the sample image features X ν The sample text description feature X w and the sample trajectory feature X r The data is processed into serialized data, and then the feature vector positions are aligned. Based on the sample fusion features, a controllable image description model is generated using a pre-built output model.

2. The controllable image description method based on multimodal fusion according to claim 1, characterized in that, For the original image features X ν * and the object regression box feature X ν b The images are fused to generate sample image features X. ν At that time, execute: Step A1, for the original image features X ν * Feature embedding and layer normalization are performed to obtain the intermediate image feature vector I; The regression box feature X of the object ν b Feature embedding and layer normalization are performed to obtain the intermediate image feature vector II; Step A2: After superimposing the intermediate image feature vector I and the intermediate image feature vector II, layer normalization is performed to obtain the sample image feature X. ν The sample image feature X ν The expression is as follows: ， in, ( ) represents feature embedding processing, and N[ ] represents layer normalization processing; The object regression box feature X in the above formula ν b The expression is: ， Where (x1, y1) are the coordinates of the upper left corner of the object regression box, (x2, y2) are the coordinates of the lower right corner of the object regression box, w is the width of the sample image, and h is the height of the sample image.

3. The controllable image description method based on multimodal fusion according to claim 1, characterized in that, Extract each word from the labeled language text, and mark and positionally encode the extracted words to generate sample text description features X. w At that time, execute: Step B1: Tag the extracted words to generate the first tagging information w of the annotated language text; wherein, the first tagging information w is represented as... L represents the length of the annotation language text; Step B2: Compare the length L of the tagged language text with the pre-stored maximum sentence length threshold LMAX. If the length L of the annotated language text is greater than or equal to the maximum sentence length threshold LMAX, then the first tagging information w will be... LMAX The subsequent deletion of the marker symbols yields the second marker information. ; If the length L of the annotated language text is less than the maximum sentence length threshold LMAX, then the first tagging information w is padded to obtain the second tagging information. ; Step B3, for the second tag information Each marker symbol in the array is positionally encoded to obtain the encoded symbol PWn for each marker symbol; Step B4, based on the second tag information And the encoded symbol PWn of each marker symbol, generate the word vector for each word in the annotated language text. The word vectors The expression is: ， in, Indicates the second tag information The marker for the nth word in the text. ( ) indicates feature embedding processing, and PWn represents the second labeling information. The encoded symbol of the nth marker symbol in the code; Based on the word vector of each word in the annotated language text Generate the sample text description feature X w Wherein, the sample text description feature X w Represented as , This represents the word vector of the first word, and so on. This represents the word vector of the LMAX-th word.

4. The controllable image description method based on multimodal fusion according to claim 3, characterized in that, The sample trajectory feature X r The expression is: ， in, This represents the trajectory regression box generated based on the mouse trajectory segment. The coded symbol representing the trajectory regression box, () represents feature embedding processing, and N[] represents layer normalization processing.

5. The controllable image description method based on multimodal fusion according to claim 1, characterized in that, Using a pre-built TO-Transformer feature fusion model, the sample image features X are processed. ν The sample text description feature X w and the sample trajectory feature X r When performing feature fusion to generate sample fusion features, the following steps are executed: The pre-built TO-Transformer feature fusion model includes image feature processing structures. First feedforward neural network (FNN), text description feature processing structure Trajectory feature processing structure First feature fusion structure Second feature fusion structure and third feature fusion structure ; Using the image feature processing structure For the sample image features X ν After image feature extraction, the first feedforward neural network (FNN) is used to flatten the extracted image features to obtain the image features to be fused. Using the text description feature processing structure The sample text description feature X w Perform text feature extraction to obtain the features of the text to be fused; Based on the first feature fusion structure The image features and text features to be fused are fused to obtain image-text fused features; Through the trajectory feature processing structure For the sample trajectory feature X r Feature extraction is performed to obtain the trajectory features to be fused; The second feature fusion structure is adopted The image features to be fused and the trajectory features to be fused are fused to obtain image trajectory fusion features; Based on the third feature fusion structure The image text fusion features and the image trajectory fusion features are fused to obtain sample fusion features.

6. A controllable image description system based on multimodal fusion, characterized in that, include: The image feature generation module is used to acquire sample images from the sample dataset, extract image features from the sample images using ResNet-101, and generate original image features X. ν * The sample images are processed using a Faster R-CNN-based object detection model to generate object regression boxes, and object regression box features X are generated based on these object regression boxes. ν b ; for the original image feature X ν * and the object regression box feature X ν b The images are fused to generate sample image features X. ν ; The text description feature generation module is used to obtain the annotated language text associated with the sample image in the sample dataset, extract each word from the annotated language text, and mark and positionally encode the extracted words to generate sample text description features X. w ; The trajectory feature generation module is used to acquire mouse trajectories associated with the sample image in the sample dataset, segment the mouse trajectories according to time sequence to obtain multiple mouse trajectory segments, generate trajectory regression boxes based on the mouse trajectory segments, and generate sample trajectory features X based on the trajectory regression boxes. r ; The feature fusion module is used to perform feature fusion on the sample image features X using a pre-built TO-Transformer feature fusion model. ν The sample text description feature X w and the sample trajectory feature X r Feature fusion is performed to generate sample fusion features; wherein, the TO-Transformer feature fusion model is used to respectively fuse the sample image features X ν The sample text description feature X w The sample trajectory feature X r The data is processed into serialized data, and then the feature vector positions are aligned. And an image description model generation module, which is used to generate a controllable image description model based on the sample fusion features and using a pre-built output model.

7. The controllable image description system based on multimodal fusion according to claim 6, characterized in that: The image feature generation module includes a raw image feature processing unit, an object regression bounding box feature processing unit, and an image feature fusion unit, wherein... The original image feature processing unit is used to process the original image feature X. ν * Feature embedding and layer normalization are performed to obtain the intermediate image feature vector I; The object regression box feature processing unit is used to process the object regression box feature X. ν b Feature embedding and layer normalization are performed to obtain the intermediate image feature vector II; The image feature fusion unit superimposes the intermediate image feature vector I and the intermediate image feature vector II, and then performs layer normalization processing to obtain the sample image feature X. ν The sample image feature X ν The expression is as follows: ， in, ( ) represents feature embedding processing, and N[ ] represents layer normalization processing; The object regression box feature X in the above formula ν b The expression is: ， Where (x1, y1) are the coordinates of the upper left corner of the object regression box, (x2, y2) are the coordinates of the lower right corner of the object regression box, w is the width of the sample image, and h is the height of the sample image.

8. The controllable image description system based on multimodal fusion according to claim 6, characterized in that: The text description feature generation module includes a word tagging unit, a tag information processing unit, a position encoding unit, a word vector generation unit, and a text description feature output unit, wherein... The word tagging unit is used to tag the extracted words and generate first tagging information w of the annotated language text; wherein, the first tagging information w is represented as L represents the length of the annotation language text; The tagging information processing unit is used to compare the length L of the tagged language text with a pre-stored maximum sentence length threshold LMAX. If the length L of the annotated language text is greater than or equal to the maximum sentence length threshold LMAX, then the first tagging information w will be... LMAX The subsequent deletion of the marker symbols yields the second marker information. At this time, the second tag information Represented as ; If the length L of the annotated language text is less than the maximum sentence length threshold LMAX, then the first tagging information w is padded to obtain the second tagging information. At this time, the second tag information Represented as ,in, and The marker between them is a zero marker; The position encoding unit is used for the second marker information. Each marker symbol in the array is positionally encoded to obtain the encoded symbol PWn for each marker symbol; The word vector generation unit is used to generate words based on the second tag information. And the encoded symbol PWn of each marker symbol, generate the word vector for each word in the annotated language text. The word vectors The expression is: ， in, Indicates the second tag information The marker for the nth word in the text. ( ) indicates feature embedding processing, and PWn represents the second labeling information. The encoded symbol of the nth marker symbol in the code; The text description feature output unit is used to output features based on the word vector of each word in the annotated language text. Generate the sample text description feature X w Wherein, the sample text description feature X w Represented as , This represents the word vector of the first word, and so on. This represents the word vector of the LMAX-th word.

9. A controllable image description device based on multimodal fusion, characterized in that: The method includes a memory, a processor, and a controllable image description program based on multimodal fusion stored in the memory and executable on the processor. When executed by the processor, the controllable image description program based on multimodal fusion implements the steps of the controllable image description method based on multimodal fusion as described in any one of claims 1-5.

10. A readable storage medium having instructions stored thereon, characterized in that: When executed by the processor, this instruction implements the steps of the controllable image description method based on multimodal fusion as described in any one of claims 1-5.

Images