Image description method, device, storage medium and terminal based on local representation enhancement

Abstract

The present invention discloses an image description method and device based on local representation enhancement, a storage medium and a terminal, wherein the method uses a trained image description model, and the image description model includes the following modules: a multimodal feature encoding module, which is mainly used to obtain a set of visual features and word embedding features; a visual local representation enhancement module, which is mainly used to select target word embedding features, obtain a first visual attention feature, and obtain the hidden state at the current moment based on the first visual attention feature, etc.; a multimodal interaction enhancement module, which is mainly used to obtain a second visual attention feature at the current moment, and update a multimodal feature memory pool, thereby obtaining interactive context information at the current moment; a semantic local representation enhancement module, which is mainly used to obtain semantic context information at the current moment; and a word prediction module, which is mainly used to predict the description word at the current moment. The present invention adopts a cyclic convolutional network to enhance the degree of interaction between multimodal information and improve the quality of generated sentences.

Description

Technical Field

[0001] The present invention relates to the field of image description technology, and in particular to an image description method and device based on local representation enhancement, a storage medium and a terminal. Background Art

[0002] In the Internet age, images and text are two important data for people to perceive and understand the real world in their daily lives. The image description task combines computer vision tasks and natural language processing tasks, with the goal of enabling machines to generate a natural description sentence for an image. Image description is one of the basic tasks for achieving human-computer interaction. It can be used as an auxiliary tool for blind people to perceive the world, an image sentence annotation tool, and an intelligent social media tool, etc., and has important research and application value.

[0003] Early researchers used template-based image description methods, which used classifiers to extract information such as objects and attributes in images and filled them into manually specified sentence templates to obtain sentences describing the image content. However, such methods rely too much on manually designed templates, which limits the complexity and flexibility of generated sentences.

[0004] With the development of deep learning technology and the remarkable success of machine translation tasks, current image description methods have achieved good results using the encoder-decoder framework. However, there are still some problems with existing image description methods. From the perspective of sentence quality, the sentences generated by existing methods have the problem of insufficient detail information and insufficient accuracy. From the perspective of the network framework of the image description model, the existing image description model does not fully utilize the visual fine-grained information (such as visual area features) and language local information (such as the semantic information of text words in the sentence), resulting in the model not fully capturing the local representation from vision and language, which in turn leads to insufficient detail information obtained by the model; at the same time, the model does not fully interact with the information of the image and text modalities, resulting in deviations in the semantic understanding of multimodal data, affecting the quality of generated sentences. In addition, when generating description sentences, the existing image description model fails to effectively combine the characteristics of convolutional calculations with the requirements for obtaining fine-grained information in the image description model and the interaction requirements of multimodal data, and the advantages of convolutional network feature extraction are not fully utilized. Summary of the invention

[0005] The technical problem to be solved by the present invention is that the image description model in the existing image description method does not fully utilize the visual fine-grained information and the language local information, so that the model obtains insufficient detail information, resulting in deviations in the semantic understanding of multimodal data and affecting the quality of generated sentences.

[0006] In order to solve the above technical problems, the present invention provides an image description method based on local representation enhancement, comprising:

[0007] Acquire an image to be described, and input the image to be described into a trained image description model to obtain a description sentence of the image to be described;

[0008] The image description model includes a multimodal feature encoding module, a visual local representation enhancement module, a multimodal interaction enhancement module, a semantic local representation enhancement module and a word prediction module;

[0009] The multimodal feature encoding module is used to obtain visual features from the image to be trained or the image to be described, and obtain a word embedding feature set based on text data, and initialize the hidden state at the last moment, the memory state at the last moment, the second visual attention feature at the last moment, and the semantic context information at the last moment to zero, wherein the visual features include an image region feature set and a global image feature;

[0010] The visual local representation enhancement module is used to select a target word embedding feature from the word embedding feature set, or use the word embedding feature corresponding to the description word predicted at the previous moment as the target word embedding feature, input the image area feature set and the semantic context information at the previous moment into the attention mechanism, obtain a first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state at the previous moment, the memory state at the previous moment and the second visual attention feature at the previous moment, obtain the hidden state at the current moment through the first convolutional network and the gate mechanism;

[0011] The multimodal interaction enhancement module is used to input the image region feature set and the hidden state at the current moment into the visual attention mechanism, obtain the second visual attention feature at the current moment, and update the multimodal feature memory pool based on the second visual attention feature at the current moment and the hidden state at the current moment, and obtain the interaction context information at the current moment through the second convolutional network based on the target word embedding feature and the current multimodal feature memory pool;

[0012] The semantic local representation enhancement module is used to obtain the semantic context information at the current moment through the third convolutional network and the fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature;

[0013] The word prediction module is used to predict the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and to determine whether to end the prediction process based on the description word at the current moment.

[0014] Preferably, the image description model training process includes:

[0015] Acquire a training data set, and group all training data in the training data set to obtain multiple training data groups, wherein each of the training data includes an image to be trained and text data corresponding to the image to be trained;

[0016] Using the Mth training data set in the training data set as the training data set to be input;

[0017] The current image description model is trained based on the training data group to be input, and the training parameters of the current image description model are optimized based on the loss function, and it is determined whether M is equal to a preset threshold value. If so, the evaluation index of the training data set in this round of training is calculated, and it is determined whether the evaluation index meets the training termination condition. If so, the training is stopped, and the current image description model is used as the trained image description model. If the evaluation index does not meet the training termination condition, M is initialized to 1, and the Mth group of training data in the training data set is re-used as the training data group to be input, so as to continue a new round of training for the current image description model. If M is not equal to the preset threshold value, M is increased by 1, and the Mth group of training data in the training data set is re-used as the training data group to be input, so as to continue training the current image description model.

[0018] Among them, the initial value of M is 1, the preset threshold is the number of training data groups in the training data set, and one round of training is completed when all training data groups in the training data set are trained once in sequence.

[0019] Preferably, calculating the evaluation index of the training data set in the current round of training includes: taking the text data corresponding to the image to be trained as a reference value, calculating the preset evaluation index of the prediction description sentence corresponding to each image to be trained in the current round of training, and calculating the average value of the preset evaluation indexes of the prediction description sentences corresponding to all the images to be trained in the current round of training as the evaluation index of the training data set in the current round of training;

[0020] Judging whether the evaluation index meets the training termination condition includes: taking the current round of training as the starting point, selecting the evaluation index of a preset number of rounds of training in sequence as the evaluation parameter, and calculating the difference between two adjacent evaluation parameters as the training growth value, and judging whether all the training growth values ​​are less than a preset threshold value. If so, it means that the evaluation index corresponding to the current round of training meets the training termination condition, otherwise, it means that the evaluation index corresponding to the current round of training does not meet the training termination condition.

[0021] Preferably, in the process of training the current image description model, the training process of the image description model on a single training data includes:

[0022] The multimodal feature encoding module obtains corresponding visual features based on the image to be trained in the training data, obtains a word embedding feature set based on the text data corresponding to the image to be trained, and initializes the hidden state at the last moment, the memory state at the last moment, the second visual attention feature at the last moment, and the semantic context information at the last moment to zero, wherein the visual features include an image region feature set and a global image feature;

[0023] The visual local representation enhancement module selects a target word embedding feature from the word embedding feature set, inputs the visual feature and the semantic context information at the previous moment into the attention mechanism to obtain a first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state at the previous moment, the memory state at the previous moment, and the second visual attention feature at the previous moment, obtains the hidden state at the current moment through the first convolutional network and the gate mechanism;

[0024] The multimodal interaction enhancement module inputs the visual feature and the hidden state at the current moment into the visual attention mechanism to obtain the second visual attention feature at the current moment, and updates the current multimodal feature memory pool based on the second visual attention feature at the current moment and the hidden state at the current moment, and obtains the interaction context information at the current moment through the second convolutional network based on the target word embedding feature and the current multimodal feature memory pool;

[0025] The semantic local representation enhancement module obtains the semantic context information at the current moment through the third convolutional network and the fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature;

[0026] The word prediction module predicts the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and determines whether the description word at the current moment meets the sentence end condition. If so, the prediction process is terminated, and a predicted description sentence is formed based on all the description words predicted by the training data. Otherwise, the visual local representation enhancement module re-selects the target word embedding feature from the word embedding feature set to obtain the description word at the next moment.

[0027] Preferably, the visual local representation enhancement module inputs the visual features and the semantic context information of the previous moment into the attention mechanism to obtain a first visual attention feature, and based on the first visual attention feature, the target word embedding feature, the hidden state of the previous moment, the memory state of the previous moment and the second visual attention feature of the previous moment, obtains the hidden state of the current moment through the first convolutional network and the gate mechanism, including:

[0028] Calculating the relevance of each of the image region features in the image region feature set with the semantic context information at the previous moment, and selecting a first preset number of image region features from the image region feature set in descending order of relevance to form a selected image region feature set;

[0029] Acquire a first visual attention feature based on the image region feature set and the correlation degree of each of the image region features in the image region feature set;

[0030] Based on the target word embedding feature, the sum of the first visual attention feature and the global image feature, the second visual attention feature at the previous moment, and the hidden state at the previous moment, the four-dimensional feature set is respectively expanded a first preset number of times to obtain a four-dimensional feature set, and the four-dimensional feature set and the selected image area feature set are concatenated as the input of the first convolutional network;

[0031] The first layer of the first convolutional network aggregates each selected image region feature in the selected image region feature set with features in the four-dimensional feature set to obtain a multimodal local representation with enhanced visual detail information, and the second and third layers of the first convolutional network perform full information interaction on the multimodal local representation with enhanced visual detail information through a two-dimensional convolution kernel to obtain a visually enhanced multimodal representation;

[0032] Based on the multimodal representation of visual enhancement, the hidden state at the previous moment and the memory state at the previous moment, the hidden state at the current moment is obtained through a gate mechanism.

[0033] Preferably, the multimodal interaction enhancement module acquires the interaction context information at the current moment based on the target word embedding feature and the current multimodal feature memory pool through a second convolutional network, including:

[0034] Expanding the target word embedding feature by a second preset number of times to obtain a one-dimensional feature set, and concatenating the one-dimensional feature set, the hidden state memory pool in the current multimodal feature memory pool, and the second visual attention feature memory pool in the current multimodal feature memory pool as input to the second convolutional network;

[0035] The first layer of the second convolutional network aggregates each second visual attention feature in the second visual attention feature memory pool in the current multimodal feature memory pool and each hidden state in the hidden state memory pool in the current multimodal feature memory pool with the features in the one-dimensional feature set to obtain an interactively enhanced multimodal local representation, and the second and third layers of the second convolutional network respectively perform full information interaction on the interactively enhanced multimodal local representation through a two-dimensional convolution kernel to obtain multimodal interactively enhanced contextual information.

[0036] Preferably, the semantic local representation enhancement module obtains the semantic context information at the current moment based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature through the third convolutional network and the fusion layer, including:

[0037] Expand the target word embedding feature, the sum of the first visual attention feature and the global image feature, and the hidden state at the previous moment by a second preset number of times to obtain a three-dimensional feature set, and concatenate the three-dimensional feature set with the hidden state memory pool in the current multimodal feature memory pool as an input of the third convolutional network;

[0038] The first layer of the third convolutional network aggregates each hidden state in the hidden state memory pool in the current multimodal feature memory pool with the features in the three-dimensional feature set to obtain a semantically enhanced multimodal local representation, and the second and third layers of the third convolutional network perform full information interaction on the semantically enhanced multimodal local representation through a two-dimensional convolution kernel to obtain a semantically enhanced multimodal representation;

[0039] The semantically enhanced multimodal representation is input into the fusion layer for further interaction to obtain the semantic context information at the current moment.

[0040] In order to solve the above technical problems, the present invention also provides an image description device based on local representation enhancement, including an image description module;

[0041] The image description module is used to obtain an image to be described, and input the image to be described into a trained image description model to obtain a description sentence of the image to be described;

[0042] The image description model includes a multimodal feature encoding module, a visual local representation enhancement module, a multimodal interaction enhancement module, a semantic local representation enhancement module and a word prediction module;

[0043] The multimodal feature encoding module is used to obtain visual features from the image to be trained or the image to be described, and obtain a word embedding feature set based on text data, and initialize the hidden state at the last moment, the memory state at the last moment, the second visual attention feature at the last moment, and the semantic context information at the last moment to zero, wherein the visual features include an image region feature set and a global image feature;

[0044] The visual local representation enhancement module is used to select a target word embedding feature from the word embedding feature set, or use the word embedding feature corresponding to the description word predicted at the previous moment as the target word embedding feature, input the image area feature set and the semantic context information at the previous moment into the attention mechanism, obtain a first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state at the previous moment, the memory state at the previous moment and the second visual attention feature at the previous moment, obtain the hidden state at the current moment through the first convolutional network and the gate mechanism;

[0045] The multimodal interaction enhancement module is used to input the image region feature set and the hidden state at the current moment into the visual attention mechanism, obtain the second visual attention feature at the current moment, and update the multimodal feature memory pool based on the second visual attention feature at the current moment and the hidden state at the current moment, and obtain the interaction context information at the current moment through the second convolutional network based on the target word embedding feature and the current multimodal feature memory pool;

[0046] The semantic local representation enhancement module is used to obtain the semantic context information at the current moment through the third convolutional network and the fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature;

[0047] The word prediction module is used to predict the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and to determine whether to end the prediction process based on the description word at the current moment.

[0048] In order to solve the above technical problem, the present invention also provides a storage medium on which a computer program is stored, and when the program is executed by a processor, an image description method based on local representation enhancement is implemented.

[0049] In order to solve the above technical problem, the present invention further provides a terminal, comprising: a processor and a memory, wherein the memory is communicatively connected to the processor;

[0050] The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal performs an image description method based on local representation enhancement.

[0051] Compared with the prior art, one or more embodiments of the above scheme may have the following advantages or beneficial effects:

[0052] The image description method based on local representation enhancement provided by the embodiment of the present invention is applied. The image description model used includes a recurrent convolutional network for multimodal tasks. The model effectively introduces fine-grained information of different modalities by using the convolution operation characteristics, strengthens the model's ability to capture multimodal local representations, improves the degree of interaction between multimodal information, and improves the quality of generated sentences. Furthermore, the model uses a visual local representation enhancement module to allow visual fine-grained features to fully interact with multimodal features, and to allow multimodal local representations enhanced with visual detail information to fully interact with each other; a semantic local representation enhancement module is used to allow hidden states generated at past moments to fully interact with multimodal features, and to allow multimodal local representations enhanced with semantics to fully interact with each other. A convolutional network is used to allow features containing similar semantics in visual and language modalities at the same moment to fully interact with multimodal features, and to allow multimodal local representations enhanced with semantic interactions to fully interact with each other; the collaborative work of the three convolutional networks can enrich the detailed information captured by the model, enhance the model's understanding of multimodal features, and thereby improve the quality of the overall description sentence.

[0053] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0055] Figure 1 A schematic diagram showing the structure of an image description model in an image description method based on local representation enhancement in Embodiment 1 of the present invention is shown;

[0056] Figure 2 A schematic diagram of a process of training a single training data by an image description model in Embodiment 1 of the present invention is shown;

[0057] Figure 3 A schematic diagram of the structure of a terminal according to a fourth embodiment of the present invention is shown. DETAILED DESCRIPTION

[0058] The following will describe the implementation methods of the present invention in detail with reference to the accompanying drawings and embodiments, so that the implementation process of how the present invention applies technical means to solve technical problems and achieve technical effects can be fully understood and implemented accordingly. It should be noted that as long as there is no conflict, the various embodiments of the present invention and the various features in the embodiments can be combined with each other, and the technical solutions formed are all within the protection scope of the present invention.

[0059] With the development of deep learning technology and the remarkable success of machine translation tasks, current image description methods have achieved good results using the encoder-decoder framework. However, there are still some problems with existing image description methods. From the perspective of sentence quality, the sentences generated by existing methods have the problem of insufficient detail information and insufficient accuracy. From the perspective of the network framework of the image description model, the existing image description model does not fully utilize the fine-grained visual information (such as visual area features) and local language information (such as the semantic information of text words in the sentence), resulting in the model not fully capturing the local representation from vision and language, which in turn leads to insufficient detail information obtained by the model; at the same time, the model does not fully interact with the information from the visual and language modalities, so there is a deviation in the semantic understanding of multimodal data, which affects the quality of generated sentences. In addition, when generating description sentences, the existing image description model does not consider the effective combination of the characteristics of convolutional calculation with the fine-grained information requirements and interaction requirements of the image description model, and the advantages of convolutional network feature extraction are not fully utilized.

[0060] Embodiment 1

[0061] In order to solve the technical key point confirmation problem existing in the prior art, an embodiment of the present invention provides an image description method based on local representation enhancement.

[0062] The image description method based on local representation enhancement in an embodiment of the present invention includes acquiring an image to be described, and inputting the image to be described into a trained image description model, and the trained image description model outputs a description sentence corresponding to the image to be described.

[0063] Figure 1 FIG. 1 shows a schematic diagram of the structure of an image description model in an image description method based on local representation enhancement according to an embodiment of the present invention; FIG. Figure 1 The image description model in the above-mentioned image description method based on local representation enhancement includes a multimodal feature encoding module, a visual local representation enhancement module, a multimodal interaction enhancement module, a semantic local representation enhancement module and a word prediction module.

[0064] Among them, the multimodal feature encoding module is mainly used to obtain visual features based on the image to be trained during the training process, and to obtain a word embedding feature set based on text data, or to obtain visual features based on the image to be described during the image description prediction process. It is also used to initialize the hidden state at the previous moment, the memory state at the previous moment, the second visual attention feature at the previous moment, and the semantic context information at the previous moment to zero in the initial state of training or the initial state of image description prediction.

[0065] Specifically, the multimodal feature encoding includes visual feature encoding, text feature encoding and feature initialization. Furthermore, the multimodal feature encoding module obtains visual features based on the image to be trained or the image to be described, and obtains word embedding features based on text data. The process includes:

[0066] Use the pre-trained object detection model to extract a temporary image region feature set for the image to be trained or described m is the number of temporary image region features, r m represents the temporary image region feature, preferably the temporary image region feature dimension can be d v =2048. Then, all temporary image region features in the temporary image region feature set R are mapped to the hidden state dimension through the fully connected layer. Preferably, the hidden state dimension can be selected as d n = 1024. The pre-trained target detection model can be selected from Faster RCNN, etc.

[0067] The mapped temporary image region feature set R is input into the self-attention mechanism to obtain the potential relationship between each temporary image region feature in the temporary image region feature set, and the temporary image region feature set is updated based on the potential relationship between each temporary image region feature in the temporary image region feature set to form the image region feature set R - Then, by using the image region feature set R - The average of all image region features in is used to obtain the global image feature g. - , image region feature set and global image feature g - Form visual features. The image region feature set R is obtained - and the global image feature g - The process can be expressed as follows:

[0068] R - = self-attention(fc(R))

[0069] g - =mean(R - )

[0070] Among them, fc() represents the fully connected layer mapping, self-attention() represents the feature update operation based on the self-attention mechanism, and mean() represents the average pooling operation.

[0071] The word embedding tool is used to obtain the word embedding features of all words in the text data, and all word embeddings form a word embedding feature set. The word embedding tool can be word2vec. The specific process of embedding words to obtain the corresponding word embedding features can be expressed by the following formula:

[0072] e i =emb(w i )

[0073] Among them, w i Represents the i-th word in the text data, represents word embedding features, and emb() represents word embedding mapping operations. The training data used in the training process includes the image to be trained and the text data corresponding to the image to be trained. Therefore, the above process is the process of generating word embedding features from the words in the text data during the training process.

[0074] Since the words that make up the sentences in the image description model are output in sequence, and the words have a chronological relationship, the image description model uses the features containing the word information of the previous moment to predict the words of the next moment. Therefore, whether in training or in normal prediction, the hidden state of the previous moment, the memory state of the previous moment, the second visual attention feature of the previous moment, and the semantic context information of the previous moment all need to be initialized to zero in the initial state.

[0075] The visual local representation enhancement module is mainly used to determine the target word embedding features, and input the image area feature set and the semantic context information of the previous moment into the attention mechanism to obtain the first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state of the previous moment, the memory state of the previous moment and the second visual attention feature of the previous moment, the hidden state of the current moment is obtained through the first convolutional network and the gate mechanism.

[0076] Specifically, the target word embedding feature determination process includes: during the training process, one word embedding feature in the word embedding feature set can be selected in turn as the target word embedding feature, wherein the word embedding feature arrangement order in the word embedding feature set is consistent with the original arrangement order of each word in the corresponding text data. In the image description prediction process, the word embedding feature corresponding to the description word predicted at the previous moment is required to be used as the target word embedding feature at the current moment, wherein the process of converting the description word predicted at the previous moment into the corresponding word embedding feature can be achieved through the word embedding tool of the multimodal feature encoding module. It should be noted that in the image description prediction process, the initialized target word embedding feature is empty.

[0077] The visual local representation enhancement module includes an attention mechanism, a first convolutional network, and a gate mechanism. The relevance of each image region feature in the image region feature set to the semantic context information of the previous moment is further calculated, and k image region features are selected from the image region feature set in descending order of relevance to form a selected image region feature set R s- ; The calculation process of the correlation between each image region feature and the semantic context information of the previous moment can be expressed as follows:

[0078]

[0079] Among them, α i Represents the i-th image region feature r in the image region feature set i - The degree of relevance to the semantic context information of the previous moment is the i-th image region feature r i - The weight assigned, W s are the model parameters in the image description model, for The transpose of Represents the semantic context information of the previous moment, softmax() is used to constrain the weights between 0 and 1, and the sum of the weights is 1, and k is the first preset number.

[0080] Select image region feature set R s- The process can be expressed as follows:

[0081]

[0082] Among them, top-k() represents the operation process of sorting the image region features from large to small according to the degree of relevance and selecting the top k image features.

[0083] Then, based on the image region feature set and the correlation degree of each image region feature in the image region feature set, the first visual attention feature is obtained. The specific process of obtaining the first visual attention feature can be expressed by the following formula:

[0084]

[0085] in, represents the first visual attention feature, α i Represents the i-th image region feature r in the image region feature set i - The degree of relevance to the semantic context information of the previous moment, r i - Represents the i-th image region feature in the image region feature set.

[0086] Based on the target word embedding feature, the sum of the first visual attention feature and the global image feature, the second visual attention feature of the previous moment, and the hidden state of the previous moment, we expand them k times to obtain four dimensions of k×d n The features of the image are used to form a four-dimensional feature set, and then the four-dimensional feature set is concatenated with the selected image region feature set to form as the input of the first convolutional network.

[0087] The first convolutional network has three layers. The first layer is a one-dimensional convolution operation, which is mainly used to aggregate each selected image area feature in the selected image area feature set with the features in the four-dimensional feature set to obtain a multimodal local representation with enhanced visual detail information. The second and third layers of the first convolutional network are both two-dimensional convolution operations, which are mainly used to fully interact the multimodal local representation of visual detail information through the two-dimensional convolution kernel to obtain a visually enhanced multimodal representation. The three-layer operation process of the first convolutional network can be expressed by the following formulas:

[0088]

[0089]

[0090]

[0091] Among them, W v1 ,W v2 ,W v3 The convolution kernels are 1×3, 3×3, and 3×3 respectively, and B v1 ,B v2 ,B v3is the bias, * indicates the convolution operation, Relu() is the activation function, and the calculation formula is Relu(x)=max(0,x), LeakyRelu() is also the activation function, and the calculation formula is λ can be set to 0.2, and the output channel of the third layer can be set to 4, that is, It should be noted that λ may also be set to other reasonable parameters, and no excessive restrictions are imposed on it here.

[0092] Multimodal representation of visual enhancement The four features in are used as the input of the gate mechanism respectively, and the hidden state at the previous moment and the memory state at the previous moment are combined to calculate the hidden state at the current moment. Preferably, the gate mechanism is an LSTM gate mechanism. The acquisition process of the hidden state at the current moment can be expressed by the following formula:

[0093]

[0094]

[0095]

[0096]

[0097]

[0098]

[0099] Among them, h t Indicates the hidden state at the current moment, express The i-th feature in , σ() and tanh() represent the sigmoid and hyperbolic tangent functions respectively, represents the Hadamard product.

[0100] The multimodal interaction enhancement module is mainly used to input the image area feature set and the hidden state at the current moment into the visual attention mechanism, obtain the second visual attention feature at the current moment, and update the multimodal feature memory pool based on the second visual attention feature at the current moment and the hidden state at the current moment, and obtain the interaction context information at the current moment through the second convolutional network based on the target word embedding feature and the current multimodal feature memory pool.

[0101] Specifically, the multimodal interaction enhancement module includes a visual attention mechanism, a multimodal feature memory pool, and a second convolutional network. The image region feature set and the hidden state at the current moment are further input into the visual attention mechanism to obtain the second visual attention feature at the current moment. The acquisition process of the second visual attention feature at the current moment can be expressed by the following formula:

[0102] a t =attention(h t ,R - )

[0103] Among them, a t represents the second visual attention feature at the current moment, h t Represents the hidden state at the current moment, R - It represents the feature set of the image region, and attention() represents the visual attention mechanism calculation function. There are many ways to calculate the visual attention mechanism function, such as vector addition, multiplication, and linear and nonlinear combination. It is recommended to use the combination of self-attention mechanism and common attention mechanism.

[0104] The multimodal feature memory pool is mainly used to store the second visual attention features and hidden states at multiple moments. There is an implicit semantic alignment relationship between the two features. Specifically, the multimodal feature memory pool needs to set up two memory pools of size q to store the second visual attention features a generated by the model at the previous q moments. t and the hidden state h t , the features in the two memory pools are represented as and

[0105] The hidden state and the second visual attention feature at the current moment are input into the modal feature memory pool to update the top-level features, so as to realize the update of the multimodal feature memory pool.

[0106] Expand the target word embedding feature q times to get a dimension of q×d n The feature is a one-dimensional feature set. Then the one-dimensional feature set and the hidden state memory pool in the current multimodal feature memory pool are combined. and the second visual attention feature memory pool in the current multimodal feature memory pool Splicing to form As the input of the second convolutional network; where q is the second preset number of times.

[0107] The second convolutional network has three layers. The first layer of the second convolutional network is a one-dimensional convolution operation, which is mainly used to aggregate each second visual attention feature in the second visual attention feature memory pool in the current multimodal feature memory pool and each hidden state in the hidden state memory pool in the current multimodal feature memory pool with the features in the one-dimensional feature set, respectively, to obtain interactively enhanced multimodal local representations. The second and third layers of the second convolutional network are both two-dimensional convolution operations, which are used to fully interact the aggregated local representations through two-dimensional convolution kernels to obtain the interactive context information at the current moment. The three-layer operation process of the second convolutional network can be expressed by the following formulas:

[0108]

[0109]

[0110]

[0111] Among them, W cl1 ,W cl2 ,W cl3 The convolution kernels are 1×3, 3×3, and 3×3 respectively, and B cl1 ,B cl2 ,B cl3 is the bias. Relu() and LeakyRelu() are both activation functions, setting the output channel of the last layer to 1.

[0112] The semantic local representation enhancement module is mainly used to obtain the semantic context information of the current moment through the third convolutional network and fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state of the previous moment, the first visual attention feature, the global image feature and the target word embedding feature.

[0113] The semantic local representation enhancement module includes a hidden state memory pool, a third convolutional network, and a fusion layer. Further, the hidden state memory pool is taken from the multimodal feature memory pool as one of the inputs of the long-term and short-term explicit modeling of the model in the convolutional network.

[0114] The target word embedding feature, the sum of the first visual attention feature and the global image feature, and the hidden state of the previous moment are expanded q times respectively, and three dimensions are obtained: q×d n The three-dimensional feature set is then concatenated with the hidden state memory pool in the current multimodal feature memory pool to form As the input of the third convolutional network.

[0115] The third convolutional network has three layers. The first layer is a one-dimensional convolution operation, which is used to aggregate each hidden state in the hidden state memory pool in the current multimodal feature memory pool with the features in the three-dimensional feature set to obtain a semantically enhanced multimodal local representation. The second and third layers of the third convolutional network are both two-dimensional convolution operations, which are used to fully interact the semantically enhanced multimodal local representation through a two-dimensional convolution kernel to obtain a semantically enhanced multimodal representation. The three-layer operation process of the third convolutional network can be expressed by the following formulas:

[0116]

[0117]

[0118]

[0119] Among them, W l1 ,W l2 ,W l3 The convolution kernels are 1×3, 3×3, and 3×3 respectively, and B l1 ,B l2 ,B l3 is the bias. Set the output channel of the third layer to 2, that is

[0120] The semantically enhanced multimodal representation is input into the fusion layer, and the semantically enhanced multimodal representation is further interacted to obtain the semantic context information of the current moment. The fusion layer calculation process can be expressed by the following formula:

[0121]

[0122]

[0123] in, express The i-th feature in , σ() represents the sigmoid function, represents the Hadamard product, W l Describe the model parameters for an image.

[0124] It should be noted that the specific structures of the first neural convolutional network, the second neural convolutional network and the third neural convolutional network are not only the above-mentioned forms, but can also be set to other reasonable structures, which will not be described in detail here.

[0125] The word prediction module is mainly used to predict the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and to determine whether to end the prediction process based on the description word at the current moment.

[0126] Specifically, the word prediction module includes a multilayer perceptron (MLP) and a classification layer. The multilayer perceptron consists of a linear mapping and a nonlinear function, and takes the interactive context information and the semantic context information at the current moment as input. Its calculation formula is as follows:

[0127]

[0128] Among them, [;] represents the concatenation operation, GLU() is a GLU function that divides the input features into two parts according to the dimension size. One part is used as information, and the other part is processed as a gate using the sigmoid function and outputs the multiplication result of the two.

[0129] The classification layer uses the output of the multi-layer perceptron as input to predict words. The formula is:

[0130] y t =softmax(W v out t )

[0131] in, d w is the vocabulary size, and softmax() is used to select the word with the highest weight in the vocabulary dimension as the current output word.

[0132] The word prediction module also needs to determine whether the description word obtained at the current moment meets the sentence end condition. If so, the prediction process is terminated, and all the description words predicted based on the same image to be trained or described are used to form a predicted description sentence. Otherwise, the visual local representation enhancement module re-selects the next target word embedding feature from the word embedding feature set to obtain the description word at the next moment.

[0133] The sentence ending condition is satisfied when the description word obtained at the current moment is the ending word, and when all description words predicted based on the same image to be trained or described at the current moment form a predicted description sentence that meets one of the two conditions of the maximum sentence length, it means that the sentence ending condition is satisfied.

[0134] The training process of the image description model includes the following steps:

[0135] Step S101 , obtaining a training data set, and grouping all training data in the training data set to obtain multiple training data groups, wherein each training data includes an image to be trained and text data corresponding to the image to be trained.

[0136] Step S102: taking the Mth training data set in the training data set as the training data set to be input.

[0137] The initial value of M is 1.

[0138] Step S103, training the current image description model based on the training data group to be input, and optimizing the training parameters of the current image description model based on the loss function.

[0139] The loss function is the cross entropy loss function, and the calculation formula of the loss function is as follows:

[0140]

[0141] Among them, I represents the current input image, l is the sentence length, Represents the first to t-1th words in the input sentence of the training data, L xe represents the cross entropy loss function, which is used to maximize the probability of the model outputting the text sentence of the dataset, and p represents the probability distribution of the image description model when predicting the description sentence.

[0142] Step S104, determine whether M is equal to a preset threshold, if so, go to step S105, otherwise, add 1 to M and go to step S102.

[0143] The preset threshold is set as the number of training data groups in the training data set.

[0144] Step S105, calculate the evaluation index of the training data set in this round of training, and determine whether the evaluation index meets the training termination condition. If so, go to step S106, otherwise initialize M to 1 and go to step S102 to perform a new round of training on the current image description model.

[0145] Specifically, the calculation process of the evaluation index of the training data set in a single round of training includes: taking the text data corresponding to the image to be trained as the reference value, calculating the preset evaluation index of each prediction description sentence corresponding to the image to be trained in this round of training, and calculating the average value of the preset evaluation index of all prediction description sentences corresponding to the images to be trained in this round of training, as the evaluation index of the training data set in this round of training. The preset evaluation index calculation method is to take the text data corresponding to the image to be trained in the training data as the reference value, and calculate the score of the output sentence from the aspects of word frequency, word meaning, etc. through the evaluation index (such as BLEU, CIDEr, etc.). It should be noted that other reasonable evaluation index calculation methods can also be selected.

[0146] Further, the process of judging whether the evaluation index meets the training termination condition includes: starting from the current round of training, selecting the evaluation index of the preset rounds of training as the evaluation parameter, and calculating the difference between two adjacent evaluation parameters as the training growth value, judging whether all the training growth values ​​are less than the preset threshold value, if less than, it means that the evaluation index corresponding to the current round of training meets the training termination condition, otherwise, it means that the evaluation index corresponding to the current round of training does not meet the training termination condition. The preset rounds are preferably 5.

[0147] Step S106, stop training and use the current image description model as the trained image description model

[0148] Figure 2 FIG. 1 shows a schematic diagram of a process of training a single training data by an image description model in Embodiment 1 of the present invention; Figure 2 As shown, during the training of the current image description model, the training process of the image description model for a single training data includes the following steps:

[0149] Step S301, the multimodal feature encoding module obtains the corresponding visual features based on the image to be trained in the training data, and obtains the word embedding feature set based on the text data corresponding to the image to be trained, and initializes the hidden state at the last moment, the memory state at the last moment, the second visual attention feature at the last moment, and the semantic context information at the last moment to zero, wherein the visual features include an image area feature set and a global image feature.

[0150] The process of the multimodal feature encoding module processing the training data is as described above, and will not be elaborated on here.

[0151] In step S302, the visual local representation enhancement module selects a target word embedding feature from the word embedding feature set, inputs the visual feature and the semantic context information of the previous moment into the attention mechanism to obtain a first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state of the previous moment, the memory state of the previous moment and the second visual attention feature of the previous moment, obtains the hidden state of the current moment through the first convolutional network and the gate mechanism.

[0152] The specific process of the visual local representation enhancement module obtaining the hidden state at the current moment is also as described above, and will not be elaborated on here.

[0153] In step S303, the multimodal interaction enhancement module inputs the visual features and the hidden state at the current moment into the visual attention mechanism to obtain the second visual attention features at the current moment, and updates the current multimodal feature memory pool based on the second visual attention features at the current moment and the hidden state at the current moment, and obtains the interaction context information at the current moment through the second convolutional network based on the target word embedding features and the current multimodal feature memory pool.

[0154] The specific process of the multimodal interaction enhancement module obtaining the interaction context information at the current moment is also as described above, and will not be elaborated on here.

[0155] In step S304, the semantic local representation enhancement module obtains the semantic context information of the current moment through the third convolutional network and the fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature.

[0156] The specific process of the semantic local representation enhancement module obtaining the semantic context information at the current moment is also as described above, and will not be elaborated on here.

[0157] Step S305, the word prediction module predicts the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and determines whether the description word at the current moment meets the sentence end condition. If so, all the description words predicted based on the training data form a predicted description sentence, otherwise go to step S302.

[0158] Specifically, the specific process of the word prediction module obtaining the description word at the current moment is also as described above, and it will not be described in detail here. After obtaining the description word at the current moment, it is necessary to determine whether the description word at the current moment meets the sentence end condition. If it does, the prediction process is terminated, and a predicted description sentence is formed based on all the description words predicted by the training data. Otherwise, go to step S301 so that the visual local representation enhancement module can select the target word embedding feature from the word embedding feature set, and then obtain the description word at the next moment.

[0159] Step S306, ending the prediction process.

[0160] The image description method based on local representation enhancement provided by the embodiment of the present invention adopts a recurrent convolutional network for multimodal tasks in the image description model. The model effectively introduces fine-grained information of different modalities by using the convolution operation characteristics, strengthens the model's ability to capture multimodal local representations, improves the degree of interaction between multimodal information, and improves the quality of generated sentences. Furthermore, the model adopts a visual local representation enhancement module to allow visual fine-grained features to fully interact with multimodal features, and to allow multimodal local representations enhanced with visual detail information to fully interact with each other; a semantic local representation enhancement module is adopted to allow hidden states generated at past moments to fully interact with multimodal features, and to allow multimodal local representations enhanced with semantics to fully interact with each other. A convolutional network is adopted to allow features containing similar semantics in visual and language modalities at the same moment to fully interact with multimodal features, and to allow multimodal local representations enhanced with semantic interactions to fully interact with each other; the collaborative work of the three convolutional networks can enrich the detailed information captured by the model, enhance the model's understanding of multimodal features, and thus improve the quality of the overall description sentence.

[0161] Embodiment 2

[0162] In order to solve the technical key point confirmation problem existing in the prior art, an embodiment of the present invention provides an image description device based on local representation enhancement.

[0163] The image description device based on local representation enhancement in the embodiment of the present invention includes an image description module.

[0164] The image description module is used to obtain the image to be described, and input the image to be described into the trained image description model to obtain a description sentence of the image to be described.

[0165] The image description model includes a multimodal feature encoding module, a visual local representation enhancement module, a multimodal interaction enhancement module, a semantic local representation enhancement module and a word prediction module.

[0166] The multimodal feature encoding module is used to obtain visual features from the image to be trained or described, and obtain a word embedding feature set based on text data, and initialize the hidden state of the previous moment, the memory state of the previous moment, the second visual attention feature of the previous moment, and the semantic context information of the previous moment to zero, where the visual features include an image area feature set and a global image feature.

[0167] The visual local representation enhancement module is used to select the target word embedding feature from the word embedding feature set, or use the word embedding feature corresponding to the description word predicted at the previous moment as the target word embedding feature, input the image area feature set and the semantic context information of the previous moment into the attention mechanism, obtain the first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state at the previous moment, the memory state at the previous moment and the second visual attention feature at the previous moment, obtain the hidden state at the current moment through the first convolutional network and the gate mechanism.

[0168] The multimodal interaction enhancement module is used to input the image area feature set and the hidden state at the current moment into the visual attention mechanism, obtain the second visual attention feature at the current moment, and update the multimodal feature memory pool based on the second visual attention feature at the current moment and the hidden state at the current moment, and obtain the interaction context information at the current moment through the second convolutional network based on the target word embedding feature and the current multimodal feature memory pool.

[0169] The semantic local representation enhancement module is used to obtain the semantic context information of the current moment through the third convolutional network and fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state of the previous moment, the first visual attention feature, the global image feature and the target word embedding feature.

[0170] The word prediction module is used to predict the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and to determine whether to end the prediction process based on the description word at the current moment.

[0171] The image description device based on local representation enhancement provided by the embodiment of the present invention adopts a recurrent convolutional network containing multimodal tasks in the image description model. The model effectively introduces fine-grained information of different modalities by using the convolution operation characteristics, strengthens the model's ability to capture multimodal local representations, improves the degree of interaction between multimodal information, and improves the quality of generated sentences. Further, the model adopts a visual local representation enhancement module to allow visual fine-grained features to fully interact with multimodal features, and to allow multimodal local representations enhanced with visual detail information to fully interact with each other; adopts a semantic local representation enhancement module to allow hidden states generated at past moments to fully interact with multimodal features, and to allow multimodal local representations enhanced with semantics to fully interact with each other. A convolutional network is used to allow features containing similar semantics in visual and language modalities at the same moment to fully interact with multimodal features, and to allow multimodal local representations enhanced with semantic interactions to fully interact with each other; the collaborative work of the three convolutional networks can enrich the detailed information captured by the model, enhance the model's understanding of multimodal features, and thus improve the quality of the overall description sentence.

[0172] Embodiment 3

[0173] In order to solve the above-mentioned technical problems existing in the prior art, an embodiment of the present invention further provides a storage medium storing a computer program, which can implement all the steps of the image description method based on local representation enhancement in embodiment 1 when executed by a processor.

[0174] The specific steps of the image description method based on local representation enhancement and the beneficial effects obtained by applying the readable storage medium provided by the embodiment of the present invention are the same as those in the first embodiment, and will not be described in detail here.

[0175] It should be noted that the storage medium includes various media that can store program codes, such as ROM, RAM, magnetic disk or optical disk.

[0176] Embodiment 4

[0177] In order to solve the above technical problems existing in the prior art, an embodiment of the present invention further provides a terminal.

[0178] Figure 3 The schematic diagram of the terminal structure of the fourth embodiment of the present invention is shown. Figure 3 In this embodiment, the terminal includes a processor and a memory connected to each other; the memory is used to store computer programs, and the processor is used to execute the computer programs stored in the memory, so that when the terminal executes, all steps of the image description method based on local representation enhancement in embodiment 1 can be implemented.

[0179] The specific steps of the image description method based on local representation enhancement and the beneficial effects obtained by applying the terminal provided by the embodiment of the present invention are the same as those in the first embodiment, and will not be described in detail here.

[0180] It should be noted that the memory may include random access memory (RAM) and may also include non-volatile memory, such as at least one disk storage. Similarly, the processor may also be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components.

[0181] Although the embodiments disclosed in the present invention are as above, the contents described are only embodiments adopted for facilitating the understanding of the present invention and are not intended to limit the present invention. Any technician in the technical field to which the present invention belongs can make any modifications and changes in the form and details of the implementation without departing from the spirit and scope disclosed in the present invention, but the protection scope of the present invention shall still be subject to the scope defined in the attached claims.

Claims

1. An image description method based on local representation enhancement, comprising: Acquire an image to be described, and input the image to be described into a trained image description model to obtain a description sentence of the image to be described; The image description model includes a multimodal feature encoding module, a visual local representation enhancement module, a multimodal interaction enhancement module, a semantic local representation enhancement module and a word prediction module; The multimodal feature encoding module is used to obtain visual features from the image to be trained or the image to be described, and obtain a word embedding feature set based on text data, and initialize the hidden state at the last moment, the memory state at the last moment, the second visual attention feature at the last moment, and the semantic context information at the last moment to zero, wherein the visual features include an image region feature set and a global image feature; The visual local representation enhancement module is used to select a target word embedding feature from the word embedding feature set, or use the word embedding feature corresponding to the description word predicted at the previous moment as the target word embedding feature, input the image area feature set and the semantic context information at the previous moment into the attention mechanism, obtain a first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state at the previous moment, the memory state at the previous moment and the second visual attention feature at the previous moment, obtain the hidden state at the current moment through the first convolutional network and the gate mechanism; The multimodal interaction enhancement module is used to input the image region feature set and the hidden state at the current moment into the visual attention mechanism, obtain the second visual attention feature at the current moment, and update the multimodal feature memory pool based on the second visual attention feature at the current moment and the hidden state at the current moment, and obtain the interaction context information at the current moment through the second convolutional network based on the target word embedding feature and the current multimodal feature memory pool; The semantic local representation enhancement module is used to obtain the semantic context information at the current moment through the third convolutional network and the fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature; The word prediction module is used to predict the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and to determine whether to end the prediction process based on the description word at the current moment.

2. The image description method according to claim 1, characterized in that: The image description model training process includes: Acquire a training data set, and group all training data in the training data set to obtain multiple groups of training data, wherein each of the training data includes an image to be trained and text data corresponding to the image to be trained; Using the Mth training data set in the training data set as the training data set to be input; The current image description model is trained based on the training data group to be input, and the training parameters of the current image description model are optimized based on the loss function, and it is determined whether M is equal to a preset threshold value. If so, the evaluation index of the training data set in this round of training is calculated, and it is determined whether the evaluation index meets the training termination condition. If so, the training is stopped, and the current image description model is used as the trained image description model. If the evaluation index does not meet the training termination condition, M is initialized to 1, and the Mth group of training data in the training data set is re-used as the training data group to be input, so as to continue a new round of training for the current image description model. If M is not equal to the preset threshold value, M is increased by 1, and the Mth group of training data in the training data set is re-used as the training data group to be input, so as to continue training the current image description model. Among them, the initial value of M is 1, the preset threshold is the number of training data groups in the training data set, and one round of training is completed when all training data groups in the training data set are trained once in sequence.

3. The image description method according to claim 2, characterized in that: Calculating the evaluation index of the training data set in the current round of training includes: taking the text data corresponding to the image to be trained as a reference value, calculating the preset evaluation index of the prediction description sentence corresponding to each image to be trained in the current round of training, and calculating the average value of the preset evaluation indexes of the prediction description sentences corresponding to all the images to be trained in the current round of training as the evaluation index of the training data set in the current round of training; Judging whether the evaluation index meets the training termination condition includes: taking the current round of training as the starting point, selecting the evaluation index of a preset number of rounds of training in sequence as the evaluation parameter, and calculating the difference between two adjacent evaluation parameters as the training growth value, and judging whether all the training growth values ​​are less than a preset threshold value. If so, it means that the evaluation index corresponding to the current round of training meets the training termination condition, otherwise, it means that the evaluation index corresponding to the current round of training does not meet the training termination condition.

4. The image description method according to claim 2, characterized in that: During the training of the current image description model, the training process of the image description model for a single training data includes: The multimodal feature encoding module obtains corresponding visual features based on the image to be trained in the training data, obtains a word embedding feature set based on the text data corresponding to the image to be trained, and initializes the hidden state at the last moment, the memory state at the last moment, the second visual attention feature at the last moment, and the semantic context information at the last moment to zero, wherein the visual features include an image region feature set and a global image feature; The visual local representation enhancement module selects a target word embedding feature from the word embedding feature set, inputs the visual feature and the semantic context information at the previous moment into the attention mechanism to obtain a first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state at the previous moment, the memory state at the previous moment, and the second visual attention feature at the previous moment, obtains the hidden state at the current moment through the first convolutional network and the gate mechanism; The multimodal interaction enhancement module inputs the visual feature and the hidden state at the current moment into the visual attention mechanism to obtain the second visual attention feature at the current moment, and updates the current multimodal feature memory pool based on the second visual attention feature at the current moment and the hidden state at the current moment, and obtains the interaction context information at the current moment through the second convolutional network based on the target word embedding feature and the current multimodal feature memory pool; The semantic local representation enhancement module obtains the semantic context information at the current moment through the third convolutional network and the fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature; The word prediction module predicts the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and determines whether the description word at the current moment meets the sentence end condition. If so, the prediction process is terminated, and a predicted description sentence is formed based on all the description words predicted by the training data. Otherwise, the visual local representation enhancement module re-selects the target word embedding feature from the word embedding feature set to obtain the description word at the next moment.

5. The image description method according to claim 4, characterized in that: The visual local representation enhancement module inputs the visual features and the semantic context information of the previous moment into the attention mechanism to obtain a first visual attention feature, and based on the first visual attention feature, the target word embedding feature, the hidden state of the previous moment, the memory state of the previous moment and the second visual attention feature of the previous moment, obtains the hidden state of the current moment through the first convolutional network and the gate mechanism, including: Calculating the relevance of each of the image region features in the image region feature set with the semantic context information at the previous moment, and selecting a first preset number of image region features from the image region feature set in descending order of relevance to form a selected image region feature set; Acquire a first visual attention feature based on the image region feature set and the correlation degree of each of the image region features in the image region feature set; Based on the target word embedding feature, the sum of the first visual attention feature and the global image feature, the second visual attention feature at the previous moment, and the hidden state at the previous moment, the four-dimensional feature set is respectively expanded a first preset number of times to obtain a four-dimensional feature set, and the four-dimensional feature set and the selected image area feature set are concatenated as the input of the first convolutional network; The first layer of the first convolutional network aggregates each selected image region feature in the selected image region feature set with features in the four-dimensional feature set to obtain a multimodal local representation with enhanced visual detail information, and the second and third layers of the first convolutional network perform full information interaction on the multimodal local representation with enhanced visual detail information through a two-dimensional convolution kernel to obtain a visually enhanced multimodal representation; Based on the multimodal representation of visual enhancement, the hidden state at the previous moment and the memory state at the previous moment, the hidden state at the current moment is obtained through a gate mechanism.

6. The image description method according to claim 4, characterized in that: The multimodal interaction enhancement module obtains the interaction context information at the current moment through a second convolutional network based on the target word embedding feature and the current multimodal feature memory pool, including: Expanding the target word embedding feature by a second preset number of times to obtain a one-dimensional feature set, and concatenating the one-dimensional feature set, the hidden state memory pool in the current multimodal feature memory pool, and the second visual attention feature memory pool in the current multimodal feature memory pool as input to the second convolutional network; The first layer of the second convolutional network aggregates each second visual attention feature in the second visual attention feature memory pool in the current multimodal feature memory pool and each hidden state in the hidden state memory pool in the current multimodal feature memory pool with the features in the one-dimensional feature set to obtain an interactively enhanced multimodal local representation, and the second and third layers of the second convolutional network respectively perform full information interaction on the interactively enhanced multimodal local representation through a two-dimensional convolution kernel to obtain multimodal interactively enhanced contextual information.

7. The image description method according to claim 4, characterized in that: The semantic local representation enhancement module obtains the semantic context information at the current moment based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature through the third convolutional network and the fusion layer, including: Expand the target word embedding feature, the sum of the first visual attention feature and the global image feature, and the hidden state at the previous moment by a second preset number of times to obtain a three-dimensional feature set, and concatenate the three-dimensional feature set with the hidden state memory pool in the current multimodal feature memory pool as an input of the third convolutional network; The first layer of the third convolutional network aggregates each hidden state in the hidden state memory pool in the current multimodal feature memory pool with the features in the three-dimensional feature set to obtain a semantically enhanced multimodal local representation, and the second and third layers of the third convolutional network perform full information interaction on the semantically enhanced multimodal local representation through a two-dimensional convolution kernel to obtain a semantically enhanced multimodal representation; The semantically enhanced multimodal representation is input into the fusion layer for further interaction to obtain the semantic context information at the current moment.

8. An image description device based on local representation enhancement, characterized in that: Includes image description module; The image description module is used to obtain an image to be described, and input the image to be described into a trained image description model to obtain a description sentence of the image to be described; The image description model includes a multimodal feature encoding module, a visual local representation enhancement module, a multimodal interaction enhancement module, a semantic local representation enhancement module and a word prediction module; The multimodal feature encoding module is used to obtain visual features from the image to be trained or the image to be described, and obtain a word embedding feature set based on text data, and initialize the hidden state at the last moment, the memory state at the last moment, the second visual attention feature at the last moment, and the semantic context information at the last moment to zero, wherein the visual features include an image region feature set and a global image feature; The visual local representation enhancement module is used to select a target word embedding feature from the word embedding feature set, or use the word embedding feature corresponding to the description word predicted at the previous moment as the target word embedding feature, input the image area feature set and the semantic context information at the previous moment into the attention mechanism, obtain a first visual attention feature, and based on the first visual attention feature, the global image feature, the target word embedding feature, the hidden state at the previous moment, the memory state at the previous moment and the second visual attention feature at the previous moment, obtain the hidden state at the current moment through the first convolutional network and the gate mechanism; The multimodal interaction enhancement module is used to input the image region feature set and the hidden state at the current moment into the visual attention mechanism, obtain the second visual attention feature at the current moment, and update the multimodal feature memory pool based on the second visual attention feature at the current moment and the hidden state at the current moment, and obtain the interaction context information at the current moment through the second convolutional network based on the target word embedding feature and the current multimodal feature memory pool; The semantic local representation enhancement module is used to obtain the semantic context information at the current moment through the third convolutional network and the fusion layer based on the hidden state memory pool in the current multimodal feature memory pool, the hidden state at the previous moment, the first visual attention feature, the global image feature and the target word embedding feature; The word prediction module is used to predict the description word at the current moment based on the interaction context information at the current moment and the semantic context information at the current moment, and to determine whether to end the prediction process based on the description word at the current moment.

9. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the image description method based on local representation enhancement according to any one of claims 1 to 7 is implemented.

10. A terminal, characterized in that: include: A processor and a memory, wherein the memory is communicatively connected to the processor; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory, so that the terminal performs the image description method based on local representation enhancement as claimed in any one of claims 1 to 7.

Images