Image description generation method, device, equipment and medium based on transfer learning

Through the image description generation method based on transfer learning, the attribute generation model and text generation model are used to obtain high-level semantic information in the image, which solves the problems of poor generation effect and difficult training in the prior art, and achieves better image description generation The effect and model training difficulty are reduced.

CN114693997BActive Publication Date: 2024-11-08PING AN TECH (SHENZHEN) CO LTD
View PDF 3 Cites 0 Cited by

Patent Information

Application Number
CN202210415615.5
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2022-04-20
Publication Date
2024-11-08
Estimated Expiration
2042-04-20

AI Technical Summary

Technical Problem

The existing image description generation methods cannot pay attention to the high-level semantic information in the image, resulting in poor generation effect and poor text readability, and model training is difficult.

Method used

Through a transfer learning method, high-level semantic information in the target image is obtained, and image description generation is generated using attribute generation model and text generation model. The training data set includes multiple training images and corresponding description statements, build attribute dictionaries and fine-tune multiple categories Classifier, reduces the difficulty of training the attribute generation model.

Benefits of technology

It improves the effect of image description generation, the readability and accuracy of the generated text, and reduces the difficulty of training the attribute generation model.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN114693997B_ABST
    Figure CN114693997B_ABST
Patent Text Reader

Abstract

The present invention relates to artificial intelligence technology, and provides a method, device, equipment and medium for generating image description based on transfer learning, the method comprising: obtaining a target image; preprocessing the target image to determine a target area; inputting the target area into a trained attribute generation model to obtain an attribute target vector; inputting the attribute target vector into a trained text generation model to obtain a prediction sentence. According to the scheme provided by an embodiment of the present invention, an attribute target vector containing high-level semantic information in a target image can be obtained through an attribute generation model, thereby improving the effect of image description generation, and the difficulty of training the attribute generation model can be reduced through transfer learning.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to, but is not limited to, the field of artificial intelligence technology, and in particular to a method, device, equipment and medium for generating image descriptions based on transfer learning. Background Art

[0002] Image captioning, as a comprehensive research direction combining natural language processing (NLP) and computer vision (CV), has made a lot of progress and applications.

[0003] At present, the image description generation method that combines Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) can support end-to-end training, pay attention to more image information, obtain better textual image attention information and solve the problem of RNN forgetting long sentences; however, the existing image description generation method cannot pay attention to the high-level semantic information in the image, resulting in poor image description generation effect and poor readability of the generated text; in addition, in the training process of the image description generation model, the model training is difficult. Summary of the invention

[0004] The following is a summary of the subject matter described in detail herein. This summary is not intended to limit the scope of the claims.

[0005] The embodiments of the present invention provide a method, apparatus, device and medium for generating image description based on transfer learning, which can obtain an attribute target vector containing high-level semantic information in a target image through an attribute generation model, thereby improving the effect of image description generation, and can reduce the training difficulty of the attribute generation model through transfer learning.

[0006] In a first aspect, the present invention also provides an image description generation method based on transfer learning, the method comprising: obtaining a target image; preprocessing the target image to determine a target area; inputting the target area into a trained attribute generation model to obtain an attribute target vector; inputting the attribute target vector into a trained text generation model to obtain a prediction sentence; wherein the attribute generation model is trained by the following steps: obtaining a training data set, wherein the training data set includes a plurality of training images and description sentences corresponding to each of the training images, the description sentences including a plurality of attribute words; constructing an attribute dictionary based on the attribute words, and generating a prediction sentence based on the description sentences and the attribute words; A property dictionary is used to determine the real attribute label corresponding to the description sentence; transfer learning is performed on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image; based on the attribute dictionary and a preset classification algorithm, the multi-category classifier is fine-tuned, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features; the training image and the real attribute label are used as training data to train the fine-tuned pre-trained model to obtain an attribute generation model.

[0007] In some embodiments, constructing an attribute dictionary based on the attribute words includes: determining the frequency of occurrence of the attribute words in the description sentence; screening the attribute words based on the frequency of occurrence and a preset dictionary capacity value to determine target attribute words; and establishing an attribute dictionary based on the target attribute words.

[0008] In some embodiments, the method of using the training image and the real attribute label as training data to train the fine-tuned pre-trained model to obtain an attribute generation model includes: inputting the training image into the fine-tuned pre-trained model to determine an attribute training vector; determining a first model loss function according to the real attribute label and the attribute training vector; and updating the model parameters of the fine-tuned pre-trained model based on the first model loss function to obtain an attribute generation model; wherein the formula of the first model loss function is:

[0009]

[0010] Wherein, J is the first model loss determined by the first model loss function, N is the number of the training images, and c is the dictionary capacity value; p ij is the probability sequence p corresponding to the i-th training image i The jth element in p i =[p i1 ,p i2,...,p ic ], p ij It refers to the probability that the jth target attribute word exists in the i-th training image; y ij is the jth element in the true attribute label corresponding to the i-th training image, y ij The calculation steps are as follows: when the i-th training image has the j-th target attribute word, then y ij =1, otherwise y ij =0, 1≤i≤N, and i and N are both positive integers.

[0011] In some embodiments, the preprocessing of the target image to determine the target area includes: based on a normalized cut algorithm and a preset number of clusters, segmenting the target image to obtain multiple clusters, wherein each cluster includes at least one sub-area; for each cluster, based on the target image and a preset scoring algorithm, scoring all sub-areas of the cluster to determine a scoring value, wherein the scoring value corresponds to the sub-area one-to-one; for each cluster, based on the scoring value and a preset number of attribute areas, filtering out a target attribute area from all sub-areas of the cluster; and determining the target area according to the target image and the target attribute area.

[0012] In some embodiments, the text generation model is trained by the following steps: obtaining a training data set, wherein the training data set includes a plurality of training images and descriptive words corresponding to each of the training images; inputting the training data set into the trained attribute generation model to obtain a plurality of attribute training vectors; processing the attribute training vectors at n time steps based on a preset gated recurrent network to obtain prediction words corresponding to the n time steps, wherein the gated recurrent network includes an update gate and a reset gate; obtaining a prediction sentence based on the prediction word; determining a second model loss function based on the attribute training vector and the prediction sentence; updating the model parameters of the gated recurrent network based on the second model loss function to obtain a text generation model; wherein the gated recurrent network The input of the recurrent network at the t-th time step includes the hidden state of the gated recurrent network at the t-1th time step and the corresponding prediction word at the t-th time step, the prediction word corresponding to the gated recurrent network at the t-th time step is determined by the output of the gated recurrent network at the t-1th time step, the hidden state of the gated recurrent network at the t-1th time step is updated by the update gate and the reset gate of the gated recurrent network at the t-1th time step, the update gate and the reset gate of the gated recurrent network at the t-th time step are determined by the hidden state of the gated recurrent network at the t-1th time step and the corresponding prediction word at the t-th time step, the t-th time step is any one of the n time steps, 1≤t≤n, and t and n are both positive integers.

[0013] In some embodiments, the gated recurrent network processes the attribute training vector at n time steps by the following formula:

[0014] z t =σ(h t-1 ·W zh +x t ·W zx ]),

[0015] r t =σ(h t-1 ·W rh +x t ·W rx ]),

[0016]

[0017]

[0018] Among them, z t is the update gate of the gated recurrent network at the tth time step, r t is the reset gate of the gated recurrent network at the tth time step, is the candidate hidden state of the gated recurrent network at the tth time step, h t is the hidden state of the gated recurrent network at the tth time step; W zh , W zx , W rh , W rx , W o These are all preset learning parameters, σ represents the sigmoid function, ⊙ represents the calculation of dot product, and x t =W s S t , W s is the preset learnable word vector parameter matrix, S t is the predicted word corresponding to the gated recurrent network at the tth time step.

[0019] In some embodiments, the formula of the second model loss function is:

[0020]

[0021] Wherein, C is the second model loss determined by the second model loss function, N is the number of the training images, i is the order of the training images input into the attribute generation model, 1≤i≤N, and i and N are both positive integers, S (i) is the predicted word corresponding to the i-th training image, V att (I (i) ) is the attribute training vector corresponding to the i-th training image, L (i) For the S (i) The corresponding sentence length, y t is the output of the gated recurrent network at the tth time step, is the polynomial used for regularization, log p(S (i) |V att (I (i) )) is calculated as follows:

[0022]

[0023] Where S = {S1,...,S L}, S∈R K , S is the prediction statement, S L is the Lth predicted word, L is the S L The corresponding sentence length, R is the real number domain, K is the dictionary dimension of the attribute dictionary, S t is the predicted word corresponding to the gated recurrent network at the tth time step, S 1:t-1 is the prediction statement corresponding to the gated recurrent network at the t-1th time step, Vatt (I) is the attribute training vector, p(S t |S 1:t-1 ,V att (I)) refers to the V att (I) and the S 1:t-1 Produce the S t probability.

[0024] In a second aspect, the present invention also provides an image description generation device based on transfer learning, the device comprising: an acquisition unit, used to acquire a target image; a preprocessing unit, used to preprocess the target image and determine a target area; an attribute generation unit, used to input the target area into a trained attribute generation model to obtain an attribute target vector; a description generation unit, used to input the attribute target vector into a trained text generation model to obtain a prediction sentence;

[0025] The attribute generation model is trained by the following steps: obtaining a training data set, wherein the training data set includes a plurality of training images and a description statement corresponding to each of the training images, and the description statement includes a plurality of attribute words; constructing an attribute dictionary based on the attribute words, and determining the real attribute label corresponding to the description statement according to the description statement and the attribute dictionary; performing transfer learning on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training images; fine-tuning the multi-category classifier based on the attribute dictionary and a preset classification algorithm, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features; using the training images and the real attribute labels as training data, training the fine-tuned pre-trained model to obtain the attribute generation model.

[0026] In a third aspect, an embodiment of the present invention further provides an electronic device, comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the image description generation method based on transfer learning as described in the first aspect above is implemented.

[0027] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are used to execute the image description generation method based on transfer learning as described in the first aspect above.

[0028] The embodiment of the present invention includes: obtaining a target image; preprocessing the target image to determine a target area; inputting the target area into a trained attribute generation model to obtain an attribute target vector; inputting the attribute target vector into a trained text generation model to obtain a prediction sentence; wherein the attribute generation model is trained by the following steps: obtaining a training data set, wherein the training data set includes a plurality of training images and description sentences corresponding to each of the training images, and the description sentences include a plurality of attribute words; constructing an attribute dictionary based on the attribute words, and determining a real attribute label corresponding to the description sentence according to the description sentence and the attribute dictionary; performing transfer learning on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image; fine-tuning the multi-category classifier based on the attribute dictionary and a preset classification algorithm, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features; using the training image and the real attribute label as training data, training the fine-tuned pre-trained model to obtain an attribute generation model. According to the solution provided by the embodiment of the present invention, transfer learning is performed on a preset image classification model, and then the attribute dictionary constructed using attribute words is fine-tuned and trained to obtain an attribute generation model. The target area obtained by preprocessing the target image is input into the attribute generation model, and then the attribute target vector containing high-level semantic information in the target image is obtained through the attribute generation model, thereby improving the effect of image description generation, and reducing the training difficulty of the attribute generation model through transfer learning.

[0029] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are used to provide a further understanding of the technical solution of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the technical solution of the present invention and do not constitute a limitation on the technical solution of the present invention.

[0031] Figure 1 is a flowchart of a method for generating image description based on transfer learning provided by an embodiment of the present invention;

[0032] Figure 2 is a flow chart of a method for training an attribute generation model provided by another embodiment of the present invention;

[0033] Figure 3is a flow chart of establishing an attribute dictionary provided by another embodiment of the present invention;

[0034] Figure 4 is a flow chart of obtaining an attribute generation model provided by another embodiment of the present invention;

[0035] Figure 5 is a flow chart of determining a target area provided by another embodiment of the present invention;

[0036] Figure 6 is a flowchart of a text generation model training method provided by another embodiment of the present invention;

[0037] Figure 7 is a system block diagram of an image description generation method based on transfer learning provided by another embodiment of the present invention;

[0038] Figure 8 is a structural diagram of an image description generating device based on transfer learning provided by another embodiment of the present invention;

[0039] Fig. 9 It is a structural diagram of an electronic device provided by another embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0041] In the description of the present invention, “several” means one or more, “more” means more than two, “greater than”, “less than”, “exceed”, etc. are understood as not including the number, and “above”, “below”, “within”, etc. are understood as including the number.

[0042] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims or the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0043] Image captioning, as a comprehensive research direction combining natural language processing (NLP) and computer vision (CV), has made a lot of progress and applications.

[0044] At present, the image description generation method that combines Convolutional Neural Networks (CNN) and Recurrent Neural Networks (RNN) can support end-to-end training, pay attention to more image information, obtain better textual image attention information and solve the problem of RNN forgetting long sentences; however, the existing image description generation method cannot pay attention to the high-level semantic information in the image, resulting in poor image description generation effect and poor readability of the generated text; in addition, in the training process of the image description generation model, the model training is difficult.

[0045] In order to solve the problems of poor image description generation and great difficulty in model training, the present invention provides a method, device, equipment and medium for image description generation based on transfer learning, the method comprising: obtaining a target image; preprocessing the target image to determine a target area; inputting the target area into a trained attribute generation model to obtain an attribute target vector; inputting the attribute target vector into a trained text generation model to obtain a prediction sentence; wherein the attribute generation model is trained by the following steps: obtaining a training data set, wherein the training data set comprises a plurality of training images and description sentences corresponding to each training image, the description sentence comprising a plurality of attribute words; based on the attribute words Construct an attribute dictionary, and determine the real attribute label corresponding to the description sentence according to the description sentence and the attribute dictionary; perform transfer learning on the preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image; fine-tune the multi-category classifier based on the attribute dictionary and the preset classification algorithm, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features; use the training image and the real attribute label as training data to train the fine-tuned pre-trained model to obtain an attribute generation model. According to the scheme provided by the embodiment of the present invention, by performing transfer learning on the preset image classification model, fine-tuning the attribute dictionary constructed by using the attribute words, and training to obtain the attribute generation model, the target area obtained by pre-processing the target image is input into the attribute generation model, and then the attribute target vector containing high-level semantic information in the target image is obtained through the attribute generation model, thereby improving the effect of image description generation, and reducing the training difficulty of the attribute generation model through transfer learning.

[0046] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence.

[0047] Artificial intelligence is the study of the design principles and implementation methods of various intelligent machines, so that the machines have the functions of perception, reasoning and decision-making. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning. With the research and progress of artificial intelligence technology, artificial intelligence has been studied and applied in many fields, such as common smart homes, smart customer service, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, robots, smart medical care, smart cities, etc. I believe that with the development of technology, artificial intelligence will be applied in more fields and play an increasingly important role.

[0048] Machine Learning (ML) is a multi-disciplinary subject involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, etc. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance.

[0049] Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning and other technologies.

[0050] First, several nouns involved in the present invention are analyzed:

[0051] Convolutional Neural Networks (CNN) is a type of feedforward neural network that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0052] Recurrent Neural Network (RNN) is a type of recurrent neural network that takes sequence data as input, recursively in the direction of sequence evolution, and all nodes (recurrent units) are connected in a chain-like manner;

[0053] Transfer learning, given a source domain D consisting of a feature space X and a marginal probability distribution P(X) s and learning task T s , and the target domain D which also consists of feature space and marginal probability distribution t and learning task T t , the purpose of transfer learning is to use D s and T s The knowledge in the target domain D t The objective function f T () process, note D s With D t Not equal, T s With T t Not equal;

[0054] The MS COCO dataset, the full name of which is Microsoft Common Objects in Context, is a large and rich dataset for object detection, segmentation, and captioning. This dataset aims at scene understanding and is mainly captured from complex daily scenes. The objects in the images are calibrated through precise segmentation.

[0055] Gated Recurrent Unit (GRU) is a gating mechanism in recurrent neural networks. Similar to other gating mechanisms, it aims to solve the gradient vanishing / exploding problem in standard RNNs while retaining the long-term information of the sequence. GRU contains a reset gate and an update gate.

[0056] Normalized Cut is a cluster grouping technique that is widely used in data processing and image processing. The idea of ​​image segmentation is to regard an image as a graph, calculate the weighted graph, and then segment it into regions with the same characteristics (texture, color, brightness, etc.);

[0057] Intersection-over-Union (IoU) is the overlap ratio between the candidate bounding box generated in target detection and the original marked box (ground truth bounding box), that is, the ratio of their intersection to their union;

[0058] Stochastic Gradient Descent (SGD), also known as incremental gradient descent, is an iterative method for optimizing a differentiable objective function; this method iteratively updates weights and biases by computing the gradient of the loss function over mini-batches of data.

[0059] The embodiments of the present invention are further described below in conjunction with the accompanying drawings.

[0060] like Figure 1 and Figure 2 As shown, Figure 1 is a flowchart of a method for generating image description based on transfer learning provided by an embodiment of the present invention. Figure 2 A flowchart of an attribute generation model training method provided by an embodiment of the present invention, in which the image description generation method based on transfer learning includes but is not limited to the following steps S110 to S140:

[0061] Step S110, acquiring a target image;

[0062] Step S120, preprocessing the target image to determine the target area;

[0063] Step S130, inputting the target area into the trained attribute generation model to obtain an attribute target vector;

[0064] Step S140, inputting the attribute target vector into the trained text generation model to obtain a predicted sentence;

[0065] The attribute generation model training method of step S130 includes but is not limited to the following steps S210 to S250:

[0066] Step S210, obtaining a training data set, wherein the training data set includes a plurality of training images and a description sentence corresponding to each training image, and the description sentence includes a plurality of attribute words;

[0067] Step S220, constructing an attribute dictionary based on the attribute words, and determining a real attribute label corresponding to the description sentence according to the description sentence and the attribute dictionary;

[0068] Step S230, performing transfer learning on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image;

[0069] Step S240, fine-tuning the multi-category classifier based on the attribute dictionary and the preset classification algorithm, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features;

[0070] Step S250: Using the training images and the real attribute labels as training data, the fine-tuned pre-trained model is trained to obtain an attribute generation model.

[0071] It can be understood that when training the attribute generation model, a training data set is first obtained. The training data set is the MSCOCO data set. The MS COCO data set contains multiple training images and corresponding description sentences, which can meet the requirements of attribute generation model training. The description sentence consists of multiple attribute words. An attribute dictionary is constructed through the attribute words. Based on the preset Sigmoid classification algorithm and in combination with the attribute dictionary, the pre-trained model obtained by transfer learning is fine-tuned to determine the mapping relationship between image features and attribute words belonging to the attribute dictionary, and then the training image and the real attribute label are used for training to obtain the attribute generation model; when generating image description, the target image is preprocessed and input into the attribute generation model to obtain the attribute target vector, and finally the prediction sentence is obtained by using the text generation model; based on this, by performing transfer learning on the preset image classification model, the attribute dictionary constructed by the attribute words is fine-tuned and trained to obtain the attribute generation model, the target area obtained by preprocessing the target image is input into the attribute generation model, and then the attribute target vector containing high-level semantic information in the target image is obtained through the attribute generation model, thereby improving the effect of image description generation, and the difficulty of training the attribute generation model can be reduced through transfer learning.

[0072] It should be noted that the attribute target vector contains high-level semantic information of the target image, and the text generation model can generate prediction sentences through the attribute target vector.

[0073] In specific practice, the image classification model is the VGG-16 model. The VGG-16 model is used to process the single-label problem of image classification. After transfer learning, it can process multi-label problems and reduce the training difficulty of the attribute generation model. For the VGG-16 model after transfer learning, the VGG-16 model contains multiple convolutional layers and multiple fully connected layers. The last fully connected layer is input into the fine-tuned multi-category classifier to obtain the attribute generation model.

[0074] It is worth noting that the processing method of the VGG-16 model is a technology well known to those skilled in the art and will not be elaborated here.

[0075] In addition, refer to Figure 3 , in one embodiment, Figure 2 Step S220 in the illustrated embodiment also includes but is not limited to the following steps S310 to S330:

[0076] Step S310, determining the frequency of occurrence of the attribute word in the description sentence;

[0077] Step S320, screening the attribute words based on the occurrence frequency and the preset dictionary capacity value to determine the target attribute words;

[0078] Step S330: Establish an attribute dictionary according to the target attribute word.

[0079] It can be understood that the number of categories of the multi-category classifier is the dictionary capacity value. The dictionary capacity value is preset to 256, and the number of categories of the multi-category classifier is 256; the frequency of occurrence of each attribute word refers to the number of each attribute word in all description sentences. In the process of establishing an attribute dictionary, the 256 attribute words with the highest occurrence frequency are selected as the target attribute words, and an attribute dictionary is established. The number of the target attribute words in all description sentences is greater than 5.

[0080] It is worth noting that the method for determining the frequency of occurrence of attribute words in the description sentence can be calculated by traversing the attribute words in the description sentence, which is not limited here.

[0081] It should be noted that an attribute word is a single word in a description sentence, and the part of speech of the attribute word is not limited.

[0082] In addition, refer to Figure 4 , in one embodiment, Figure 2 Step S250 in the illustrated embodiment also includes but is not limited to the following steps S410 to S430:

[0083] Step S410, inputting the training image into the fine-tuned pre-training model to determine the attribute training vector;

[0084] Step S420, determining a first model loss function according to the true attribute label and the attribute training vector;

[0085] Step S430, based on the first model loss function, updating the model parameters of the fine-tuned pre-trained model to obtain an attribute generation model;

[0086] Among them, the formula of the first model loss function is:

[0087]

[0088] Wherein, J is the first model loss determined by the first model loss function, N is the number of training images, and c is the dictionary capacity value;

[0089] p ij is the probability sequence p corresponding to the i-th training image i The jth element in p i =[p i1 ,p i2 ,...,p ic ], p ij It refers to the probability that the jth target attribute word exists in the i-th training image;

[0090] y ij is the jth element in the true attribute label corresponding to the i-th training image, y ij The calculation steps are:

[0091] When the i-th training image has the j-th target attribute word, then y ij =1, otherwise y ij =0, 1≤i≤N, and i and N are both positive integers.

[0092] It is understandable that through ij and p ij The Sigmoid cross entropy is calculated and used as the first model loss, that is, as a measure of error. The Sigmoid cross entropy is used to update the model parameters of the fine-tuned pre-trained model, so that the attribute generation model can improve the confidence of the generated attribute training vector.

[0093] It should be noted that the log function is a logarithmic loss function, which can well characterize the probability distribution. For multi-label classification tasks, the log function can be used to obtain the confidence of each category. In the training process of the attribute generation model, the log function can obtain the confidence of each training image for the multi-category classifier obtained by fine-tuning the attribute dictionary.

[0094] In addition, refer to Figure 5 , in one embodiment, Figure 1 Step S120 in the illustrated embodiment also includes but is not limited to the following steps S510 to S540:

[0095] Step S510, based on the normalized cut algorithm and the preset number of clusters, the target image is segmented to obtain a plurality of clusters, wherein each cluster includes at least one sub-region;

[0096] Step S520, for each cluster, based on the target image and a preset scoring algorithm, scoring is performed on all sub-regions of the cluster to determine a scoring value, wherein the scoring value corresponds to the sub-region one by one;

[0097] Step S530, for each cluster, based on the score value and the preset number of attribute regions, filter out the target attribute region from all sub-regions of the cluster;

[0098] Step S540, determining the target area according to the target image and the target attribute area.

[0099] In specific practice, the number of clusters is preset to m, and the number of attribute regions is preset to k. Based on the normalized cut algorithm, the target image is segmented to obtain m clusters, each cluster including at least one subregion; for each cluster, all subregions in the cluster are scored, and the k subregions with the highest scores are selected as target attribute regions, wherein the scoring method includes but is not limited to calculating the intersection-and-union ratio between each subregion and the target image, and taking the ratio of the intersection-and-union ratios as the scoring value, which is equivalent to selecting k subregions with larger intersection-and-union ratios as target attribute regions; for m clusters, there are a total of mk target attribute regions, and after adding the target image, there are a total of mk+1 regions, and the mk+1 regions are taken as target regions.

[0100] It is worth noting that the calculation method of the intersection-over-union ratio is a technique well known to those skilled in the art and will not be elaborated here.

[0101] like Figure 6 As shown, in one embodiment, Figure 1 The text generation model training method of step S140 in the illustrated embodiment includes but is not limited to the following steps S610 to S660:

[0102] Step S610, obtaining a training data set, wherein the training data set includes a plurality of training images and description words corresponding to each training image;

[0103] Step S620, inputting the training data set into the trained attribute generation model to obtain a plurality of attribute training vectors;

[0104] Step S630, based on a preset gated recurrent network, the attribute training vector is processed at n time steps to obtain a predicted word corresponding to the n time steps, wherein the gated recurrent network includes an update gate and a reset gate;

[0105] Step S640, obtaining a predicted sentence according to the predicted word;

[0106] Step S650, determining a second model loss function according to the attribute training vector and the prediction statement;

[0107] Step S660, based on the second model loss function, updating the model parameters of the gated recurrent network to obtain a text generation model;

[0108] Among them, the input of the gated recurrent network at the t-th time step includes the hidden state of the gated recurrent network at the t-1th time step and the corresponding prediction word at the t-th time step, the corresponding prediction word of the gated recurrent network at the t-1th time step is determined by the output of the gated recurrent network at the t-1th time step, the hidden state of the gated recurrent network at the t-1th time step is updated by the update gate and the reset gate of the gated recurrent network at the t-1th time step, the update gate and the reset gate of the gated recurrent network at the t-1th time step are determined by the hidden state of the gated recurrent network at the t-1th time step and the corresponding prediction word at the t-th time step, the t-th time step is any one of the n time steps, 1≤t≤n, and t and n are both positive integers.

[0109] It can be understood that using the attribute training vector obtained by the trained attribute generation model to train the gated recurrent network and using the trained gated recurrent network as a text generation model can ensure the accuracy of the predicted sentences.

[0110] In addition, in one embodiment, the gated recurrent network processes the attribute training vector at n time steps through the following formula:

[0111] z t =σ(h t-1 ·W zh +x t ·W zx ]),

[0112] r t =σ(h t-1 ·W rh +x t ·W rx ]),

[0113]

[0114]

[0115] Among them, z t is the update gate of the gated recurrent network at the tth time step, r t is the reset gate of the gated recurrent network at the tth time step, is the candidate hidden state of the gated recurrent network at the tth time step, h t is the hidden state of the gated recurrent network at the tth time step; W zh , W zx , W rh , W rx , W o These are all preset learning parameters, σ represents the sigmoid function, ⊙ represents the calculation of dot product, and x t =W s S t , W s is the preset learnable word vector parameter matrix, S t is the predicted word corresponding to the gated recurrent network at the tth time step.

[0116] It can be understood that the inputs of the reset gate and update gate in the gated recurrent unit are the current time step input and the hidden state of the previous time step, and the output is calculated by a fully connected layer whose activation function is the sigmoid function; if the element value in the reset gate is close to 0, the hidden state of the previous time step is discarded, and if the element value is close to 1, the hidden state of the previous time step is retained, and then the result of the dot product is concatenated with the input of the current time step, and then the candidate hidden state is calculated through a fully connected layer with the activation function tanh, and the value range of all its elements is [-1,1]; the hidden state of the current time step is calculated by the update gate of the current time step, the hidden state of the previous time step, and the candidate hidden state of the current time step.

[0117] It is worth noting that the use of the sigmoid activation function for processing is a technique well known to those skilled in the art and will not be elaborated herein.

[0118] In addition, in one embodiment, the formula of the second model loss function is:

[0119]

[0120] Where C is the second model loss determined by the second model loss function, N is the number of training images, i is the order of the training images of the input attribute generation model, 1≤i≤N, and i and N are both positive integers, S (i) is the predicted word corresponding to the i-th training image, V att (I (i) ) is the attribute training vector corresponding to the i-th training image, L (i) For S (i) The corresponding sentence length, y t is the output of the gated recurrent network at the tth time step, is the polynomial used for regularization, log p(S (i) |V att (I (i) )) is calculated as follows:

[0121]

[0122] Where S = {S1,...,SL}, S∈R K , S is the prediction sentence, S L is the Lth predicted word, L is S L The corresponding sentence length, R is the real number domain, K is the dictionary dimension of the attribute dictionary, S t is the predicted word corresponding to the gated recurrent network at the tth time step, S 1:t-1 is the prediction sentence corresponding to the gated recurrent network at the t-1th time step, V att (I) is the attribute training vector, p(S t |S 1:t-1 ,V att (I)) refers to V att (I) and S 1:t-1 Generate S t probability.

[0123] It is understandable that log p(S|V att (I)) refers to calculating the log-likelihood. Calculating the log-likelihood in the second model loss function can make the predicted sentence generated by the text generation model closer to the description of the target image itself.

[0124] It should be noted that S is a prediction sentence, which is composed of at least one prediction word. S is connected to the beginning of S with S0, which is a start mark for indicating the start of generating a prediction sentence. S is connected to the end of S with S L+1 , S L+1 It is the end flag, used to indicate the end of generating the prediction statement.

[0125] In practice, the gated recurrent network updates the model through the SGD method, taking 100 image-sentence pairs as small batch samples and iterating.

[0126] In addition, refer to Figure 7 , Figure 7 It is a system block diagram of an image description generation method based on transfer learning provided by another embodiment of the present invention.

[0127] It can be understood that when training the attribute generation model, a training data set is first obtained, the training data set includes multiple training images and corresponding description sentences, the description sentences are composed of multiple attribute words, an attribute dictionary is constructed through the attribute words, and then the pre-trained model obtained by transfer learning is fine-tuned and trained to obtain the attribute generation model; when generating image description, the target image is preprocessed, and then the target area obtained by the preprocessing is input into the attribute generation model, and then the attribute target vector is obtained, and finally the prediction sentence is obtained by using the text generation model; based on this, by performing transfer learning on the preset image classification model, and then fine-tuning the attribute dictionary constructed by the attribute words, and training to obtain the attribute generation model, the target area obtained by preprocessing the target image is input into the attribute generation model, and then the attribute target vector containing high-level semantic information in the target image is obtained through the attribute generation model, thereby improving the effect of image description generation, and the difficulty of training the attribute generation model can be reduced through transfer learning.

[0128] It should be noted that in the attribute generation model, the average pooling layer is used to average pool the outputs of the multi-category classifier to obtain the attribute target vector.

[0129] It is worth noting that using the average pooling layer to perform average pooling processing is a technology well known to those skilled in the art and will not be elaborated here.

[0130] In addition, refer to Figure 8 The present invention also provides an image description generating device 800 based on transfer learning, comprising:

[0131] An acquisition unit 810, configured to acquire a target image;

[0132] A preprocessing unit 820, used to preprocess the target image and determine the target area;

[0133] The attribute generation unit 830 is used to input the target area into the trained attribute generation model to obtain an attribute target vector;

[0134] A description generation unit 840, used to input the attribute target vector into the trained text generation model to obtain a predicted sentence;

[0135] The attribute generation model is trained through the following steps:

[0136] Acquire a training data set, wherein the training data set includes a plurality of training images and a description sentence corresponding to each training image, and the description sentence includes a plurality of attribute words;

[0137] Building an attribute dictionary based on the attribute words, and determining the real attribute label corresponding to the description sentence according to the description sentence and the attribute dictionary;

[0138] Performing transfer learning on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image;

[0139] Based on the attribute dictionary and the preset classification algorithm, the multi-category classifier is fine-tuned, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features;

[0140] The training images and true attribute labels are used as training data to train the fine-tuned pre-trained model to obtain the attribute generation model.

[0141] It can be understood that the specific implementation of the image description generation device 800 based on transfer learning is basically the same as the specific implementation example of the image description generation method based on transfer learning mentioned above, and will not be repeated here; based on this, by performing transfer learning on the preset image classification model, and then fine-tuning the attribute dictionary constructed using attribute words, and training to obtain the attribute generation model, the target area obtained by preprocessing the target image is input into the attribute generation model, and then the attribute target vector containing high-level semantic information in the target image is obtained through the attribute generation model, thereby improving the effect of image description generation, and the training difficulty of the attribute generation model can be reduced through transfer learning.

[0142] In addition, refer to Fig. 9 An embodiment of the present invention further provides an electronic device, the electronic device 900 comprising: a memory 910, a processor 920, and a computer program stored in the memory 910 and executable on the processor 920.

[0143] The processor 920 and the memory 910 may be connected via a bus or in other ways.

[0144] The non-transient software program and instructions required to implement the image description generation method based on transfer learning in the above embodiment are stored in the memory 910. When executed by the processor 920, the image description generation method based on transfer learning in the above embodiment is executed, for example, the above described Figure 1 Steps S110 to S140 of the method, Figure 2 Steps S210 to S250 of the method, Figure 3 Steps S310 to S330 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S540 of the method, Figure 6In the method steps S610 to S660, a target image is obtained; the target image is preprocessed to determine a target area; the target area is input into a trained attribute generation model to obtain an attribute target vector; the attribute target vector is input into a trained text generation model to obtain a prediction sentence; wherein the attribute generation model is trained by the following steps: obtaining a training data set, wherein the training data set includes a plurality of training images and description sentences corresponding to each training image, and the description sentence includes a plurality of attribute words; constructing an attribute dictionary based on the attribute words, and determining the real attribute label corresponding to the description sentence according to the description sentence and the attribute dictionary; performing transfer learning on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image; based on the attribute dictionary and a preset classification algorithm, the multi-category classifier is fine-tuned, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features; the training image and the real attribute label are used as training data to train the fine-tuned pre-trained model to obtain the attribute generation model. Based on this, the preset image classification model is transferred learned, and then the attribute dictionary constructed by attribute words is fine-tuned and trained to obtain the attribute generation model. The target area obtained by preprocessing the target image is input into the attribute generation model, and then the attribute target vector containing high-level semantic information in the target image is obtained through the attribute generation model, thereby improving the effect of image description generation, and the training difficulty of the attribute generation model can be reduced through transfer learning.

[0145] In addition, an embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions, which are executed by a processor or a controller, for example, by a processor in the above electronic device embodiment, so that the above processor can execute the image description generation method based on transfer learning in the above embodiment, for example, execute the above described Figure 1 Steps S110 to S140 of the method, Figure 2 Steps S210 to S250 of the method, Figure 3 Steps S310 to S330 of the method, Figure 4 Steps S410 to S430 of the method, Figure 5 Steps S510 to S540 of the method, Figure 6In the method steps S610 to S660, a target image is obtained; the target image is preprocessed to determine a target area; the target area is input into a trained attribute generation model to obtain an attribute target vector; the attribute target vector is input into a trained text generation model to obtain a prediction sentence; wherein the attribute generation model is trained by the following steps: obtaining a training data set, wherein the training data set includes a plurality of training images and description sentences corresponding to each training image, and the description sentence includes a plurality of attribute words; constructing an attribute dictionary based on the attribute words, and determining the real attribute label corresponding to the description sentence according to the description sentence and the attribute dictionary; performing transfer learning on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image; based on the attribute dictionary and a preset classification algorithm, the multi-category classifier is fine-tuned, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features; the training image and the real attribute label are used as training data to train the fine-tuned pre-trained model to obtain the attribute generation model. Based on this, the preset image classification model is transferred learned, and then the attribute dictionary constructed by attribute words is fine-tuned and trained to obtain the attribute generation model. The target area obtained by preprocessing the target image is input into the attribute generation model, and then the attribute target vector containing high-level semantic information in the target image is obtained through the attribute generation model, thereby improving the effect of image description generation, and the training difficulty of the attribute generation model can be reduced through transfer learning.

[0146] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0147] It will be appreciated by those skilled in the art that all or some of the steps and systems in the disclosed method above may be implemented as software, firmware, hardware and appropriate combinations thereof. Some physical components or all physical components may be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or may be implemented as hardware, or may be implemented as an integrated circuit, such as an application specific integrated circuit. Such software may be distributed on a computer-readable medium, which may include a computer storage medium (or a non-transitory medium) and a communication medium (or a temporary medium). As known to those skilled in the art, the term computer storage medium includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, disk storage or other magnetic storage devices, or any other medium that may be used to store desired information and may be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically embodies computer readable instructions, data structures, program modules, or other data in a modulated data signal such as a carrier wave or other transport mechanism, and may include any information delivery media.

[0148] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the above-mentioned implementation mode. Technical personnel familiar with the field can also make various equivalent deformations or substitutions without violating the spirit of the present invention. These equivalent deformations or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. A method for generating image description based on transfer learning, characterized in that: The method comprises: Get the target image; Preprocessing the target image to determine the target area; Inputting the target area into a trained attribute generation model to obtain an attribute target vector; Inputting the attribute target vector into the trained text generation model to obtain a predicted sentence; The attribute generation model is trained through the following steps: Acquire a training data set, wherein the training data set includes a plurality of training images and a description sentence corresponding to each of the training images, and the description sentence includes a plurality of attribute words; constructing an attribute dictionary based on the attribute words, and determining a real attribute label corresponding to the description sentence according to the description sentence and the attribute dictionary; Performing transfer learning on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image; Based on the attribute dictionary and a preset classification algorithm, fine-tuning the multi-category classifier, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features; Using the training images and the true attribute labels as training data, training the fine-tuned pre-trained model to obtain an attribute generation model; Wherein, constructing an attribute dictionary based on the attribute words includes: Determining the frequency of occurrence of the attribute word in the description sentence; Based on the occurrence frequency and the preset dictionary capacity value, the attribute words are screened to determine the target attribute words; Establishing an attribute dictionary according to the target attribute word; The step of using the training image and the real attribute label as training data to train the fine-tuned pre-trained model to obtain an attribute generation model includes: Inputting the training image into the fine-tuned pre-training model to determine an attribute training vector; Determine a first model loss function according to the true attribute label and the attribute training vector; Based on the first model loss function, updating the model parameters of the fine-tuned pre-trained model to obtain an attribute generation model; The formula of the first model loss function is: Wherein, J is the first model loss determined by the first model loss function, N is the number of the training images, and c is the dictionary capacity value; p ij is the probability sequence p corresponding to the i-th training image i The jth element in p i =[p i1 ,p i2 ,...,p ic ], p ij It refers to the probability that the jth target attribute word exists in the i-th training image; y ij is the jth element in the true attribute label corresponding to the i-th training image, y ij The calculation steps are: When the i-th training image has the j-th target attribute word, then y ij =1, otherwise y ij =0, 1≤i≤N, and i and N are both positive integers.

2. The method according to claim 1, characterized in that The preprocessing of the target image to determine the target area includes: Based on a normalized cut algorithm and a preset number of clusters, the target image is segmented to obtain a plurality of clusters, wherein each of the clusters includes at least one sub-region; For each of the clusters, based on the target image and a preset scoring algorithm, scoring is performed on all sub-regions of the cluster to determine a scoring value, wherein the scoring value corresponds to the sub-region one by one; For each of the clusters, based on the score value and a preset number of attribute regions, a target attribute region is screened out from all sub-regions of the cluster; A target area is determined according to the target image and the target attribute area.

3. The method according to claim 1, characterized in that The text generation model is trained by the following steps: Acquire a training data set, wherein the training data set includes a plurality of training images and a description word corresponding to each of the training images; Inputting the training data set into the trained attribute generation model to obtain a plurality of attribute training vectors; Based on a preset gated recurrent network, the attribute training vector is processed at n time steps to obtain a predicted word corresponding to the n time steps, wherein the gated recurrent network includes an update gate and a reset gate; Obtain a predicted sentence according to the predicted word; Determining a second model loss function according to the attribute training vector and the prediction statement; Based on the second model loss function, updating the model parameters of the gated recurrent network to obtain a text generation model; Among them, the input of the gated recurrent network at the t-1 time step includes the hidden state of the gated recurrent network at the t-1 time step and the corresponding prediction word at the t-1 time step, the prediction word corresponding to the gated recurrent network at the t-1 time step is determined by the output of the gated recurrent network at the t-1 time step, the hidden state of the gated recurrent network at the t-1 time step is updated by the update gate and the reset gate of the gated recurrent network at the t-1 time step, the update gate and the reset gate of the gated recurrent network at the t-1 time step are determined by the hidden state of the gated recurrent network at the t-1 time step and the corresponding prediction word at the t-1 time step, the t-th time step is any one of the n time steps, 1≤t≤n, and t and n are both positive integers.

4. The method according to claim 3, characterized in that The gated recurrent network processes the attribute training vector at n time steps using the following formula: z t =σ(h t-1 ·W zh +x t ·W zx ]), r t =σ(h t-1 ·W rh +x t W rx ]), Among them, z t is the update gate of the gated recurrent network at the tth time step, r t is the reset gate of the gated recurrent network at the tth time step, is the candidate hidden state of the gated recurrent network at the tth time step, h t is the hidden state of the gated recurrent network at the tth time step; W zh , W zx , W rh , W rx , W o These are all preset learning parameters, σ represents the sigmoid function, ⊙ represents the calculation of dot product, and x t =W s S t , W s is the preset learnable word vector parameter matrix, S t is the predicted word corresponding to the gated recurrent network at the tth time step.

5. The method according to claim 3, characterized in that: The formula of the second model loss function is: Wherein, C is the second model loss determined by the second model loss function, N is the number of the training images, i is the order of the training images input into the attribute generation model, 1≤i≤N, and i and N are both positive integers, S (i) is the predicted word corresponding to the i-th training image, V att (I (i) ) is the attribute training vector corresponding to the i-th training image, L (i) For the S (i) The corresponding sentence length, y t is the output of the gated recurrent network at the tth time step, is the polynomial used for regularization, logP(S (i) |V att (I (i) )) is calculated as follows: Where S = {S1,...,S L }, S∈R K , S is the prediction statement, S L is the Lth predicted word, L is the S L The corresponding sentence length, R is the real number domain, K is the dictionary dimension of the attribute dictionary, S t is the predicted word corresponding to the gated recurrent network at the tth time step, S 1:t-1 is the prediction statement corresponding to the gated recurrent network at the t-1th time step, V att (I) is the attribute training vector, p(S t |S 1:t-1 ,V att (I)) refers to the V att (I) and the S 1:t-1 Produce the S t probability.

6. A device for generating image description based on transfer learning, characterized in that: The device comprises: An acquisition unit, used for acquiring a target image; A preprocessing unit, used for preprocessing the target image to determine the target area; An attribute generation unit, used for inputting the target area into a trained attribute generation model to obtain an attribute target vector; A description generation unit, used for inputting the attribute target vector into a trained text generation model to obtain a predicted sentence; The attribute generation model is trained through the following steps: Acquire a training data set, wherein the training data set includes a plurality of training images and a description sentence corresponding to each of the training images, and the description sentence includes a plurality of attribute words; constructing an attribute dictionary based on the attribute words, and determining a real attribute label corresponding to the description sentence according to the description sentence and the attribute dictionary; Performing transfer learning on a preset image classification model to obtain a pre-trained model, wherein the pre-trained model includes a convolutional neural network and a multi-category classifier connected in sequence, and the convolutional neural network is used to extract image features of the training image; Based on the attribute dictionary and a preset classification algorithm, fine-tuning the multi-category classifier, wherein the fine-tuned multi-category classifier is used to determine the attribute words belonging to the attribute dictionary corresponding to the image features; Using the training images and the true attribute labels as training data, training the fine-tuned pre-trained model to obtain an attribute generation model; Wherein, constructing an attribute dictionary based on the attribute words includes: Determining the frequency of occurrence of the attribute word in the description sentence; Based on the occurrence frequency and the preset dictionary capacity value, the attribute words are screened to determine the target attribute words; Establishing an attribute dictionary according to the target attribute word; The step of using the training image and the real attribute label as training data to train the fine-tuned pre-trained model to obtain an attribute generation model includes: Inputting the training image into the fine-tuned pre-training model to determine an attribute training vector; Determine a first model loss function according to the true attribute label and the attribute training vector; Based on the first model loss function, updating the model parameters of the fine-tuned pre-trained model to obtain an attribute generation model; The formula of the first model loss function is: Wherein, J is the first model loss determined by the first model loss function, N is the number of the training images, and c is the dictionary capacity value; p ij is the probability sequence p corresponding to the i-th training image i The jth element in p i =[p i1 ,p i2 ,...,p ic ], p ij It refers to the probability that the jth target attribute word exists in the i-th training image; y ij is the jth element in the true attribute label corresponding to the i-th training image, y ij The calculation steps are: When the i-th training image has the j-th target attribute word, then y ij =1, otherwise y ih =0, 1≤i≤N, and i and N are both positive integers.

7. An electronic device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the computer program, the method for generating image description based on transfer learning as described in any one of claims 1 to 5 is implemented.

8. A computer-readable storage medium storing computer-executable instructions, characterized in that: The computer executable instructions are used to execute the image description generation method based on transfer learning as described in any one of claims 1 to 5.

Citation Information

Patent Citations

  • Training method and device of image description statement generation module and electronic equipment

    CN114358203A

  • Artificial intelligence apparatus and method for recognizing object included in image data

    US20210142127A1

  • Method and apparatus for generating text summary, computer device and storage medium

    WO2020107878A1