Image description method and device, computer equipment, and storage medium

Abstract

The embodiments of the present application provide an image description method and apparatus, a computer device, and a storage medium, which belong to the field of artificial intelligence technology. It includes: obtaining an original image, extracting features from the original image to obtain image features; performing region detection on the original image based on the image features to obtain a target region image; extracting features from the target region image to obtain a region feature vector; extracting and processing the region feature vector through a topic generation model to obtain topic data; the topic data includes a topic word vector and moment state information; performing word prediction on the topic data through a word generation model to obtain description words; splicing the description words according to the moment state information to obtain a target description text for describing the original image. Through multiple feature extractions, the target description text can contain more image details; the topic generation model and the word generation model are used in turn to hierarchically generate a description text with coherent semantics.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to an image description method and apparatus, a computer device, and a storage medium. Background Art

[0002] With the development of artificial intelligence technology, image description technology has become a solution to understand the content of images. Among them, image description technology is used to enable computers to understand the content of images and generate corresponding description texts. At present, it is generally performed by detecting objects in the original image and generating corresponding description texts. However, this method will result in the semantics of the description text being not coherent enough. Summary of the invention

[0003] The main purpose of the embodiments disclosed in this application is to propose an image description method and apparatus, a computer device, and a storage medium, which sequentially use a topic generation model and a word generation model to hierarchically generate a description text with coherent semantics.

[0004] To achieve the above objective, a first aspect of the disclosed embodiment of the present application proposes an image description method, the method comprising:

[0005] Get the original image;

[0006] Performing feature extraction on the original image to obtain image features;

[0007] Performing region detection on the original image according to the image features to obtain a target region image;

[0008] Extracting features from the target area image to obtain a regional feature vector;

[0009] The regional feature vector is extracted and processed by a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and time state information corresponding to the topic word vector;

[0010] Perform word prediction on the topic data through a word generation model to obtain description words;

[0011] Each of the description words is spliced ​​according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image.

[0012] In some embodiments, extracting features from the target area image to obtain a regional feature vector includes:

[0013] Preprocessing the target area image to obtain a preliminary area image;

[0014] Performing convolution processing on the preliminary region image to obtain a convolution feature vector;

[0015] Pooling is performed on the convolution feature vector to obtain the regional feature vector.

[0016] In some embodiments, preprocessing the target area image to obtain a preliminary area image includes:

[0017] Performing feature mapping on the target area image to obtain a preliminary mapping image;

[0018] The preliminary mapping image is resized according to a preset size to obtain the preliminary region image.

[0019] In some embodiments, the step of performing a size transformation on the preliminary mapping image according to a preset size to obtain the preliminary region image includes:

[0020] Acquire the first coordinate of the target area image;

[0021] Calculate a second coordinate according to the first coordinate and the preset size;

[0022] The size of the preliminary mapping image is adjusted according to the second coordinates to obtain the preliminary region image.

[0023] In some embodiments, extracting and processing the regional feature vectors by using a topic generation model to obtain topic data includes:

[0024] Inputting the regional feature vector into the topic generation model; wherein the topic generation model includes a recurrent layer and a hidden layer;

[0025] Performing cyclic iterative processing on the regional feature vector through the cyclic layer to obtain the subject word vector;

[0026] The moment state information is obtained from the hidden layer according to the subject word vector.

[0027] In some embodiments, the step of performing word prediction on the topic data through a word generation model to obtain a description word includes:

[0028] Obtaining an activation function of the word generation model;

[0029] Inputting the subject word vector and the moment state information into the word generation model;

[0030] Calculate the subject word vector and the moment state information according to the activation function to obtain at least one candidate word;

[0031] The description word is obtained from the candidate words.

[0032] In some embodiments, obtaining the description word from the candidate words includes:

[0033] Calculating the distribution probability of each candidate word in a preset word set;

[0034] The candidate word with the largest distribution probability is obtained as the description word.

[0035] A second aspect of the disclosed embodiment of the present application provides an image description device, the device comprising:

[0036] Image acquisition module: used to obtain the original image;

[0037] The first feature extraction module is used to extract features from the original image to obtain image features;

[0038] Region detection module: used for performing region detection on the original image according to the image features to obtain a target region image;

[0039] The second feature extraction module is used to extract features from the target area image to obtain a regional feature vector;

[0040] Data extraction module: used to extract and process the regional feature vector through a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and the time state information corresponding to the topic word vector;

[0041] Word prediction module: used to predict words on the topic data through a word generation model to obtain description words;

[0042] Word splicing module: used for splicing each of the description words according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image.

[0043] The third aspect of the disclosed embodiments of the present application provides a computer device, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute the image description method as described in any one of the embodiments of the first aspect of the present application.

[0044] The fourth aspect of the disclosed embodiments of the present application proposes a storage medium, which is a computer-readable storage medium. The storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image description method as described in any one of the embodiments of the first aspect of the present application.

[0045] The present application discloses an image description method and apparatus, a computer device, and a storage medium, which are proposed in the embodiments, by acquiring an original image; extracting features from the original image to obtain image features; performing region detection on the original image according to the image features to obtain a target region image; extracting features from the target region image to obtain a region feature vector; extracting and processing the region feature vector through a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and the moment state information of the corresponding topic word vector; performing word prediction on the topic data through a word generation model to obtain description words; splicing and processing each description word according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image. The present application discloses an embodiment, through multiple feature extractions, the generated target description text can contain more image details; in addition, based on the region detection of the original image, the target description text is generated hierarchically using the topic generation model and the word generation model in turn, and a description text with coherent semantics can be generated.

[0046] Other features and advantages of the embodiments disclosed in this application will be described in the subsequent description, and in part will become apparent from the description, or understood by practicing the embodiments disclosed in this application. The purposes and other advantages of the embodiments disclosed in this application can be achieved and obtained through the structures specifically pointed out in the description, claims and drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] The accompanying drawings are used to provide further understanding of the technical solutions of the embodiments disclosed in this application and constitute a part of the specification. Together with the embodiments of the embodiments disclosed in this application, they are used to explain the technical solutions of the embodiments disclosed in this application and do not constitute a limitation on the technical solutions of the embodiments disclosed in this application.

[0048] Figure 1 is a flow chart of an image description method provided by an embodiment disclosed in the present application;

[0049] Figure 2 yes Figure 1 Flow chart of step S400 in FIG.

[0050] Figure 3 yes Figure 2 Flow chart of step S410 in FIG.

[0051] Figure 4 yes Figure 3 Flow chart of step S412 in FIG.

[0052] Figure 5 yes Figure 1 Flow chart of step S500 in FIG.

[0053] Figure 6 yes Figure 1Flow chart of step S600 in FIG.

[0054] Figure 7 yes Figure 6 Flow chart of step S640 in FIG.

[0055] Figure 8 It is a module structure block diagram of the image description device provided by the embodiment disclosed in this application;

[0056] Fig. 9 It is a schematic diagram of the hardware structure of the computer device provided in the embodiment disclosed in this application. DETAILED DESCRIPTION

[0057] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0058] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0059] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0060] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments disclosed in the present application. However, those skilled in the art will appreciate that the technical solutions disclosed in the present application may be put into practice without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects disclosed in the present application.

[0061] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0062] The flowcharts shown in the accompanying drawings are only exemplary and do not necessarily include all the contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps can be decomposed, and some operations / steps can be combined or partially combined, so the actual execution order may change according to actual conditions.

[0063] First, some nouns involved in this application are analyzed:

[0064] Artificial Intelligence (AI): is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science. AI attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robots, language recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] Image Caption: It is a comprehensive problem that integrates computer vision, natural language processing and machine learning. It is similar to translating a picture into a descriptive text. This task is very easy for humans, but very challenging for machines. It not only requires the use of models to understand the content of the picture, but also requires the use of natural language to express the relationship between them.

[0066] Feature extraction: In machine learning, pattern recognition, and image processing, feature extraction starts with an initial set of measurements and builds derived values ​​(features) that are intended to be informative and non-redundant, thereby facilitating subsequent learning and generalization steps and, in some cases, leading to better interpretability. Feature extraction is related to dimensionality reduction. The quality of features has a crucial impact on generalization capabilities.

[0067] Feature Map: It is the result of convolution of the input image through a neural network. It represents a feature in the neural space, and its resolution depends on the step size of the previous convolution kernel.

[0068] Convolutional Neural Networks (CNN): A type of feedforward neural network (FNN) that includes convolution calculations and has a deep structure. It is one of the representative algorithms of deep learning.

[0069] Long Short-Term Memory (LSTM) is a time-recurrent neural network designed to solve the long-term dependency problem of general recurrent neural networks (RNN). All RNNs have a chain form of repeated neural network modules. In standard RNNs, this repeated structural module has only a very simple structure.

[0070] Pooling: It is an important concept in convolutional neural networks. It is actually a form of downsampling. There are many different forms of nonlinear pooling functions, and "Maxpooling" is the most common one. It divides the input image into several rectangular regions and outputs the maximum value for each sub-region. Intuitively, the reason why this mechanism works is that after a feature is discovered, its precise position is far less important than its relative position to other features. The pooling layer will continuously reduce the spatial size of the data, so the number of parameters and the amount of computation will also decrease, which also controls overfitting to a certain extent. Generally speaking, pooling layers are periodically inserted between the convolutional layers of CNN. The pooling layer usually acts on each input feature separately and reduces its size.

[0071] Super-resolution test sequence (Visual Geometry Group, VGG): is one of the CNN models, that is, a deep convolutional neural network constructed by using a series of small-sized convolution kernels and pooling layers of size 3x3. VGG uses 3×3 convolution layers and pooling layers to extract features, and uses three fully connected layers at the end of the network, and uses the output of the last fully connected layer as the classification prediction. In VGG, each convolution layer will use ReLU as the activation function, and dropout is added after the fully connected layer to suppress overfitting.

[0072] Region Proposal Network (RPN): Its main function is to generate region proposals, that is, region proposals can be regarded as many potential bounding boxes (also called anchors, which are rectangular boxes containing 4 coordinates).

[0073] Faster R-CNN: It is a regional convolutional neural network, which can be simply regarded as a model of region proposal network + Fast-R-CNN, using region proposal network (RPN) to replace the selective search method in Fast-R-CNN. Its steps are roughly as follows: input the whole picture into CNN to obtain convolution features; input the convolution features into RPN to obtain the feature information of the candidate box; for the features extracted from the candidate box, use the classifier to determine whether it belongs to a specific class; for the candidate box belonging to a certain category, use the regressor to further adjust its position.

[0074] Feature mapping: also known as dimensionality reduction, is the process of mapping the feature vector of high-dimensional multimedia data to one-dimensional or low-dimensional space.

[0075] Gated Recurrent Unit (GRU): GRU is a gating mechanism in recurrent neural networks (RNNs). Similar to other gating mechanisms, it aims to solve the gradient vanishing / exploding problem in standard RNNs while retaining the long-term information of the sequence. GRU performs as well as LSTM on many sequence tasks such as speech recognition, but it has fewer parameters than LSTM, including only a reset gate and an update gate.

[0076] Bilinear interpolation: also called bilinear interpolation, its core idea is to perform linear interpolation in two directions. As an interpolation algorithm in numerical analysis, bilinear interpolation is widely used in signal processing, digital image and video processing.

[0077] Activation Functions: Activation functions are very important for artificial neural network models to learn and understand very complex and nonlinear functions. They introduce nonlinear characteristics into our network. In neurons, the inputs are weighted, summed, and then a function is applied. This function is the activation function. The activation function is introduced to increase the nonlinearity of the neural network model.

[0078] Fully connected layer: Each node is connected to all nodes in the previous layer to combine the features extracted previously. Due to its fully connected nature, the fully connected layer generally has the most parameters.

[0079] Hidden layer: In a neural network, all layers except the input layer and the output layer are called hidden layers. Hidden layers do not directly receive signals from the outside world, nor do they directly send signals to the outside world. The significance of a single hidden layer is to abstract the features of the input data into another dimensional space to show its more abstract features, which can be better linearly divided. The significance of multiple hidden layers is the multi-level abstraction of input features, and the ultimate goal is to better linearly divide different types of data.

[0080] Cross Entropy: Mainly used to measure the difference information between two probability distributions. The performance of language models is usually measured by cross entropy and complexity (perplexity). The meaning of cross entropy is the difficulty of text recognition using the model, or from the perspective of compression, how many bits are used to encode each word on average. The meaning of complexity is the average number of branches that the model uses to represent the text, and its reciprocal can be regarded as the average probability of each word. Smoothing refers to assigning a probability value to the unobserved N-gram combination to ensure that the word sequence can always get a probability value through the language model.

[0081] Cross entropy loss function: It is a smooth function, which is essentially the application of cross entropy in information theory to classification problems. Examples of classifiers corresponding to the cross entropy loss function include logistic regression, artificial neural networks, and support vector machines with probability output.

[0082] Gradient Descent: Gradient Descent is an iterative method that can be used to solve least squares problems (both linear and nonlinear). When solving the model parameters of machine learning algorithms, that is, unconstrained optimization problems, gradient descent is one of the most commonly used methods. Another commonly used method is the least squares method. When solving the minimum value of the loss function, the gradient descent method can be used to iterate step by step to obtain the minimized loss function and model parameter values. Conversely, if the maximum value of the loss function needs to be solved, the gradient ascent method needs to be used to iterate. In machine learning, two gradient descent methods have been developed based on the basic gradient descent method, namely stochastic gradient descent and batch gradient descent.

[0083] With the development of artificial intelligence technology, image description technology has become a solution to understand the content of images. Among them, image description technology is used to enable computers to understand the content of images and generate corresponding description texts. At present, it is generally performed by performing target detection on the original image and directly generating corresponding description texts based on the target detection results, but this will cause the generated description texts to be more scattered, which in turn leads to the lack of semantic coherence of the description texts.

[0084] Based on this, the disclosed embodiments of the present application propose an image description method and apparatus, a computer device, and a storage medium, which can obtain an original image; extract features from the original image to obtain image features; perform region detection on the original image according to the image features to obtain a target region image; extract features from the target region image to obtain a region feature vector; extract and process the region feature vector through a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and the moment state information of the corresponding topic word vector; perform word prediction on the topic data through a word generation model to obtain description words; splice each description word according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image. The disclosed embodiments of the present application can make the generated target description text contain more image details through multiple feature extractions; in addition, based on the region detection of the original image, the target description text is generated hierarchically using the topic generation model and the word generation model in turn, and a description text with coherent semantics can be generated.

[0085] The embodiments disclosed in the present application provide an image description method and apparatus, a computer device, and a storage medium, which are specifically described through the following embodiments. First, the image description method in the embodiments disclosed in the present application is described.

[0086] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0087] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0088] The image description method provided in the disclosed embodiments of the present application relates to the field of artificial intelligence. The image description method provided in the disclosed embodiments of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, or a smart watch, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the image description method, etc., but is not limited to the above forms.

[0089] The disclosed embodiments of the present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronic devices, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0090] Reference Figure 1 According to the image description method of the first aspect of the embodiment disclosed in this application, it includes but is not limited to steps S100 to S800.

[0091] Step S100, obtaining an original image;

[0092] Step S200, extracting features from the original image to obtain image features;

[0093] Step S300, performing region detection on the original image according to image features to obtain a target region image;

[0094] Step S400, extracting features from the target area image to obtain a regional feature vector;

[0095] Step S500, extracting and processing the regional feature vectors through a topic generation model to obtain topic data;

[0096] Step S600, predicting words on the topic data through a word generation model to obtain description words;

[0097] Step S700, concatenating each description word according to the momentary state information to obtain a target description text.

[0098] In step S100 of some embodiments, an original image is obtained. The original image refers to an image that needs to be described in text. The purpose of the embodiments disclosed in this application is to translate the content of the original image into text, that is, the target description text.

[0099] In step S200 of some embodiments, feature extraction is performed on the original image to obtain image features. Specifically, an encoder, such as a VGG-16 network in a VGG model, can be used to extract features from the original image. Input into the VGG-16 network for feature extraction processing to obtain the feature image output by the model, that is, in Among them, W, H, and C are all image sizes, W represents width, H represents height, and C represents the number of channels. ′ is the width after passing through the VGG-16 network, and H' is the height after passing through the VGG-16 network.

[0100] Specifically, those skilled in the art can set the specific network layers and the number of network layers in the VGG-16 network according to actual needs. For example, the specific network layers and the number of network layers of the VGG-16 model of the embodiment disclosed in the present application can be set to: 13 convolutional layers, 3 fully connected layers and 5 pooling layers.

[0101] In addition, in this model, the embodiments disclosed in this application may also consider removing the last pooling layer. It should be noted that the role of the pooling layer is to reduce the size of the parameter matrix, for example, the width and height of the parameter matrix can be reduced to half of the original, and the number of network channels does not change. Under normal circumstances, adding a pooling layer to a neural network can filter out some unimportant information during the image compression process, but since the embodiments disclosed in this application require region detection of the original image, it is possible to consider removing one pooling layer to retain more image features to improve the accuracy of region detection.

[0102] In step S300 of some embodiments, region detection is performed on the original image based on image features to obtain a target region image. Specifically, the image features need to be input into the RPN for region detection. The model will automatically divide at least one target region and a target region image corresponding to the target region based on the image features.

[0103] In step S400 of some embodiments, feature extraction is performed on the target region image to obtain a region feature vector, and the region feature vector is used to generate a target description text.

[0104] In step S500 of some embodiments, the regional feature vector is extracted and processed by a topic generation model to obtain topic data. The topic sentence data generated by the topic generation model can determine how many sentences are contained in the generated image description, that is, how many topics are contained; wherein the topic data includes topic word vectors and the momentary state information of the corresponding topic word vectors, and the topic word vectors and the momentary state information are used for subsequent word prediction and generation of words corresponding to the sentences.

[0105] In step S600 of some embodiments, a word generation model is used to perform word prediction on the topic data to obtain a plurality of description words corresponding to the sentence.

[0106] In step S700 of some embodiments, each description word in each sentence is spliced ​​according to the momentary state information to obtain a generated topic sentence, and multiple topic sentences are spliced ​​to form a target description text, wherein the target description text is used to describe the original image.

[0107] In practical applications, assuming that the topic data generated by the topic generation model for extraction processing includes 4 topic word vectors, such as topic vector 1, topic vector 2, topic vector 3 and topic vector 4, the word generation model will perform word prediction based on the above 4 topic word vectors. For example, multiple description words that can be generated by performing word prediction on topic vector 1 are "There", "is", "a" and "competition", and the above description words are spliced ​​to generate topic sentence 1, that is, "There is a competition"; multiple description words that can be generated by performing word prediction on topic vector 2 are "Three", "women", "are", "playing", "field" and "hockey", and the above description words are spliced ​​to generate topic sentence 2, that is, "Three women are playing field hockey"; multiple description words that can be generated by performing word prediction on topic vector 3 are "The", "one", "in", "red", "is", "hitting", "the" and "ball", and the above description words are spliced ​​to generate topic sentence 3, that is, "The one in red is hitting the ball"; multiple description words that can be generated by performing word prediction on topic vector 4 are "The", "other", "two", "in", "white", "are" and "defending", and the above description words are concatenated to generate topic sentence 4, that is, "The other two in white aredefending"; the above-generated topic sentences 1, 2, 3 and 4 are concatenated to form the final target description text.

[0108] The disclosed embodiment of the present application obtains an original image; extracts features from the original image to obtain image features; performs region detection on the original image based on the image features to obtain a target region image; extracts features from the target region image to obtain a region feature vector; extracts and processes the region feature vector through a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and the moment state information of the corresponding topic word vector; performs word prediction on the topic data through a word generation model to obtain description words; splices each description word based on the moment state information to obtain a target description text; wherein the target description text is used to describe the original image. The disclosed embodiment of the present application can make the generated target description text contain more image details through multiple feature extractions; in addition, based on the region detection of the original image, the target description text is generated hierarchically using the topic generation model and the word generation model in turn, and a description text with coherent semantics can be generated.

[0109] In some embodiments, Figure 2 As shown, step S400 specifically includes but is not limited to step S410, step S420 and step S430.

[0110] Step S410, preprocessing the target area image to obtain a preliminary area image;

[0111] Step S420, performing convolution processing on the preliminary region image to obtain a convolution feature vector;

[0112] Step S430, performing pooling processing on the convolution feature vector to obtain a regional feature vector.

[0113] In step S410 of some embodiments, the target area image is preprocessed to obtain a preliminary area image that meets preset conditions, wherein the preset conditions may refer to a specific image dimension and image size, etc.

[0114] In step S420 of some embodiments, the preliminary region image is convolved to obtain a convolution feature image. Specifically, the image features of the preliminary region image may be input into a certain number of fully connected layers, such as two fully connected layers. The connected layers convolute the image features of each preliminary region image and compress them into a D-dimensional vector, which is recorded as It should be noted that the dimension of the feature vector is consistent with the number of target regions, or the number of preliminary region images.

[0115] In step S430 of some embodiments, the convolution feature vector is pooled to obtain a regional feature vector. Specifically, the D-dimensional convolution feature vector is input into a pooling layer for average pooling or maximum pooling to retain relevant information of each feature region to obtain a regional feature vector.

[0116] Specifically, the process of pooling the convolution feature vector is shown in formula (1):

[0117]

[0118] Among them, v p is the regional feature vector obtained after pooling, P represents the number of convolution feature vectors and also the dimension of the regional feature vector.

[0119] In some embodiments, Figure 3 As shown, step S410 specifically includes but is not limited to step S411 and step S412.

[0120] Step S411, performing feature mapping on the target area image to obtain a preliminary mapping image;

[0121] Step S412, resizing the preliminary mapping image according to a preset size to obtain a preliminary region image.

[0122] In step S411 of some embodiments, after performing region detection on the original image, it is also necessary to perform feature mapping on the target region image to obtain a preliminary mapping image, that is, a feature map corresponding to the target region image.

[0123] In step S412 of some embodiments, the size of the preliminary mapping image is adjusted to a preset size, for example, the size of the preliminary mapping image is adjusted to X×Y, where X is a preset length and Y is a preset width, to obtain a preliminary region image.

[0124] In some embodiments, Figure 4 As shown, step S412 specifically includes but is not limited to step S4121, step S4122 and step S4123.

[0125] Step S4121, obtaining the first coordinates of the target area image;

[0126] Step S4122, calculating a second coordinate according to the first coordinate and a preset size;

[0127] Step S4123, adjusting the size of the preliminary mapping image according to the second coordinates to obtain a preliminary regional image.

[0128] In step S4121 of some embodiments, the first coordinate of the target area image is obtained. Specifically, let the target area image be I′, and the first coordinate refers to the coordinate (x′) of any point in the target area image I′. i,j ,y′ i,j ).

[0129] In step S4122 of some embodiments, the reverse projection coordinates of the preliminary mapping image to the target area image, i.e., the second coordinates, are calculated according to the first coordinates and the preset size. Specifically, for any point coordinate (x) in the preliminary mapping image I″ i,j ,y″ i,j ) projected onto the target region image I′ is Then use the bilinear interpolation method to calculate (x′ i,j ,y′ i,j ) coordinate point, which is the pixel I" of the corresponding point in I" c,i，j , i.e. the second coordinate, the calculation formula (2) is as follows:

[0130]

[0131] Among them, k(d)=max(0,1-|d|), which represents the distance between two points.

[0132] In step S4123 of some embodiments, the size of the preliminary mapping image is adjusted according to the second coordinates to obtain a preliminary regional image. Specifically, after determining the second coordinates, the preliminary preset image is scaled according to the second coordinates, for example, the coordinates of a certain point of the preset image are adjusted to the position of the second coordinates, or the coordinates of the preset image are adjusted with reference to the position of the second coordinates.

[0133] In some embodiments, Figure 5 As shown, step S500 specifically includes but is not limited to step S510, step S520 and step S530.

[0134] Step S510, inputting the regional feature vector into a topic generation model; wherein the topic generation model includes a recurrent layer and a hidden layer;

[0135] Step S520, performing a loop iterative process on the regional feature vector through a loop layer to obtain a topic word vector;

[0136] Step S530, obtaining the moment state information from the hidden layer according to the topic word vector.

[0137] In step S510 of some embodiments, the regional feature vector is input into a topic generation model; wherein the topic generation model includes a recurrent layer and a hidden layer, the recurrent layer is used to generate a topic word vector, and the hidden layer is used to output the moment state information; in practical applications, the topic generation module of the embodiment disclosed in the present application may adopt a single-layer LSTM model.

[0138] In step S520 of some embodiments, the regional feature vector is iteratively processed through the loop layer of the topic generation model to generate a topic word vector. That is, the topic word vector generated by the LSTM model determines the number of sentences contained in the final image description, that is, the number of topics.

[0139] In step S530 of some embodiments, the moment state information is obtained from the hidden layer of the LSTM according to the subject word vector, wherein the moment state information refers to the hidden state of the hidden layer in the LSTM model at the moment corresponding to the generation of the subject word vector. Assume that the hidden state of the hidden layer at each moment is H is the size of the hidden layer. In the embodiment disclosed in the present application, the hidden state has two functions. One is to classify the hidden layer to determine whether the sentence generated at this moment is the last sentence in the description of the image generation, which is recorded as The second is that the hidden state at this moment will be used as the input of the word generation model to generate the corresponding topic sentence Each P-dimensional word vector

[0140] In some embodiments, Figure 6 As shown, step S600 specifically includes but is not limited to step S610, step S620, step S630 and step S640.

[0141] Step S610, obtaining an activation function of a word generation model;

[0142] Step S620, inputting the subject word vector and the moment state information into the word generation model;

[0143] Step S630, calculating the subject word vector and the moment state information according to the activation function to obtain at least one candidate word;

[0144] Step S640, obtaining description words from the candidate words.

[0145] In step S610 of some embodiments, an activation function of a word generation model is obtained, wherein the word generation model is used to generate description words of the corresponding topic sentence according to the topic data. In practical applications, the word generation module may adopt a two-layer GRU model.

[0146] In step S620 of some embodiments, the topic word vector and the moment state information are input into the word generation model as input.

[0147] In step S630 of some embodiments, the topic word vector and the moment state information are calculated according to the activation function to obtain at least one candidate word. Specifically, the initial input of the word generation model is the topic word vector output by the topic generation model at that moment, and the hidden state corresponding to the hidden layer in the word generation model at each moment is input into the word generation model. It should be noted that the process of calculating the candidate words is shown in formula (3), formula (4) and formula (5):

[0148]

[0149]

[0150]

[0151] Among them, x -1 represents the P-dimensional topic word vector generated by sentence-level LSTM, which is used as the initial input of the GRU model. t represents the candidate words generated by the GRU model, S 0 To mark the start, is the hidden state of the hidden layer in the topic generation model at the previous moment, p t+1 Represents the distribution rate of the t+1th candidate word in the entire preset word set.

[0152] In step S640 of some embodiments, description words are selected from the candidate words to generate a topic sentence.

[0153] In some embodiments, Figure 7 As shown, step S640 specifically includes but is not limited to step S641 and step S642.

[0154] Step S641, calculating the distribution probability of each candidate word in the preset word set;

[0155] Step S642, obtaining the candidate word with the largest distribution probability as the description word.

[0156] In step S641 of some embodiments, the distribution probability of each candidate word in the preset word set is calculated. Specifically, it can be calculated by formula (6):

[0157] p t+1 =p(S t+1 |I,S 0 ,...,S t ) (6)

[0158] Among them, p t+1 represents the distribution rate of the t+1th candidate word in the entire preset word set, and I represents the preset word set.

[0159] In step S642 of some embodiments, the candidate word with the largest distribution probability is obtained as the description word. Specifically, the candidate word with the largest distribution probability is selected as the output of the t-th word in the corresponding sentence, until the hidden state of the candidate word with the largest probability corresponds to an end mark, then the sentence generation is completed and the iteration is terminated. In addition, after each GRU model generates the description words of its own topic sentence, these topic sentences are connected according to the moment state information to form a total image description paragraph.

[0160] In practical applications, the topic generation model and the word generation model can form an overall image description model, and the loss function of the entire image description model is the weighted sum of the hierarchical recurrent networks of the topic generation model and the thorn generation model. In addition, according to the calculated loss value, the gradient descent algorithm can be used to update the model parameters of the topic generation model, so as to obtain a trained topic generation model and further improve the accuracy of image description.

[0161] The present application discloses an image description method proposed in an embodiment, which includes obtaining an original image; extracting features from the original image to obtain image features; performing region detection on the original image based on the image features to obtain a target region image; extracting features from the target region image to obtain a region feature vector; extracting and processing the region feature vector through a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and the moment state information of the corresponding topic word vector; performing word prediction on the topic data through a word generation model to obtain description words; splicing and processing each description word based on the moment state information to obtain a target description text; wherein the target description text is used to describe the original image. The present application discloses an embodiment, through multiple feature extractions, the generated target description text can contain more image details; in addition, based on the region detection of the original image, the target description text is generated hierarchically using the topic generation model and the word generation model in turn, and a description text with coherent semantics can be generated.

[0162] In some embodiments, Figure 8 As shown, an image description device 800 is also provided, which can implement the above-mentioned image description method. The image description device 800 includes: an image acquisition module 810, a first feature extraction module 820, a region detection module 830, a second feature extraction module 840, a data extraction module 850, a word prediction module 860 and a word splicing module 870, wherein the image acquisition module 810 is used to acquire the original image; the first feature extraction module 820 is used to extract features from the original image to obtain image features; the region detection module 830 is used to perform region detection on the original image according to the image features to obtain the target region image; the second feature extraction module 840 is used to extract features from the target area image to obtain a regional feature vector; the data extraction module 850 is used to extract and process the regional feature vector through a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and the moment state information of the corresponding topic word vector; the word prediction module 860 is used to perform word prediction on the topic data through a word generation model to obtain a description word; the word splicing module 870 is used to splice each description word according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image.

[0163] The image description device 800 of the embodiment disclosed in the present application is used to execute the image description method in the above embodiment. Its specific processing process is the same as the image description method in the above embodiment, and will not be described in detail here.

[0164] The image description device 800 proposed in the disclosed embodiment of the present application obtains an original image; extracts features from the original image to obtain image features; performs region detection on the original image according to the image features to obtain a target region image; extracts features from the target region image to obtain a region feature vector; extracts and processes the region feature vector through a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and the moment state information of the corresponding topic word vector; performs word prediction on the topic data through a word generation model to obtain description words; splices each description word according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image. The disclosed embodiment of the present application can make the generated target description text contain more image details through multiple feature extractions; in addition, based on the region detection of the original image, the target description text is generated hierarchically using the topic generation model and the word generation model in turn, and a description text with coherent semantics can be generated.

[0165] The disclosed embodiment of the present application also provides a computer device, including:

[0166] at least one processor, and

[0167] a memory communicatively connected to at least one processor; wherein,

[0168] The memory stores instructions, and the instructions are executed by at least one processor so that when the at least one processor executes the instructions, an image description method as described in any one of the embodiments of the first aspect of the present application is implemented.

[0169] Combine the following Fig. 9 The hardware structure of the computer device is described in detail. The computer device includes: a processor 910 , a memory 920 , an input / output interface 930 , a communication interface 940 and a bus 950 .

[0170] The processor 910 may be implemented by a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided by the embodiments disclosed in this application;

[0171] The memory 920 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 920 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 920, and the processor 910 calls and executes the image description method of the embodiment disclosed in this application;

[0172] Input / output interface 930, used to implement information input and output;

[0173] Communication interface 940, used to realize communication interaction between the device and other devices, which can be realized by wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.); and

[0174] bus 950 , which transmits information between the various components of the device (e.g., processor 910 , memory 920 , input / output interface 930 , and communication interface 940 );

[0175] The processor 910 , the memory 920 , the input / output interface 930 , and the communication interface 940 are connected to each other in communication within the device via a bus 950 .

[0176] The disclosed embodiment of the present application further provides a storage medium, which is a computer-readable storage medium. The computer-readable storage medium stores computer-executable instructions, and the computer-executable instructions are used to enable a computer to execute the image description method of the disclosed embodiment of the present application.

[0177] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0178] The present application discloses an image description method and apparatus, a computer device, and a storage medium, which are proposed in the embodiments, by acquiring an original image; extracting features from the original image to obtain image features; performing region detection on the original image according to the image features to obtain a target region image; extracting features from the target region image to obtain a region feature vector; extracting and processing the region feature vector through a topic generation model to obtain topic data; wherein the topic data includes a topic word vector and the moment state information of the corresponding topic word vector; performing word prediction on the topic data through a word generation model to obtain description words; splicing and processing each description word according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image. The present application discloses an embodiment, through multiple feature extractions, the generated target description text can contain more image details; in addition, based on the region detection of the original image, the target description text is generated hierarchically using the topic generation model and the word generation model in turn, and a description text with coherent semantics can be generated.

[0179] The embodiments described in the embodiments disclosed in this application are intended to more clearly illustrate the technical solutions of the embodiments disclosed in this application, and do not constitute a limitation on the technical solutions provided by the embodiments disclosed in this application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments disclosed in this application are equally applicable to similar technical problems.

[0180] It can be understood by those skilled in the art that Figures 1 to 7 The technical solutions shown in the figure do not constitute a limitation on the embodiments disclosed in the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.

[0181] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0183] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described here. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0184] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0185] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0186] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0187] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0188] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), disk or optical disk and other media that can store programs.

[0189] The preferred embodiments of the embodiments disclosed in the present application are described above with reference to the accompanying drawings, but the scope of the rights of the embodiments disclosed in the present application is not limited thereby. Any modification, equivalent substitution and improvement made by those skilled in the art without departing from the scope and essence of the embodiments disclosed in the present application shall be within the scope of the rights of the embodiments disclosed in the present application.

Claims

1. An image description method, characterized in that: The method comprises: Get the original image; Performing feature extraction on the original image through a VGG-16 network to obtain image features; wherein the VGG-16 network includes 13 convolutional layers, 3 fully connected layers and 4 pooling layers; Input the image features into RPN to perform region detection on the original image to obtain a target region image; Performing feature mapping on the target area image to obtain a preliminary mapping image; Acquire the first coordinate of the target area image; Calculate a second coordinate according to the first coordinate and a preset size; adjusting the size of the preliminary mapping image according to the second coordinates to obtain a preliminary region image; Performing convolution processing on the preliminary region image to obtain a convolution feature vector; Performing pooling processing on the convolution feature vector to obtain a regional feature vector; Inputting the regional feature vector into a topic generation model; wherein the topic generation model includes a recurrent layer and a hidden layer; The regional feature vector is iteratively processed by the cyclic layer to obtain a topic word vector; Acquire moment state information from the hidden layer according to the subject word vector; Performing word prediction on the subject word vector and the moment state information corresponding to the subject word vector through a word generation model to obtain a description word; wherein the word generation model is a GRU model; Each of the description words is spliced ​​according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image.

2. The method according to claim 1, characterized in that: The step of performing word prediction on the topic data through a word generation model to obtain description words includes: Obtaining an activation function of the word generation model; Inputting the subject word vector and the moment state information into the word generation model; Calculate the subject word vector and the moment state information according to the activation function to obtain at least one candidate word; The description word is obtained from the candidate words.

3. The method according to claim 2, characterized in that The obtaining the description word from the candidate words comprises: Calculating the distribution probability of each candidate word in a preset word set; The candidate word with the largest distribution probability is obtained as the description word.

4. An image description device, characterized in that: The device comprises: Image acquisition module: used to obtain the original image; The first feature extraction module is used to extract features of the original image through a VGG-16 network to obtain image features; wherein the VGG-16 network includes 13 convolutional layers, 3 fully connected layers and 4 pooling layers; Region detection module: used for inputting the image features into RPN to perform region detection on the original image to obtain a target region image; A second feature extraction module: used to perform feature mapping on the target area image to obtain a preliminary mapping image, obtain a first coordinate of the target area image, calculate a second coordinate according to the first coordinate and a preset size, adjust the size of the preliminary mapping image according to the second coordinate to obtain a preliminary area image, perform convolution processing on the preliminary area image to obtain a convolution feature vector, and perform pooling processing on the convolution feature vector to obtain a regional feature vector; Data extraction module: used to input the regional feature vector into the topic generation model; wherein the topic generation model includes a circulation layer and a hidden layer; the regional feature vector is subjected to cyclic iterative processing by the circulation layer to obtain a topic word vector; and the moment state information is obtained from the hidden layer according to the topic word vector; A word prediction module: used to perform word prediction on the subject word vector and the moment state information corresponding to the subject word vector through a word generation model to obtain a description word; wherein the word generation model is a GRU model; Word splicing module: used for splicing each of the description words according to the moment state information to obtain a target description text; wherein the target description text is used to describe the original image.

5. A computer device, characterized in that: The computer device comprises a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor is used to execute: the image description method according to any one of claims 1 to 3.

6. A storage medium, the storage medium being a computer-readable storage medium, characterized in that: The computer readable storage contains a computer program, and when the computer program is executed by a computer, the computer is configured to execute: the image description method according to any one of claims 1 to 3.

Images