Image description generation method, training method, device, medium and system

Abstract

The application discloses an image description generation method, a training method, an apparatus, a medium and a system, and the generation method comprises the following steps: acquiring a to-be-described image, detecting a target bounding box in the to-be-described image and a feature representation of the target bounding box, predicting a relationship between the target bounding boxes according to the target bounding box and the feature representation, constructing a directed graph according to the target bounding box, the feature representation and the relationship, obtaining an intermediate feature set according to the feature representation, performing attention feature extraction according to the intermediate feature set and the directed graph to obtain an attention feature, obtaining a final image feature according to the attention feature and the feature representation, acquiring an initial input text, performing several times of prediction according to the initial input text and the final image feature until a word meeting a preset condition is obtained, and obtaining a description output according to the word meeting the preset condition and the input text of the last time of prediction. The application can improve the accuracy of image description and the fluency of generated sentences, and can be widely applied to the field of image recognition.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image recognition, and in particular to an image description generation method, a training method, an apparatus, a medium and a system. BACKGROUND

[0002] The existing image description methods can be divided into two types: a retrieval-based method and a template matching-based method. The former mainly constructs a picture-text pair, compares the picture to be described with the images in the prepared data set for similarity when performing the image description task, selects the images in the data set that are similar to the picture to be described, takes the description as a candidate sentence, and reorganizes the candidate sentence as the final output of the picture to be described. The latter mainly forms a basic understanding of the image and finds some image features, and then generates a description based on these features. Usually, multiple descriptions are generated, and then a high-level description is selected as the output. The key of this method lies in extracting manual visual features and generating descriptions through a language model.

[0003] However, both of the above methods have some defects. The former method relies too much on the existing data set and cannot form new sentences, but only combines the existing sentences. If there are only a few sentences in the existing data set that are similar to the picture to be described, the output description will not be accurate. The latter method can better describe the visual content of the image than the former method, but the generated sentences are not fluent. SUMMARY

[0004] To solve one of the above problems, the purpose of the present application is to provide an image description generation method, a training method, an apparatus, a medium and a system, which can improve the accuracy of image description and the fluency of generated sentences.

[0005] In one aspect, the present application provides an image description generation method, comprising the following steps:

[0006] An image to be described is obtained, a target bounding box in the image to be described and a feature representation of the target bounding box are detected, and a relationship between the target bounding boxes is predicted according to the target bounding box and the feature representation;

[0007] A directed graph is constructed according to the target bounding box, the feature representation and the relationship. The nodes of the directed graph include the target bounding box and the feature representation, and the edges of the directed graph include the relationship;

[0008] An intermediate feature set is obtained according to the feature representation, an attention feature is extracted according to the intermediate feature set and the directed graph, and a final image feature is obtained according to the attention feature and the feature representation;

[0009] obtaining an initial input text, performing autoregressive prediction for a plurality of times according to the initial input text and the final image feature until a word meeting a preset condition and / or a text meeting a preset condition are obtained, and obtaining a description output of the image to be described according to the word meeting the preset condition and the input text of the last autoregressive prediction.

[0010] Optionally, the target bounding box in the image to be described and the feature representation of the target bounding box are detected, specifically including:

[0011] The overall feature of the image to be described is detected, and a plurality of target bounding boxes are obtained by performing region extraction on the overall feature.

[0012] The overall feature is subjected to feature mapping, and the overall feature after feature mapping is subjected to feature fusion with the plurality of target bounding boxes to obtain the feature representation of each target bounding box.

[0013] Optionally, the relationship between the target bounding boxes is predicted according to the target bounding boxes and the feature representation, specifically including:

[0014] The target bounding boxes are combined in pairs to form a plurality of bounding box pairs, and the intersection value of the two target bounding boxes in each bounding box pair is calculated.

[0015] The relationship between the target bounding boxes is predicted by using the bounding box pair with the intersection value greater than a preset value and the feature representation of the two target bounding boxes in the bounding box pair.

[0016] Optionally, the intermediate feature set includes a first intermediate feature, a second intermediate feature and a third intermediate feature, and the intermediate feature set is obtained according to the feature representation, specifically including:

[0017] The first intermediate feature is extracted from the feature representation according to a first preset feature extraction matrix and a first preset bias.

[0018] The second intermediate feature is extracted from the feature representation according to a second preset feature extraction matrix and a second preset bias.

[0019] The third intermediate feature is extracted from the feature representation according to a third preset feature extraction matrix and a third preset bias.

[0020] Optionally, the attention feature is extracted according to the intermediate feature set and the directed graph, specifically including:

[0021] The correlation coefficient is obtained by extracting the coincident feature of the first intermediate feature from the second intermediate feature.

[0022] The directed graph is used as a mask to screen the correlation coefficient to obtain a mask-screened correlation coefficient.

[0023] The correlation coefficients filtered by the mask are normalized to obtain attention coefficients, and the third intermediate feature is extracted by using the attention coefficients to obtain an attention feature.

[0024] Optionally,

[0025] The self-recurrence prediction is performed several times according to the initial input text and the final image feature until a word meeting a preset condition is obtained, and the self-recurrence prediction specifically includes:

[0026] A first predicted word is obtained according to the initial input text and the final image feature;

[0027] From the second prediction, the last predicted word is spliced with the last predicted input text to obtain the input text for the current prediction;

[0028] The position information of the last predicted word is determined, and the text feature for the current prediction is obtained according to the input text for the current prediction and the position information of the last predicted word;

[0029] The word for the current prediction is obtained according to the text feature for the current prediction and the final image feature, the word for the current prediction is taken as the last predicted word, the input text for the current prediction is taken as the last predicted input text, the step of splicing the last predicted word with the last predicted input text is returned to be executed until the word for the current prediction meets the preset condition, and the word for the current prediction is taken as the word meeting the preset condition. On the other hand, the present application provides a training method of an image description generation network, and the training method specifically includes:

[0030] A sample image is obtained, the sample image is preprocessed, the preprocessed sample image is input into the image description generation network, and a description output of the sample image is obtained; the description output of the sample image is obtained by the above-mentioned generation method;

[0031] A loss value and a gradient are calculated according to the description output of the sample image and a preset loss function, the parameters of the image description generation network are adjusted according to the gradient, the training of the image description generation network is ended until the loss value meets a preset condition.

[0032] On the other hand, the present application provides an image description generation device, which includes:

[0033] At least one processor;

[0034] At least one memory for storing at least one program;

[0035] When the at least one program is executed by the at least one processor, the at least one processor implements the method as described above.

[0036] In another aspect, the present application provides a computer-readable storage medium, which stores a processor-executable program, and the processor-executable program, when executed by a processor, is used to perform the method as described above.

[0037] In another aspect, the present application provides an image description generation system, comprising an image acquisition module and a computer device connected with the image acquisition module; wherein,

[0038] The image acquisition module is configured to acquire a to-be-detected image or a training image, and upload the to-be-detected image or the training image to the computer device.

[0039] The computer device comprises:

[0040] at least one processor;

[0041] at least one memory for storing at least one program;

[0042] When the at least one program is executed by the at least one processor, the at least one processor implements the method as described above.

[0043] The implementation of the present application has the following beneficial effects: the present application fully excavates the potential structural relationship between objects in the image by acquiring a to-be-described image, detecting target bounding boxes and feature representations of the target bounding boxes in the to-be-described image, predicting the relationship between the target bounding boxes according to the target bounding boxes and the feature representations, and constructing a directed graph according to the target bounding boxes, the feature representations and the relationship, wherein the nodes of the directed graph include the target bounding boxes and the feature representations, and the edges of the directed graph include the relationship, thereby improving the accuracy of subsequent image description; the present application improves the accuracy of image description and the fluency of generated sentences by obtaining an intermediate feature set according to the feature representations, performing attention feature extraction according to the intermediate feature set and the directed graph to obtain attention features, obtaining final image features according to the attention features and the feature representations, obtaining an initial input text, performing autoregressive prediction several times according to the initial input text and the final image features until a word satisfying a preset condition and / or a text satisfying a preset condition are obtained, and obtaining a description output of the to-be-described image according to the word satisfying the preset condition and the input text of the last autoregressive prediction. BRIEF DESCRIPTION OF DRAWINGS

[0044] Figure 1is a step flow chart of a method for generating an image description provided by the present application;

[0045] Figure 2 is a process schematic diagram of a method for generating an image description provided by the present application;

[0046] Figure 3 is a structure and process schematic diagram of an image encoder provided by the present application;

[0047] Figure 4 is a step flow chart of a method for training an image description generation network provided by the present application;

[0048] Figure 5 is a structure schematic diagram of an image description generation device provided by the present application;

[0049] Figure 6 is a structure schematic diagram of an image description generation system provided by the present application. DETAILED DESCRIPTION

[0050] The present application will be further described below in conjunction with the accompanying drawings and specific embodiments. For the step numbers in the following embodiments, they are only set for the convenience of description and explanation, and the order between the steps is not limited in any way, and the execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0051] It should be noted that although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described can be executed in a different order than the module division in the system or the order in the flow chart. The terms "first / S100", "second / S200" and the like in the specification and claims and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0052] Reference herein to "embodiment" means that the particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the present application. The appearance of the phrase in various places in the specification does not necessarily all refer to the same embodiment, nor is it necessarily mutually exclusive of other embodiments. It is explicitly and implicitly understood that the embodiments described herein can be combined.

[0053] The image description generation method disclosed by the present application can be applied to an image description generation network, and the image description generation network includes a target detection network, a predicted relationship network and an image description network.

[0054] As shown in Figure 1 Figure 1 ​is a step flow chart of an image description generation method, in some embodiments, the present application provides an image description generation method, which comprises the following steps:

[0055] S100, obtaining an image to be described, detecting target bounding boxes in the image to be described and feature representations of the target bounding boxes, and predicting relationships between target bounding boxes according to the target bounding boxes and the feature representations.

[0056] Specifically, a target detection network is used to detect a plurality of target bounding boxes and the feature representation of each target bounding box in the image to be described, for example, the image to be described is input into a faster-rcnn network, and a plurality of target bounding boxes and the feature representation of each target bounding box are obtained according to formula (1):

[0057] box, f box =faster-rcnn(I) (1)

[0058] Wherein, faster-rcnn() is a target detection network, I is an image to be described, box is a detected target bounding box, f box is the feature representation of the content in the target bounding box.

[0059] The faster-rcnn network includes ResNet101, a region proposal network, a feature mapping module and a feature fusion module, as shown in Figure 2 Figure 2 is a process diagram of image description generation, wherein ① represents a directed graph, ② represents a neighbor matrix, A represents "on", B represents "inside", C represents "beside", target bounding boxes in the image to be described and feature representations of the target bounding boxes are detected, and the specific steps include:

[0060] S110, detecting the overall feature of the image to be described, and performing region extraction on the overall feature to obtain a plurality of target bounding boxes.

[0061] Specifically, ResNet101 extracts the overall feature of the image to be described, and inputs the overall feature into a region proposal network to obtain a plurality of target bounding boxes.S120, feature mapping is performed on the overall feature, and the feature-mapped overall feature is fused with a plurality of target bounding boxes to obtain the feature representation of each target bounding box.

[0062] Specifically, the overall feature is feature-mapped, and the feature-mapped feature is fused with the region feature to obtain the feature representation of each target bounding box, and the final feature representation f box is determined by predicting each target bounding box, wherein the feature representation can be a feature vector. ​

[0063] In some embodiments, predicting the relationship between the target bounding box and the feature representation specifically includes:

[0064] S130. Combine the target bounding boxes in pairs to form several bounding box pairs, and calculate the intersection value of the two target bounding boxes in each bounding box pair.

[0065] Specifically, the target bounding boxes obtained in step S110 are combined in pairs to form several bounding boxes. It can be understood that several includes one or more, for example, forming three target bounding boxes H, G and R. The two bounding boxes are combined in pairs to form three bounding box pairs HG, HR and GR.

[0066] Calculate the intersection value (IoU value) between the two target bounding boxes according to formula (2):

[0067]

[0068] Here, box1 and box2 are two different target bounding boxes.

[0069] It is worth noting that in pairwise intersections, there is a distinction between subject and object. The subject is the one seeking combination, and the object is the one being combined. In the following explanation, to avoid confusion, it is agreed that the term preceding the formula is the subject, and the term following the formula is the object.

[0070] S140. The relationship between the target bounding boxes is predicted by using the bounding box pairs with cross values ​​greater than a preset value and the features of the two target bounding boxes in the bounding box pairs.

[0071] Specifically, the feature representations of two target bounding boxes with an intersection value greater than a preset value, such as 0.3, are concatenated according to formula (3):

[0072] f = concat(f1) box f2 box (3)

[0073] Among them, f1 box As the main feature representation, f2 box Let f be the feature representation of the object, and f be the feature representation after fusion. Then, the fused feature f is passed through two linear layers to obtain the relationship f4 between the two target bounding boxes, as shown in formulas (4)-(7):

[0074] f1 = Linear1(f) (4)

[0075] f2 = ReLU(f1) (5)

[0076] f3 = Linear2(f2) (6)

[0077] f4 = ReLU(f3) (7)

[0078] In this layer, Linear1() is a downsampling linear layer with a size of 4096*2048, ReLU() is a non-linear activation function, and Linear2() is an upsampling linear layer with a size of 2048*4096. After feature extraction through two linear layers, the relationship between the two target bounding boxes is obtained, and the relationship is directional.

[0079] S200. A directed graph is constructed based on the target bounding box, the feature representation, and the relationship.

[0080] The nodes of the directed graph include the target bounding box and the feature representation, and the edges of the directed graph include the relations.

[0081] Specifically, the target bounding boxes, feature representations and relationships obtained above are used to represent bounding boxes with cross values ​​greater than a preset value as triplets of subject, relation and object, resulting in several triplets. By merging triplets with the same target based on subject, relation and object, several directed graphs can be obtained.

[0082] S300. Obtain an intermediate feature set based on the feature representation, extract attention features based on the intermediate feature set and the directed graph, and obtain the final image features based on the attention features and the feature representation.

[0083] Specifically, such as Figure 3 As shown, Figure 3 This is a schematic diagram of the structure and processing of an image encoder. The above steps can be implemented using an image encoder containing a self-attention module, such as a transformer-based encoder structure, using the feature representation f of the target bounding box obtained in step S120 above. box As input, several intermediate features are calculated to form an intermediate feature set, for example, three intermediate features: the first intermediate feature, the second intermediate feature, and the third intermediate feature.

[0084] In some embodiments, the process of obtaining the three intermediate features Q, K, and V specifically includes:

[0085] S310. Extract the first intermediate feature from the feature representation according to the first preset feature extraction matrix and the first preset bias.

[0086] Specifically, the first preset feature is W Q The first preset bias is b Q According to formula (8), the first intermediate feature Q is obtained:

[0087] Q = W Q f box +b Q (8)

[0088] S320、according to the second preset feature extraction matrix and the second preset bias, extracting a second intermediate feature from the feature representation.

[0089] Specifically, the second preset feature is W K , the second preset bias is b K , and the second intermediate feature K is obtained according to formula (9):

[0090] K = W K f box +b K (9)

[0091] S330、according to the third preset feature extraction matrix and the third preset bias, extracting a third intermediate feature from the feature representation.

[0092] As shown in formula (10):

[0093] Specifically, the third preset feature is W V , the third preset bias is b V , and the third intermediate feature V is obtained according to formula (10):

[0094] V = W V f box +b V (10)

[0095] wherein, the element values in the first preset feature extraction matrix to the third preset feature extraction matrix are different, and the element values in the first preset bias to the third preset bias are different.

[0096] In some embodiments, according to the intermediate feature set and the directed graph, attention feature extraction is performed to obtain attention features, specifically including:

[0097] S340, extracting a coincidence feature of the first intermediate feature from the second intermediate feature to obtain a correlation coefficient.

[0098] Specifically, taking the first intermediate feature Q as a template, the coincidence feature of the first intermediate feature Q in the second intermediate feature K is extracted, the coincidence feature refers to the similar features in Q and K, and the correlation coefficient of the vector in K relative to the vector in Q is obtained according to formula (11):

[0099]

[0100] wherein, a is the correlation coefficient of K relative to Q, dim is the dimension size of f box , and * is multiplication.

[0101] S350, screen the correlation coefficients with the directed graph as a mask to obtain mask-screened correlation coefficients.

[0102] Specifically, the directed graph obtained in the foregoing step is converted into an adjacency matrix, and the adjacency matrix is used as a mask to screen the correlation coefficients a, as shown in formula (12):

[0103] a' = a & D (12)

[0104] where D is the adjacency matrix, a' is the correlation coefficient after mask screening, & is a relationship and operation, and by using the directed graph in the form of the adjacency matrix as a mask, the parts of the image that are worth noting can be screened out, and the amount of calculation is reduced.

[0105] S360, normalize the mask-screened correlation coefficients to obtain attention coefficients, and use the attention coefficients to perform feature extraction on the third intermediate feature to obtain an attention feature.

[0106] Specifically, the mask-screened correlation coefficients are normalized by using a softmax() method to obtain final attention coefficients attn, as shown in formula (13):

[0107]

[0108] where a ′ The mask-screened correlation coefficients include a feature representation of all target bounding boxes, a i ′ is any one of the mask-screened correlation coefficients, a j is any one of the correlation coefficients.

[0109] Finally, the features in the third intermediate feature V are screened by using the attention coefficients, and the attention feature extraction is completed, as shown in formula (14):

[0110] x = attn * V (14)

[0111] where x is the result after the attention feature extraction.

[0112] In some embodiments, obtaining a final image feature according to the attention feature and the feature representation specifically includes:

[0113] The attention feature x is fused with the feature representation f box After passing through the Add & Norm() layer, the Add & Norm() structure includes two linear layers, the feature fusion after passing through the two linear layers completes the entire image feature extraction process, as shown in formulas (15) and (16):

[0114] x' = Add & Norm(x) = LayerNorm(f box + x) (15)

[0115] x1' = Linear2(ReLU(Linear1( x′ )) (16)

[0116] where LayerNorm() is a function of processing dimension, x1' is the final image feature obtained by encoding, Linear1() and Linear2() are two linear functions, ReLU() is an activation function, and the entire encoder has 6 layers.

[0117] S400, obtaining an initial input text, and performing several times of autoregressive prediction according to the initial input text and the final image feature until a word meeting a preset condition and / or a text meeting a preset condition are obtained, and obtaining a description output of the image to be described according to the word meeting the preset condition and the input text of the last autoregressive prediction.

[0118] Specifically, as Figure 2 shown, the text encoder is used to process the input text to obtain a text feature, the structure of the text encoder is the same as that of the image encoder without the mask module, and the process of obtaining the text feature is as follows: taking the input text as input, calculating several intermediate features to form an intermediate feature set, for example, three intermediate features, a first intermediate feature, a second intermediate feature and a third intermediate feature, omitting the process of mask screening, and the above process extracts image features, and in this step, text features are extracted.

[0119] The text feature and the final image feature are input into the cross-modal attention module to obtain a description output.

[0120] The process of several times of autoregressive prediction specifically includes the following steps:

[0121] S410, obtaining a first predicted word according to the initial input text and the final image feature.

[0122] Specifically, the initial input text can be but is not limited to a preset word, the preset word is input into the above text encoder, the output of the text encoder is a text feature, and the text feature and the image feature are input into the cross-modal attention module, and the output thereof is the first predicted word.

[0123] The processing process of the cross-modal attention module is as follows:

[0124] The structure of the cross-modal attention module is the same as Figure 3The structure of the second prediction model is similar to that of the first prediction model, except that there is no mask input module, and no mask screening is needed. The input is the final image feature x1' and the text feature t. Wherein, Q is the text feature t, K, V are all image features x1', and there is no mask module. The final fusion feature Y is obtained as shown in formula (17):

[0125] Y = crossmodelAttn(x1', t) (17)

[0126] Wherein, crossmodelAttn() is a cross-modal attention function.

[0127] After obtaining the fusion feature, it is put into a linear layer with a size of 512x the number of words in the word table, and then the final fusion feature Y is normalized by the softmax() function as shown in formula (18):

[0128] T = softmax(Linear(Y)) (18)

[0129] Wherein, T is the probability of each word in the word table being selected in the prediction at time t. Finally, the word with the highest value in T is taken as the output word at the current time, that is, the input word at the next time.

[0130] For example, the first prediction, the preset word is SOS, which represents the start of the sequence. The text encoder processes SOS to obtain the text feature, and inputs the text feature and the image feature x1' into the cross-modal attention module to obtain a word E.

[0131] S420, from the second prediction, the last predicted word and the input text of the last prediction are spliced to obtain the input text of the current prediction, the position information of the last predicted word is determined, and the text feature of the current prediction is obtained according to the input text of the current prediction and the position information of the last predicted word.

[0132] Wherein, the spliced input text includes all the words before the last prediction (including the last prediction).

[0133] Specifically, the position information of the last predicted word is determined as shown in formula (19)-(20):

[0134]

[0135]

[0136] Wherein, p is the position of the word in the sentence, d is the total dimension of the word encoding, i is the position of the current dimension, and PE(p,2i) and PE(p,2i+1) are position information, that is, a marker representing the order of words.

[0137] The token representing the order of the word and the spliced input text are input into the text editor to obtain a text feature of the current prediction.

[0138] The formulas (19)-(20) are calculated by dimension, the position of the last predicted word in the spliced input text is taken as input to obtain a 1x1x512 size position vector, then the last predicted word is added before the text encoder to obtain a text feature added with position information, and the text feature is input into the text encoder for text feature extraction. For example, the preset word is SOS, the word obtained by the first prediction is E, and the text feature output by the text editor in the current prediction (the second prediction) is [SOS, E].

[0139] For example, the word output by the second prediction is F, the position information of the word F is obtained after the word F is obtained, the word F and the position information thereof are input into the text editor together with [SOS, E], and the text feature output is [SOS, E, F].

[0140] S430, obtaining a word of the current prediction according to the text feature of the current prediction and the final image feature, taking the word of the current prediction as the word of the last prediction, and taking the input text of the current prediction as the input text of the last prediction, returning to execute the step of splicing the word of the last prediction and the input text of the last prediction until the word of the current prediction satisfies a preset condition, and taking the word of the current prediction as the word satisfying the preset condition.

[0141] The preset condition can be, but is not limited to, that the word output by the current prediction is EOS or that the text obtained by splicing the word output by the current prediction and the input text of the current prediction satisfies a preset length seqlen, EOS represents end of sequence, and in application, the word is output by time step, only one word is output by one time step, and the time step can be regarded as a specified length of a sentence.

[0142] Specifically, the process of obtaining the current prediction word according to the current prediction text feature and the final image feature is the same as step S410, the difference is that the input is the current prediction text feature and the final image feature, and the output is the current prediction word. After obtaining the current prediction word, the current prediction is taken as the last prediction, the input text of the current prediction is taken as the input text of the last prediction, and the step of splicing the last prediction word and the input text of the last prediction is returned to be executed until the current prediction word meets the preset condition, and the loop is ended. Until the current prediction word meets the preset condition, the loop is ended, which can include but is not limited to including until the last word output is EOS, the loop is ended, or the text obtained by splicing the word output by the current prediction and the input text of the current prediction meets the preset length seqlen, the loop is ended, and all the words at all times are the description output of the image to be described.

[0143] Implementing the present application includes the following benefits:

[0144] The present application constructs a directed graph of the bounding box of each object in the image, fully excavates the potential structural relationship between each object in the image, and improves the accuracy of image description. In semantic description, not only the sentence is matched according to the image feature, but also the word is output according to the text feature of each prediction and the final image feature in order, and the description output of the image to be described is obtained, which improves the accuracy of image description and the fluency of generated sentences.

[0145] In some embodiments, the present application provides a training method of an image description generation network, which specifically includes:

[0146] Obtaining a sample image, pre-processing the sample image, inputting the pre-processed sample image into the image description generation network, and obtaining the description output of the sample image; the description output of the sample image is obtained by the above-mentioned generation method;

[0147] According to the description output of the sample image and the preset loss function, the loss value and the gradient are calculated, the parameters of the image description generation network are adjusted according to the gradient, and the training of the image description generation network is ended until the loss value meets the preset condition.

[0148] In some embodiments, as shown in Figure 4 , the training method of the image description generation network includes the following steps: Figure 4 is a step flow chart of a training method of an image description generation network, the image description generation network includes an object detection network, a prediction relationship network and an image description network, and the specific process of the image description generation network is as follows:

[0149] S10, obtaining a sample image, preprocessing the sample image, training a target detection network by using the preprocessed sample image, and obtaining a trained target detection network.

[0150] The target detection network is used for detecting a target bounding box in the sample image and a feature representation of the target bounding box.

[0151] The description output is obtained according to the foregoing generation method.

[0152] The sample image is obtained by the following method.

[0153] The data set is prepared, and each subdivided field image data set constructed by itself can be directly used, or various public data sets can be used, for example, a public data set Visual Genome data set and MSCOCO data set are used, the Visual Genome data set contains 100,000 images, each image is labeled with objects, attributes and pair relationships between objects, the MSCOCO data set is an open source public data set for image recognition, description, detection research and competition, contains more than 120,000 images, each image has 5 description sentences, the data is more abundant, and a model with more perfect description can be trained, and the objects in the MSCOCO data set are completely contained in the Visual Genome data set.

[0154] There are many target categories in the Visual Genome data set, in order to simplify the calculation and retain a certain generalization performance, the images in the Visual Genome data set are screened, and the screening conditions include:

[0155] (1) The target object appears in the data set enough significantly.

[0156] (2) The target in the MSCOCO data set must be completely contained.

[0157] Therefore, the target screening result is 200 target categories with the most appearance times in the Visual Genome data set and all targets in the MSCOCO data set.

[0158] The preprocessing specifically includes the following steps:

[0159] The image is normalized, the input image is scaled to 592*592*3 size image, then a circle of 0 value is added outside the image boundary, and zero-padding operation is performed, so that in the convolution process, the boundary information of the image can be properly reserved without rapid attenuation.

[0160] When the image is zoomed, the bounding box needs to be scaled according to the size of the image zooming to prevent the bounding box from being offset due to the change of the image size. The scaling method adopted by the bounding box is shown in formula (21):

[0161]

[0162] wherein newbox is the scaled bounding box, oldbox is the bounding box before scaling, factor is the scaling factor of the bounding box, w is the length of the image, h is the width of the image, and the format of newbox and oldbox is (x1, y1, x2, y2), (x1, y1) is the coordinate of the upper left corner of the image, and (x2, y2) is the coordinate of the lower right corner of the image.

[0163] Specifically, a target detection network is trained by using sample images in a preprocessed Visual Genome dataset, and the processing process of the target detection network is shown in steps S110 and S120.

[0164] S20, training a prediction relationship network by using the preprocessed training images and the trained target detection network to obtain a trained prediction relationship network.

[0165] The prediction relationship network is used to predict the relationship between the target bounding boxes according to the target bounding boxes and the feature representations, and a directed graph is formed according to the target bounding boxes, the feature representations and the relationship; the nodes of the directed graph include the target bounding boxes and the feature representations, and the edges of the directed graph include the relationship.

[0166] Specifically, sample images in a preprocessed Visual Genome dataset are input into a target detection network to obtain bounding boxes and feature representations of the bounding boxes, and a relationship prediction network is trained by using the bounding boxes and the feature representations of the bounding boxes to obtain a trained relationship prediction network. The processing process of the relationship prediction network is shown in steps S130, S140, S200 and the sub-steps thereof, which will not be repeated here.

[0167] S30, training an image description network by using the preprocessed sample images, the trained target detection network and the trained prediction relationship network to obtain a trained image description network.

[0168] The image description network is used to obtain an intermediate feature set according to the feature representations, to perform attention feature extraction according to the intermediate feature set and the directed graph to obtain attention features, to obtain final image features according to the attention features and the feature representations, to determine text features, and to obtain a description output of the sample image according to the text features and the final image features.

[0169] Specifically, the sample images from the preprocessed MSCOCO dataset are input into the trained object detection network to detect bounding boxes and their feature representations. The bounding boxes and their feature representations are then input into the trained relation prediction network to obtain a directed graph. This directed graph is then input into the image description network to train the image description network.

[0170] The image description network processing procedure is shown in step S300 and its sub-steps, and step S400 and its sub-steps. It is worth noting that in step S400 and its sub-steps, during training, for the input text vector, one-hot encoding is first used to convert the text into vector form, and then a linear layer is used to transform it into the desired dimension, which is 512 dimensions in this invention. The other steps are the same and will not be described again here.

[0171] The cross-entropy loss function of the training image description network is shown in Equation (22):

[0172]

[0173] Where M is the number of word categories, N is the total number of sentences used for training, and y ic Let be the sign function, meaning that in the i-th sentence, at the current moment, if word c appears, it is set to 1; otherwise, it is set to 0. ic Let be the probability of word c appearing in the i-th sentence at the current moment.

[0174] In some embodiments, such as Figure 5 As shown, Figure 5 This is a schematic diagram of an image description generation device. The present invention also provides an image description generation device, comprising:

[0175] At least one processor;

[0176] At least one memory for storing at least one program;

[0177] When the at least one program is executed by the at least one processor, the at least one processor implements the generation method and training method steps described in the above method embodiments.

[0178] The memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. The memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include remote memory located remotely relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0179] As Figure 6 shown, Figure 6 is a structural schematic diagram of an image description generation system, and the present application further provides an image description generation system, comprising an image acquisition module and a computer device connected with the image acquisition module; wherein,

[0180] The image acquisition module is configured to acquire a to-be-detected image or a training image, and upload the to-be-detected image or the training image to the computer device.

[0181] The computer device comprises:

[0182] at least one processor;

[0183] at least one memory for storing at least one program;

[0184] When the at least one program is executed by the at least one processor, the at least one processor implements the above-mentioned generation method and training method.

[0185] Specifically, the image acquisition module can specifically comprise at least one camera, and the computer device can be different types of electronic devices, including but not limited to desktop computers, laptop computers, and other terminals.

[0186] It can be seen that the content in the above method embodiments is applicable to the present system embodiments, the present system embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0187] It can be seen that the content in the above method embodiments is applicable to the present device embodiments, the present device embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0188] In addition, the present application also discloses a computer program product or a computer program, which is stored in a computer readable storage medium. The processor of the computer device can read the computer program from the computer readable storage medium, and the processor executes the computer program, so that the computer device executes the above-mentioned method. Similarly, the content in the above method embodiments is applicable to the present storage medium embodiments, the present storage medium embodiments specifically implement the same functions as the above method embodiments, and achieve the same beneficial effects as the above method embodiments.

[0189] The application further provides a computer readable storage medium, which stores a processor executable program, and the processor executable program is used for realizing the method when being executed by a processor.

[0190] It can be understood that all or some steps in the method disclosed above can be implemented as software, firmware, hardware and appropriate combinations thereof. Some or all physical components can be implemented as software executed by a processor, such as a central processing unit, a digital signal processor or a microprocessor, or as hardware, or as an integrated circuit, such as an application specific integrated circuit. Such software can be distributed on a computer readable medium, which can include computer storage media (or non-transitory media) and communication media (or transitory media). As known by those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by a computer. In addition, as known by those skilled in the art, communication media generally includes computer readable instructions, data structures, program modules or other data in a modulated data signal such as a carrier wave or other transport mechanism, and can include any information delivery medium.

[0191] The above is a specific description of the preferred embodiments of the application, but the application is not limited to the embodiments described above. Those skilled in the art can make various equivalent modifications or replacements without departing from the spirit of the application. These equivalent modifications or replacements are all included in the scope defined by the claims of the present application.

Claims

1. An image description generation method characterized by, The method comprises the following steps: obtaining a to-be-described image, detecting a target bounding box in the to-be-described image and a feature representation of the target bounding box, and predicting a relationship between target bounding boxes according to the target bounding box and the feature representation; constructing a directed graph according to the target bounding box, the feature representation and the relationship; the nodes of the directed graph comprise the target bounding box and the feature representation, and the edges of the directed graph comprise the relationship; obtaining an intermediate feature set according to the feature representation, performing attention feature extraction according to the intermediate feature set and the directed graph to obtain an attention feature, and obtaining a final image feature according to the attention feature and the feature representation; obtaining an initial input text, performing autoregressive prediction on the initial input text and the final image feature for a plurality of times until a word satisfying a preset condition and / or a text satisfying a preset condition are obtained, and obtaining a description output of the to-be-described image according to the word satisfying the preset condition and the input text of the last autoregressive prediction.

2. The generation method of claim 1, wherein, The detection of the target bounding box in the to-be-described image and the feature representation of the target bounding box specifically comprises: detecting an overall feature of the to-be-described image, performing region extraction on the overall feature to obtain a plurality of target bounding boxes; performing feature mapping on the overall feature, and performing feature fusion on the overall feature after feature mapping and the plurality of target bounding boxes to obtain a feature representation of each target bounding box.

3. The generation method of claim 1, wherein, The prediction of the relationship between the target bounding boxes according to the target bounding box and the feature representation specifically comprises: combining the target bounding boxes in pairs to form a plurality of bounding box pairs, and calculating the intersection value of the two target bounding boxes in each bounding box pair; using the bounding box pairs with the intersection value greater than a preset value and the feature representations of the two target bounding boxes in the bounding box pairs to predict the relationship between the target bounding boxes.

4. The generation method of claim 1, wherein, The intermediate feature set comprises a first intermediate feature, a second intermediate feature and a third intermediate feature, and the obtaining of the intermediate feature set according to the feature representation specifically comprises: extracting the first intermediate feature from the feature representation according to a first preset feature extraction matrix and a first preset bias; extracting the second intermediate feature from the feature representation according to a second preset feature extraction matrix and a second preset bias, and extracting the third intermediate feature from the feature representation according to a third preset feature extraction matrix and a third preset bias.

5. The method of generating according to any of claims 4, characterized in that, The attention feature extraction according to the intermediate feature set and the directed graph to obtain an attention feature specifically comprises: extracting a correlation coefficient from the second intermediate feature by matching the first intermediate feature; using the directed graph as a mask to screen the correlation coefficient to obtain a mask-screened correlation coefficient; performing normalization processing on the mask-screened correlation coefficient to obtain an attention coefficient, using the attention coefficient to perform feature extraction on the third intermediate feature to obtain an attention feature.

6. The method of generating according to any one of claims 1-5, wherein, The autoregressive prediction on the initial input text and the final image feature for a plurality of times until a word satisfying a preset condition is obtained specifically comprises: obtaining a first predicted word according to the initial input text and the final image feature; From the second prediction, the last predicted word is spliced with the last predicted input text to obtain the input text of the current prediction; The position information of the last predicted word is determined, and the text feature of the current prediction is obtained according to the input text of the current prediction and the position information of the last predicted word; The current predicted word is obtained according to the text feature of the current prediction and the final image feature, and the current predicted word is taken as the last predicted word, the input text of the current prediction is taken as the last predicted input text, and the step of splicing the last predicted word with the last predicted input text is returned to be executed until the current predicted word meets the preset condition, and the current predicted word is taken as the word meeting the preset condition. 7.A method for training an image description generation network, characterized in that, The training method specifically comprises: Obtaining a sample image, pre-processing the sample image, inputting the pre-processed sample image into an image description generation network, and obtaining a description output of the sample image; the description output of the sample image is obtained by the method of any one of claims 1-6; According to the description output of the sample image and the preset loss function, the loss value and the gradient are calculated, the parameters of the image description generation network are adjusted according to the gradient, and the training of the image description generation network is ended until the loss value meets the preset condition.

8. An image description generation apparatus characterized by comprising: Comprise: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-7.

9. A computer readable storage medium having stored therein a program that is executable by a processor, characterized in that, The program executable by the processor is used to execute the method of any one of claims 1-7 when executed by the processor.

10. An image description generation system characterized by comprising: Comprise an image acquisition module and a computer device connected with the image acquisition module; wherein, The image acquisition module is used for acquiring a to-be-detected image or a training image, and uploading the to-be-detected image or the training image to the computer device; The computer device comprises: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor implements the method of any one of claims 1-7.

Images