Image description method and device fusing object position relationship

Abstract

The application discloses a kind of fusion object position relationship's image description method and device, which comprises the following steps: S1.input image to be described, extract the global feature information of image to be described and candidate frame feature information;S2. according to the feature information extracted, construct object relationship scene graph;S3. the node information of each kind contained in object relationship scene graph is initially encoded;If there is proportion imbalance phenomenon in the position relationship proportion between object nodes in object relationship scene graph compared with the original position relationship of object, then according to the proportion imbalance degree, the object relationship coding feature matrix obtained by initially encoding node information is secondarily encoded;S4. the object relationship scene graph and node information after coding are input into joint decoder for joint decoding, and the text information of the image to be described is predicted.The application can fully fuse the object position relationship in image, realize efficient and accurate image description.

Description

Technical Field

[0001] The present invention relates to the field of image description, and in particular to an image description method and device integrating positional relationships of objects. Background Art

[0002] Image description technology uses the visual features of an image and the corresponding text features to enable a computer to generate a text description that is consistent with a human description. The main detection targets of image description are usually objects such as people, mountains, rivers, and landmark buildings. The difficulty lies in not only detecting the objects in the image, but also understanding the relationship between the objects and then generating a reasonable text description. In the existing technology, image description usually uses a convolutional neural network to extract visual features, and after decoding, it generates a natural sentence describing the image content. The visual features are usually features such as color, shape, or texture, and often do not pay attention to the positional relationship between objects in the image. Therefore, it is impossible to accurately capture and describe the position information of objects hidden in the image. For the same object, different shooting angles and composition methods will contain different positional relationship information. The positional relationship between objects can be used to mine valuable data hidden in the image. Traditional image description methods lack the capture and description of image position information, resulting in the final generated image description being not very accurate.

[0003] Image description requires processing data from two different modalities: image and text. This is cross-modal processing, and there is a semantic gap between the visual features of the image and the annotated text features. If you want to include positional information that accurately describes the positional relationships of objects in the image description, the following problems will arise:

[0004] 1. Images are often affected by the shooting angle, and the object information they contain is prone to disproportion. This phenomenon can also cause the model to incorrectly generate incomplete and unreasonable descriptions of object position relationships;

[0005] 2. The number of positional relationship words in the dataset annotation information is usually small, and only contains some simple, colloquial prepositional phrases. However, the annotation information has a bias problem and cannot directly generate implicit positional information relationship words.

[0006] Image description can be applied to various types of downstream application tasks. For example, in the AI+ market, by analyzing images, users can automatically generate corresponding experience texts, and generate personalized, multi-style, and multi-genre texts based on user preferences, thereby improving the intelligence and practicality of social software. For example, for travel photos, image description can be used to classify, retrieve, and describe information such as tourist types, geographical locations, landmarks, and natural scenery in travel photos. When image description in the existing technology is applied to the above-mentioned types of practical application tasks, it is difficult to achieve accurate image description because it cannot accurately capture the positional relationship between objects. SUMMARY

[0007] The technical problem solved by the present application is to provide a method and device for image description fusing object position relationship, which is simple to implement, efficient and accurate, can effectively mine and capture the position relationship and detail information between objects, and realize more accurate and reasonable image description.

[0008] To solve the above technical problems, the technical solution provided by the present application is as follows:

[0009] A method for image description fusing object position relationship, comprising the following steps:

[0010] Step S1. Input an image to be described, and extract global feature information and candidate box feature information of the image to be described respectively.

[0011] Step S2. Construct an object relationship scene graph G according to the extracted global feature information and candidate box feature information, wherein the object relationship scene graph G comprises feature nodes A, object nodes O and relationship nodes R; the feature nodes A correspond to feature information comprising the feature information of the corresponding candidate box in the candidate box feature information, the object nodes O correspond to object feature information comprising the feature information of the corresponding object in the candidate box feature information, and the relationship nodes R correspond to feature information and object information comprising the global feature information and the feature information of the corresponding candidate box in the candidate box feature information.

[0012] Step S3. Perform primary encoding on each type of node information in the object relationship scene graph G to obtain an encoded feature matrix of each type of node; if there is a proportion imbalance phenomenon between the position relationship of the object nodes in the object relationship scene graph G compared with the original position relationship of the object, then perform secondary encoding on the object relationship encoded feature matrix obtained by primary encoding of the relationship nodes R according to the proportion imbalance degree to obtain a secondary encoded object relationship encoded feature matrix.

[0013] Step S4. Input the encoded object relationship scene graph G and node information into a joint decoder for joint decoding to predict text information of the image to be described.

[0014] Further, in step S3, each type of node information in the object relationship scene graph G is encoded according to the following formula:

[0015]

[0016] wherein x i is the encoded feature matrix of the node in the object relationship scene graph G, v i is the feature information matrix, w [0] represents the embedding coding matrix corresponding to the object node O, w [1] represents the embedding coding matrix corresponding to the feature node A, w [2] represents the embedding coding matrix corresponding to the relationship node R, and d represents the dimension of the input feature vector.

[0017] Further, the step S3 of performing secondary coding on the object relationship coding feature matrix obtained by initially coding the relationship node R according to the degree of proportion imbalance comprises:

[0018] Step S31. A knowledge base dictionary is constructed from the plurality of object nouns and the plurality of position relationship words with the highest frequency selected from the image description data set, and the knowledge base dictionary is coded into the object relationship scene graph G;

[0019] Step S32. The position relationship between adjacent object nodes in the object relationship scene graph G is calculated according to the knowledge base dictionary and the boundary box position information of the object node O, and the knowledge base dictionary stores a plurality of object nouns and a plurality of position relationship words with the highest frequency;

[0020] Step S33. Whether there is a proportion imbalance phenomenon is determined according to the calculated position relationship between adjacent object nodes, and if there is, the step S34 is entered, otherwise the current calculation is exited;

[0021] Step S34. The secondary coding is performed on the object relationship coding feature matrix obtained by initially coding the relationship node R according to the degree of proportion imbalance, to obtain the object relationship coding feature matrix after secondary coding.

[0022] Further, the knowledge base dictionary is coded into the node category embedding coding matrix w α in the step S31 by using the one-hot coding method; in the step S33, whether there is a proportion imbalance phenomenon is determined by comparing the calculated position relationship between adjacent object nodes with the original position relationship proportion in the proportion common sense priori knowledge, and the proportion common sense priori knowledge stores a plurality of original position relationship proportions of objects and corresponding position relationship words.

[0023] Further, the secondary coding is performed according to the following formula in the step S34:

[0024] z i = x i ⊙ w α [q]

[0025] wherein z i is the object relationship coding feature matrix after secondary coding, x i is the object relationship coding feature matrix obtained after initial coding, is an embedding coding matrix corresponding to the proportion imbalance degree category, q represents the proportion imbalance category, and the proportion imbalance category is obtained by dividing the proportion imbalance degree value.

[0026] Further, in step S4, the GCN (graph convolution network) graph structure and node information of the encoded object relationship scene graph G are decoded using a double-layer LSTM (Long Short-Term Memory, time recurrent network) structure, and a bias gating mechanism is used to judge the generated prediction word. If the generated prediction word belongs to the pre-constructed relationship word fault-tolerant dictionary, the bias gating mechanism is used to update the double-layer LSTM.

[0027] Further, the bias gating mechanism is used to update the double-layer LSTM in , as shown in the following formula:

[0028]

[0029]

[0030] wherein z t is a bias gating mechanism that controls the retention degree of candidate information, w t-1 is the input information of the Attention LSTM layer at the current time t, is the output information of the Attention LSTM layer at the current time t, and sigma (·) is a Sigmoid activation function, ensuring that z t is 0-1.

[0031] Further, step S4 also includes an update and erase step for the object relationship scene graph node, including:

[0032] Each time the decoded prediction word is judged, if it is judged to be a word that does not belong to the object position relationship, the current object relationship scene graph G structure is not changed; if it is judged to be a word that belongs to the object position relationship, the current object relationship scene graph G structure is updated and the information of the object node O and the relationship node R is updated;

[0033] The update frequency and erase frequency of different nodes in the graph structure are recorded. If the frequency of the position relationship word exceeds the preset erase frequency β, it is determined to be an error erase, and an error erase feedback is performed to reduce the number of error erasures;

[0034] The attention score used to represent the node access intensity of each node is calculated to control the frequency of updating each node.

[0035] Further, the attention score is calculated according to the following formula:

[0036]

[0037]

[0038] wherein x t is a feature of a node, is an output information of an Attention LSTM layer, W xc and W hc are hyperparameters for calculating attention of a graph node.

[0039] The application also provides a computer device comprising a processor and a memory for storing a computer program, wherein the processor is configured to execute the computer program to perform the image description method for fusing object position relations.

[0040] Compared with the prior art, the application has the following advantages:

[0041] 1. The application forms three types of nodes, i.e., feature node A, object node O and relation node R, by extracting candidate box feature information and global feature information of an image respectively, and then constructs an object relation scene graph, encodes the node information in the object relation scene graph initially, and if the proportion of the position relations of the object nodes in the object relation scene graph is imbalanced, encodes the object relation nodes R in the object relation scene graph again according to the imbalance degree, so that the position relations and detail information between objects can be fully and effectively mined by combining the object relation scene graph construction and the twice encoding mechanism of the graph structure, the spatial position relations and detail information between objects are fused to improve the precision of image description, and the problem of incomplete and unreasonable description of the position relations of objects caused by the proportion imbalance can be solved, so that accurate and reasonable image description can be generated.

[0042] 2. The application further calculates the position relations between the adjacent object nodes in the object relation scene graph by fusing the knowledge base dictionary and common sense reasoning, so that the ability of the model to capture the implicit position relations of objects can be improved, and the precision and rationality of image description can be further improved.

[0043] 3. The application further realizes decoding by combining GCN and LSTM structures, judges the generated predicted words by using a bias gating mechanism, updates the double-layer LSTM by using the bias gating mechanism if the generated predicted words belong to the relation word fault-tolerant dictionary, and adopts an adjustable gating mechanism, so that the generalization ability of the model can be further improved.

[0044] 4. The application further constructs a relation word fault-tolerant dictionary in combination with the part-of-speech bias characteristics of the data set, sets the object relation erasing and fault-tolerant mechanism, so that the rationality and diversity of the text description can be further improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 This is a flowchart of the implementation of the image description method for integrating the position relationship of objects in this embodiment.

[0046] Figure 2 This is a flowchart of the detailed steps for implementing image description by fusing the position relationship of objects in this embodiment.

[0047] Figure 3 Schematic diagram of the encoder structure used in a specific application embodiment of the present invention.

[0048] Figure 4 2 is a schematic diagram of the decoder structure used in a specific application embodiment of the present invention.

[0049] Figure 5 It is a schematic diagram of the knowledge base dictionary format used in a specific application embodiment of the present invention.

[0050] Figure 6 1 is a schematic diagram comparing the first set of experimental results of using the method of the present invention and the traditional method for text prediction in a specific application embodiment.

[0051] Figure 7 3 is a schematic diagram comparing the second set of experimental results of using the method of the present invention and the traditional method for text prediction in a specific application embodiment. DETAILED DESCRIPTION

[0052] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0053] like Figure 1 、 2 As shown, the image description method of integrating the position relationship of objects in this embodiment specifically includes the following steps:

[0054] Step S1: Input an image to be described, and extract the global feature information and candidate frame feature information of the image to be described.

[0055] Step S11: Global feature extraction

[0056] In this embodiment, the global feature information of the image is extracted by using the ResNet101 deep residual network pre-trained on the ImageNet dataset, where ResNet101 has two basic modules: Conv Block (convolution module) and Identity Block (identity residual module). The ResNet101 input feature dimension is (1024,1024,3), and after Zero Padding (zero padding), the Conv2d (two-dimensional convolution) feature dimension is (512,512,64). After BatchNorm, ReLU layer and MaxPool (pooling) layer, and after 4 Conv Block modules and 8 IdentityBlock modules, the global feature of the image is output as (32,32,2048).

[0057] Step S12: Extracting candidate box features

[0058] In this embodiment, the Faster RCNN detection model is used to extract candidate box feature information, object feature information, and corresponding object features in the bounding box of the image, including the location information of each candidate box, such as the candidate box height h, width w, area h×w, and center point coordinates c. The feature extraction network for candidate box feature extraction specifically includes 13 Conv layers, 13 ReLU layers, and 4 Pooling layers. The extracted features are input into the RPN region candidate network to obtain object coordinate values ​​and predict the object's position.

[0059] Step S2. Construct an object relationship scene graph G based on the extracted global feature information and candidate box feature information. The object relationship scene graph G includes feature nodes A, object nodes O, and relationship nodes R. Feature node A corresponds to the feature information of the candidate box feature information corresponding to the candidate box, object node O corresponds to the object feature information corresponding to the candidate box feature information, and relationship node R corresponds to the global feature information and the feature information and object information corresponding to the candidate box in the candidate box feature information.

[0060] In this embodiment, three types of node information in the object relationship scene graph G are extracted specifically in the following manner:

[0061] The feature information corresponding to the candidate box extracted by Faster RCNN is used to construct a feature node A, that is, the information contained in the feature node A is the feature information corresponding to the candidate box;

[0062] The object feature information corresponding to the candidate frame extracted by Faster RCNN is used to construct the object node O, that is, the information contained in the object node O is the object feature information corresponding to the candidate frame;

[0063] The feature information and object information of the candidate box extracted by the Faster RCNN and the global feature information of the image extracted by the ResNet101 constitute a relationship node R, that is, the information contained in the relationship node R includes the feature information and object information of the candidate box and the global feature information.

[0064] Step S3: primary encoding is performed on the node information of the object relationship scene graph G to obtain an encoded feature matrix of each type of node; if there is a proportion imbalance phenomenon between the position relationship proportions of the object nodes in the object relationship scene graph G and the original position relationship of the object, then performing secondary encoding on the object relationship encoded feature matrix obtained by the primary encoding of the relationship node R according to the proportion imbalance degree to obtain an object relationship encoded feature matrix after secondary encoding.

[0065] In this embodiment, the primary encoding is performed on the node information of the object relationship scene graph G according to the following formula:

[0066]

[0067] wherein x i is the encoded feature matrix of the node in the object relationship scene graph G, v i is a feature information matrix, is an embedding encoding matrix corresponding to each node category, w[0] represents an embedding encoding matrix corresponding to the object node O, w[1] represents an embedding encoding matrix corresponding to the feature node A, w[2] represents an embedding encoding matrix corresponding to the relationship node R, and d represents the dimension of the input feature vector.

[0068] In this embodiment, the specific steps of the proportion imbalance judgment and the secondary encoding include:

[0069] Step S31: a knowledge base (Knoeledge Base) dictionary is constructed by selecting a plurality of object nouns with the highest frequency and a plurality of position relationship words with the highest frequency from the image description data set, and the knowledge base dictionary is encoded into the object relationship scene graph G.

[0070] In order to mine the position relationship between objects in the image, this embodiment constructs a knowledge base dictionary according to the part of speech and word frequency distribution characteristics of the image description data set, and combines the knowledge base dictionary in the encoder. Specifically, a plurality (for example, 50) of object nouns with the highest frequency and a plurality (for example, 20) of most critical position relationship words are selected to construct the knowledge base dictionary. The knowledge base dictionary is encoded into the node category embedding encoding matrix w α [q] in the encoder by using the one-hot encoding method. As shown in Figure 5 the knowledge base (Knowledge Base) dictionary is encoded into the node category embedding encoding matrix wα [q] in.

[0071] Step S32. According to the knowledge base dictionary and the bounding box position information of the object node O, the position relationship between adjacent object nodes in the object relationship scene graph G is calculated. The knowledge base dictionary stores a plurality of object nouns with the highest word frequency and a plurality of position relationship words.

[0072] In this embodiment, by configuring an object position relationship reasoning module, the position relationship between two adjacent object nodes in the object relationship scene graph is calculated by using the bounding box position information (such as height h, width w, area h*w, center point coordinate c and relative position θ) in the candidate box feature information of each object node, and the position relationship between adjacent object nodes in the object relationship scene graph G is obtained.

[0073] Step S33. According to the calculated position relationship between adjacent object nodes, it is judged whether there is a proportion imbalance phenomenon. If there is, go to step S34, otherwise exit the current calculation.

[0074] Suppose that the original position relationship proportion of 50 objects and the position relationship words corresponding to the position relationship in the proportion common sense priori knowledge are taken as an example. If the original position relationship proportion of two objects is 1:5, but the proportion calculated by the position information of the bounding box is 1:0.05, 1:0.5, 1:50 and 1:500, then it is considered that there is a proportion imbalance phenomenon. According to the proportion imbalance degree value, the imbalance degree q can be divided into multiple (for example, 5) levels.

[0075] In this embodiment, by comparing the calculated position relationship between adjacent object nodes with the original position relationship proportion in the proportion common sense priori knowledge, it is judged whether there is a proportion imbalance phenomenon. The proportion common sense priori knowledge stores the original position relationship proportion of multiple objects and the corresponding position relationship words. Specifically, if the deviation value between the calculated position relationship between adjacent object nodes and the original position relationship proportion is greater than a preset value, that is, the position relationship between adjacent object nodes calculated by the bounding box is much greater than or much smaller than the original position relationship proportion, it is considered that there is a proportion imbalance phenomenon, otherwise it is considered that there is no proportion imbalance phenomenon.

[0076] Specifically, a proportion common sense priori knowledge base is configured, and the proportion common sense is added to the knowledge base (Knowledge Base) dictionary to form a common sense reasoning module. Based on the common sense reasoning module, the proportion imbalance phenomenon of the object position relationship is judged. If the proportion imbalance phenomenon occurs, go to step S34 to perform secondary coding on the primary coding feature matrix of the relationship node R according to the proportion imbalance degree, realize a more detailed definition and classification method, and then calculate a more reasonable position relationship type, and increase the object relationship type.

[0077] Step S34. The object relationship coding feature matrix obtained by primary coding the relationship node R according to the proportion imbalance degree is secondarily coded to obtain a second coding object relationship coding feature matrix.

[0078] In this embodiment, the object relationship coding feature matrix obtained by primary coding the relationship node R is secondarily coded according to the following formula:

[0079] z i =x i ⊙w α [q](2)

[0080] wherein z i is the second coding object relationship coding feature matrix, x i is the object relationship coding feature matrix obtained by primary coding, w is the embedding coding matrix corresponding to the proportion imbalance degree category, q represents the proportion imbalance category, the proportion imbalance category is obtained by dividing the proportion imbalance degree value, and different proportion imbalance categories correspond to different imbalance degree value ranges.

[0081] The embodiment can generate more abundant object position relationships, increase rel-node relationship nodes, refine rel-class relationship node categories, and obtain the coding features of the object relationship scene graph G structure through GCN structure calculation by fusing the common sense reasoning module to optimize the GCN graph reasoning algorithm of the object position relationship graph.

[0082] As Figure 3As shown, the embodiment first extracts global feature information and candidate box feature information of the image to be described, constructs an object relationship scene graph G according to the extracted global feature information and candidate box feature information, and the object relationship scene graph G contains three types of nodes, namely feature nodes A, object nodes O and relationship nodes R. Among them, the feature node A corresponds to the feature information of the corresponding candidate box in the candidate box feature information, the object node O corresponds to the object feature information in the candidate box feature information, and the relationship node R corresponds to the global feature information and the feature information of the corresponding candidate box in the candidate box feature information and object information. Then, a knowledge base dictionary is constructed and combined with the knowledge base dictionary in the encoder, an object position relationship reasoning module is constructed according to the characteristics of the knowledge base dictionary, the node information in the object relationship scene graph G is input into the position relationship encoder by using the object position relationship reasoning module, the node information in the object relationship scene graph G is encoded for the first time, and the encoding feature matrix of each type of node is obtained; further, the imbalance degree value between objects is calculated by combining the common sense dictionary and the reasoning module, whether there is an imbalance phenomenon in the position relationship between the object nodes in the object relationship scene graph G compared with the original position relationship of the object is judged, if there is an imbalance, the object relationship encoding feature matrix obtained by the first encoding of the relationship node R is secondarily encoded according to the imbalance degree, the position relationship and the detail information between the objects are effectively mined through the two encoding mechanisms, and then a complete, reasonable and accurate image description containing the object position relationship is generated.

[0083] Step S4: input the encoded object relationship scene graph G and node information into the joint decoder for joint decoding, and predict the text information of the image to be described.

[0084] The embodiment adopts a double-layer LSTM structure to decode the GCN graph structure and node information of the encoded object relationship scene graph G, as shown in Figure 4 After decoding, the corresponding predicted text is output, the generated predicted word is judged by using the bias gate mechanism, if the generated predicted word belongs to the pre-constructed relationship word fault-tolerant dictionary, the double-layer LSTM is updated by using the bias gate mechanism, the node information in the object relationship scene graph is optimized, and finally the description text describing the content of the image is generated.

[0085] The specific calculation formula of the double-layer LSTM structure for decoding the encoding feature in the embodiment is as follows:

[0086]

[0087]

[0088]

[0089] For the information bias problem of the data set, the embodiment introduces a bias gating mechanism into the GCN graph node information updating and replacement process. The bias gating mechanism includes a relationship fault-tolerant dictionary. The relationship fault-tolerant dictionary specifically protects a plurality of (for example, 20) fixed collocation phrases related to object position relationships, such as “in front of”, “covered in”, and “far away” and the like. The generated prediction words are judged by the bias gating mechanism. If the word belongs to the relationship fault-tolerant dictionary, the bias gating mechanism is used to update the double-layer LSTM as follows:

[0090]

[0091]

[0092] wherein x t is the bias gating mechanism control candidate information retention degree, w t-1 is the input information of the Attention LSTM layer at the current time t, is the output information of the Attention LSTM layer at the current time t, and σ(·) is a Sigmoid activation function, ensuring that z t value is 0-1.

[0093] The embodiment further includes an object relationship scene graph node updating and erasing step. An erasing and updating module is set to update the node information in the object relationship scene graph G. After generating a word describing the object position relationship, it is judged whether the current object relationship scene graph G structure needs to be updated and erased. The useless words are specifically included:

[0094] (1) Each time the decoded prediction word is judged. If it is judged that it is not an object position relationship word, the current object relationship scene graph G structure is not changed. If it is judged that it is an object position relationship word, the current object relationship scene graph G structure is updated and the information of the object node O and the relationship node R is updated;

[0095] (2) The update times and erasing frequencies of different nodes in the graph structure are recorded. If the frequency of the position relationship word exceeds the preset erasing frequency β, it is judged that it is an error erasing, and an error erasing feedback is performed to reduce the number of error erasing;

[0096] (3) The attention score of each node used to represent the node access strength is calculated to control the update frequency of each node.

[0097] The embodiment further includes that some syntax words unrelated to the object position relationship are not updated before updating the graph structure each time, such as articles “the” and “of”.​

[0098] The embodiment combines the graph convolution GCN and the LSTM structure to form an image description generation model that fuses the object position relationship in the scene, can effectively mine the spatial relationship and spatial detail information between objects, and generate more accurate and reasonable image descriptions. Meanwhile, by increasing the object relationship update and erasure mechanism, the rationality and diversity of the text description can also be improved.

[0099] The embodiment sets an attention score to represent the access intensity of each node, and the greater the intensity of each node, the easier it is to be updated. The frequency of updating of each node can be controlled so that the node with greater access intensity has a higher update frequency. The attention score is calculated according to the following formula:

[0100]

[0101]

[0102] wherein x t is the feature of the node, is the output information of the Attention LSTM layer, W xc and W hc are hyperparameters for calculating the attention of the graph node.

[0103] The embodiment further increases a fault-tolerant mechanism in the decoding process to improve the diversity of the generated description image prediction text. In the fault-tolerant mechanism, a relationship word fault-tolerant dictionary is set in combination with the part-of-speech bias characteristics of the data set. If the word to be erased is in the fault-tolerant dictionary, it cannot be erased. If the fault-tolerant dictionary contains some fixed collocation phrases related to the object position relationship, such as “in front of”, “covered in”, and “far away”, a total of 20 fixed collocation words. The object relationship scene graph G structure is updated by the erasure update module.

[0104] In a specific application example, the object relationship scene graph node is updated by the erasure update module, the update times and erasure frequency β of different nodes in the graph structure are recorded, and the word describing the object position relationship is generated. Whether the current object relationship scene graph structure needs to be updated and the useless word is erased is judged. If the word belongs to a simple object position relationship word, such as “on”, “at” and “in”; or a irrelevant grammar word, such as the article “the” and “a”, the word needs to be erased. The erasure rate β is set in the erasure module, the greater the β value, the more frequent the erasure times. If the generated object relationship word exceeds a certain erasure rate, it is considered that the word is erroneously erased and the number of erroneous erasure needs to be reduced. The node attention score α tThe access intensity of the node is used to control the update of the object nodes and the relationship nodes. If the erasure rate of a node is too high, the access intensity of the node is higher, and the attention score is improved, and the node is more likely to be updated.

[0105] The embodiment combines the update erasure and bias gating mechanism in the decoding process to further optimize the node features in the object relationship scene graph, and predict more accurate image descriptions with the position relationship between objects.

[0106] As shown in Figure 2 To implement the above method, an image feature information extraction module is arranged to extract the candidate box feature information and the global feature information of the image to be described, and construct an object position relationship scene graph G according to the extracted candidate box feature information and the global feature information. The constructed position relationship scene graph G includes the node feature information A corresponding to the candidate box, the object node feature information O corresponding to the candidate box, the feature information and object information corresponding to the candidate box, and the relationship node R including the global feature information of the image. The encoding module encodes the above node information. The encoding feature matrix of each type of node obtained by the primary encoding is combined with the common sense dictionary and the common sense reasoning module to judge the position relationship proportion between the object nodes in the object relationship scene graph G obtained by the primary encoding. When the proportion is unbalanced, the object relationship encoding feature matrix obtained by the primary encoding of the relationship node R is secondary encoded according to the proportion imbalance degree, and the secondary encoded object relationship encoding feature matrix is obtained, and the encoding is completed. The object relationship encoding feature matrix after encoding is input into the decoder, the decoder adopts a double-layer LSTM structure to decode the encoding feature, combines the update erasure module and the bias gating mechanism, further optimizes the node features in the object position relationship scene graph G to realize secondary optimization, and then predicts more accurate and rich image description prediction texts with the position relationship between objects.

[0107] In order to verify the effectiveness of the present application, two image description data sets containing images, VisualGenome and MSCOCO data sets, are used in the experiment to pre-train and verify the image description generation model. In the Visual Genome data set, 96,738 images and 3,397,459 annotated sentences in the data set are used as the training set; 4,941 images and 171,759 annotated sentences in the data set are used as the verification set and the test set. In the MSCOCO data set, 112,742 images and 475,117 annotated sentences are used as the training set; 4,970 images and 20,851 annotated sentences are used as the verification set and the test set. The above-mentioned method of the present application is trained and tested by using the VisualGenome and MSCOCO data sets. When training the image description generation model, the experiment first uses the Visual Genome and MSCOCO data sets for training, and the Visual Genome is trained for 100 epochs and the MSCOCO is trained for 80 epochs. Then, the respective test sets are used for testing, and the Visual Genome is tested for 50 epochs and the MSCOCO is tested for 40 epochs. Among them, the batch-size is set to 8, the optimizer is selected as SGD, the initial learning rate is 0.001, the momentum is set to 0.9, and BLEU-4, METEOR, ROUGE, CIDEr and SPICE are used as the evaluation indexes. In the specific embodiment, two groups of text description results generated by using the method of the present application and the traditional description method respectively for the same image to be described are as shown in Figure 6 , Figure 7 From the figure, it can be seen that the predicted text generated by the present application can be more reasonable, accurate and contain more position information.

[0108] The present application also provides a computer device comprising a processor and a memory, the memory being used to store a computer program, and the processor being used to execute the computer program to perform the image description method fusing object position relationship.

[0109] It can be understood that the above method of the embodiment can be executed by a single device, such as a computer or a server, and can also be applied to a distributed scenario and completed by multiple devices in cooperation. In the distributed scenario, one of the multiple devices can only execute one or more steps in the above method of the embodiment, and the multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, and is used to execute a related program to implement the above method of the embodiment. The memory can be implemented in the form of a read-only memory (ROM), a random access memory (RAM), a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above method of the embodiment is implemented by software or firmware, the related program code is stored in the memory and executed by the processor.

[0110] The above is only a preferred embodiment of the present application, and does not limit the present application in any form. Although the present application has been disclosed as above with the preferred embodiment, it is not intended to limit the present application. Therefore, any simple modification, equivalent change and modification made to the above embodiment according to the technical essence of the present application without departing from the technical solution of the present application shall fall within the scope of protection of the technical solution of the present application.

Claims

1. An image description method integrating object position relationships, characterized in that: The following steps are involved: Step S1. Input the image to be described and extract the global feature information and candidate frame feature information of the image to be described; Step S2. Construct an object relationship scene graph G based on the extracted global feature information and the candidate box feature information, wherein the object relationship scene graph G includes feature nodes A, object nodes O, and relationship nodes R; The feature node A correspondingly contains the feature information of the candidate box in the candidate box feature information, the object node O correspondingly contains the feature information of the object in the candidate box feature information, and the relationship node R correspondingly contains the global feature information and the feature information and object information of the candidate box in the candidate box feature information; Step S3. Initially encode the information of each type of node in the object relationship scene graph G to obtain the encoding feature matrix of each type of node; Calculate the positional relationship ratio between adjacent object nodes in the object relationship scene graph G, compare the calculated positional relationship ratio between adjacent object nodes with the original positional relationship ratio in the prior knowledge of proportion common sense, and determine whether there is a proportion imbalance phenomenon. If there is a proportion imbalance phenomenon compared with the original positional relationship of the object in the object relationship scene graph G, perform secondary encoding on the object relationship encoding feature matrix obtained by the initial encoding of the relationship node R according to the degree of proportion imbalance to obtain the object relationship encoding feature matrix after secondary encoding, wherein the object relationship encoding feature matrix obtained by the initial encoding of the relationship node R is secondary encoded according to the following formula: ,in, is the object relationship encoding feature matrix after secondary encoding, is the object relationship encoding feature matrix obtained after the initial encoding, is the embedding coding matrix corresponding to the imbalance degree category, Indicates a category of proportion imbalance, wherein the category of proportion imbalance is obtained by dividing the category according to the degree of proportion imbalance; Step S4: Input the encoded object relationship scene graph G and node information into a joint decoder for joint decoding to predict the text information of the image to be described.

2. The image description method for integrating object position relationships according to claim 1, characterized in that: In step S3, the node information of each type in the object relationship scene graph G is initially encoded according to the following formula: in, is the encoding feature matrix of the nodes in the object relationship scene graph G, is the feature information matrix, is the embedding coding matrix corresponding to each node category, represents the embedding coding matrix corresponding to the object node O, represents the embedding coding matrix corresponding to feature node A, Represents the embedding coding matrix corresponding to the relationship node R, Represents the feature vector dimension of the input.

3. The image description method for integrating object position relationships according to claim 1, characterized in that: The step of performing secondary encoding on the object relationship encoding feature matrix obtained by primary encoding of the relationship node R according to the degree of imbalance in the proportion in step S3 includes: Step S31. Selecting the most frequent object nouns and position-relation words from the image description dataset to construct a knowledge base dictionary, and encoding the knowledge base dictionary into the object relationship scene graph G; Step S32. Calculate the positional relationship between adjacent object nodes in the object relationship scene graph G based on the knowledge base dictionary and the bounding box position information of the object node O, wherein the knowledge base dictionary stores a plurality of object nouns with the highest word frequency and a plurality of position relationship words; Step S33. Based on the calculated positional relationship between adjacent object nodes, determine whether there is a proportional imbalance. If so, proceed to step S34; otherwise, exit the current calculation. Step S34: Perform secondary encoding on the object relationship encoding feature matrix obtained by the initial encoding of the relationship node R according to the degree of proportion imbalance to obtain the secondary encoded object relationship encoding feature matrix.

4. The image description method for integrating object position relationships according to claim 3, characterized in that: In step S31, the knowledge base dictionary is encoded into the node category embedding coding matrix using the one-hot encoding method; in step S33, the calculated positional relationship between adjacent object nodes is compared with the original positional relationship ratio in the proportional common sense prior knowledge to determine whether there is a proportion imbalance phenomenon. The proportional common sense prior knowledge stores the original positional relationship ratios of multiple objects and the corresponding positional relationship words.

5. The image description method for integrating object position relationships according to any one of claims 1 to 4, characterized in that: In step S4, a two-layer LSTM structure is used to decode the GCN graph structure and node information of the encoded object relationship scene graph G, and a bias gating mechanism is used to judge the generated prediction words. If the generated prediction words belong to the pre-built relationship word fault-tolerant dictionary, the bias gating mechanism is used to update the two-layer LSTM.

6. The image description method for integrating object position relationships according to claim 5, characterized in that: Use bias gating mechanism to control the Update as shown in the following formula: in, It is a bias gating mechanism that controls the degree of retention of candidate information. 、 is the input information of the Attention LSTM layer at the current time t, is the current moment of the Attention LSTM layer Output information, For the Sigmoid activation function, ensure The value is 0~1.

7. The image description method for fusing object position relationships according to any one of claims 1 to 4, characterized in that: Step S4 also includes the step of updating and erasing the object relationship scene graph nodes, including: Each time the decoded predicted word is judged, if it is judged to be a word that does not belong to the object position relationship, the current object relationship scene graph G structure is not changed; if it is judged to be a word that belongs to the object position relationship, the current object relationship scene graph G structure is updated and the information of the object node O and the relationship node R is updated; Record the update times and erasure frequencies of different nodes in the graph structure. If the frequency of position-related words exceeds the preset erasure frequency, , it is determined to be an erroneous erasure, and erroneous erasure feedback is performed to reduce the number of erroneous erasures; An attention score is calculated for each node to represent the node visit intensity and to control the frequency of node updates.

8. The image description method for integrating object position relationships according to claim 7, characterized in that: The attention score is calculated according to the following formula: in, is the feature of the node, is the output information of the Attention LSTM layer, 、 and is the hyperparameter for computing graph node attention.

9. A computer device comprising a processor and a memory, wherein the memory is used to store a computer program, wherein: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

Images