Image description generation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image description generation method and device, electronic equipment and a storage medium, belongs to the technical field of artificial intelligence, and is suitable for the field of financial science and technology and the field of digital medical treatment. The method comprises the following steps: detecting an object position, an object category and an object visual feature of a target object in a sample image; identifying description keywords described by the original image and semantic features of the description keywords; according to the object position, the object category, the object visual feature, the description keyword and the semantic feature, graph structure data is generated, the graph structure data comprises graph nodes and connecting edges, and the connecting edges have edge weights; constructing an adjacent matrix according to the edge weight; performing feature extraction on the graph nodes and the adjacent matrix through a graph neural network to obtain target fusion features; updating the graph neural network according to the original image description, the adjacent matrix and the target fusion feature; the target image is subjected to image description generation through the updated image description generation model, and the accuracy of image description generation can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and is applied to the fields of financial technology and digital medicine, and particularly relates to an image description generation method and device, an electronic device and a storage medium. BACKGROUND

[0002] Image description generation is a process of converting an image into a text description, which can automatically generate a natural language description according to a given image, and is helpful for quickly understanding the main content of the image. For example, in the field of financial technology, an insurance claim image can be converted into a text description to generate a preliminary insurance claim report, thereby improving the efficiency of claim settlement. For another example, in the field of digital medicine, a text description of a medical image can be automatically generated to assist medical diagnosis.

[0003] In the process of image description generation, a traditional method models the relationship between an image and a text by cross-modal alignment between image features and text features through feature splicing or cross-attention. However, simple feature splicing cannot capture the complex relationship between image features and text features, and the cross-attention mechanism only focuses on a few regions of the image, which ignores other important information, resulting in low accuracy of image description generation. SUMMARY

[0004] The main purpose of the embodiments of the present application is to propose an image description generation method, device, electronic device and storage medium, which aims to improve the accuracy of image description generation.

[0005] To achieve the above purpose, a first aspect of the embodiments of the present application proposes an image description generation method, which comprises:

[0006] Obtaining a sample image, performing target detection on the sample image to obtain object positions, object categories and object visual features of target objects;

[0007] Obtaining an original image description of the sample image, performing keyword recognition on the original image description to obtain description keywords and semantic features of the description keywords;

[0008] Generating graph structure data according to the object positions, the object categories, the object visual features, the description keywords and the semantic features; wherein the graph structure data comprises graph nodes and connecting edges between any two graph nodes, and the connecting edges have edge weights;

[0009] Constructing an adjacency matrix according to the edge weights;

[0010] Extracting features of the graph nodes and the adjacency matrix through a preset graph neural network to obtain target fusion features;

[0011] update network parameters of the graph neural network according to the original image description, the adjacency matrix and the target fusion feature, to obtain an image description generation model;

[0012] generate an image description for a preset target image through the image description generation model.

[0013] In some embodiments, the generating the graph structure data according to the object position, the object category, the object visual feature, the description keyword and the semantic feature comprises:

[0014] constructing an image target node according to the object position, the object category and the object visual feature;

[0015] constructing a text keyword node according to the description keyword and the semantic feature;

[0016] determining the graph node according to the image target node and the text keyword node, and calculating an initial weight between any two graph nodes;

[0017] generating the connection edge between any two graph nodes according to the initial weight, and determining the edge weight of the connection edge according to the initial weight;

[0018] generating the graph structure data according to the graph node, the connection edge and the edge weight.

[0019] In some embodiments, the calculating the initial weight between any two graph nodes comprises:

[0020] calculating a matching probability between any two graph nodes through an image encoder;

[0021] calculating a node similarity between any two graph nodes;

[0022] weighting and fusing the matching probability and the node similarity to obtain the initial weight.

[0023] In some embodiments, the graph neural network comprises a graph convolution network and a graph attention network, and the extracting the feature of the graph node and the adjacency matrix through the preset graph neural network to obtain the target fusion feature comprises:

[0024] extracting a first feature of the graph node and the adjacency matrix through the graph convolution network to obtain an initial node feature;

[0025] obtaining a neighbor node of the graph node according to the adjacency matrix;

[0026] calculating an attention weight between the graph node and the neighbor node;

[0027] performing second feature extraction on the attention weight and the initial node feature through the graph attention network to obtain the target fusion feature.

[0028] In some embodiments, the performing second feature extraction on the attention weight and the initial node feature through the graph attention network to obtain the target fusion feature comprises:

[0029] performing feature screening on the initial node feature to obtain a neighborhood node feature of the neighbor node;

[0030] performing feature aggregation on the attention weight and the neighborhood node feature through the graph attention network to obtain a target node feature;

[0031] updating the initial node feature according to the target node feature to obtain the target fusion feature.

[0032] In some embodiments, the updating network parameters of the graph neural network according to the original image description, the adjacency matrix and the target fusion feature to obtain an image description generation model comprises:

[0033] generating a reference image description of the sample image according to the target fusion feature;

[0034] performing loss calculation according to the original image description and the reference image description to obtain a first loss;

[0035] performing loss calculation according to the adjacency matrix and the target fusion feature to obtain a second loss;

[0036] performing loss weighting on the first loss and the second loss to obtain a target loss;

[0037] updating network parameters of the graph neural network according to the target loss to obtain the image description generation model.

[0038] In some embodiments, the performing loss calculation according to the adjacency matrix and the target fusion feature to obtain a second loss comprises:

[0039] calculating a degree matrix of the adjacency matrix;

[0040] determining a Laplacian matrix according to the adjacency matrix and the degree matrix, and performing loss calculation according to the Laplacian matrix and the target fusion feature to obtain a sub-loss;

[0041] calculating feature similarity between the target fusion features according to the adjacency matrix;

[0042] According to the sub-loss and the feature similarity, the second loss is determined.

[0043] To achieve the above object, a second aspect of the embodiment of the present application provides an image description generation device, the device comprising:

[0044] a detection module configured to acquire a sample image, perform target detection on the sample image, and obtain object positions, object categories, and object visual features of target objects;

[0045] an identification module configured to acquire an original image description of the sample image, perform keyword identification on the original image description, and obtain description keywords and semantic features of the description keywords;

[0046] a graph structure generation module configured to generate graph structure data according to the object positions, the object categories, the object visual features, the description keywords, and the semantic features; wherein the graph structure data comprises graph nodes and connecting edges between any two of the graph nodes, and the connecting edges have edge weights;

[0047] a matrix construction module configured to construct an adjacency matrix according to the edge weights;

[0048] a feature extraction module configured to perform feature extraction on the graph nodes and the adjacency matrix by using a preset graph neural network, and obtain target fusion features;

[0049] an update module configured to update network parameters of the graph neural network according to the original image description, the adjacency matrix, and the target fusion features, and obtain an image description generation model;

[0050] a description generation module configured to perform image description generation on a preset target image by using the image description generation model.

[0051] To achieve the above object, a third aspect of the embodiment of the present application provides an electronic device, the electronic device comprising a memory and a processor, the memory storing a computer program, and the processor implementing the method of the first aspect when executing the computer program.

[0052] To achieve the above object, a fourth aspect of the embodiment of the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, and the computer program implementing the method of the first aspect when executed by a processor.

[0053] The image description generation method, the image description generation device, the electronic equipment and the computer readable storage medium provided by the embodiments of the present application can improve the accuracy of image description generation. BRIEF DESCRIPTION OF DRAWINGS BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a flowchart of the image description generation method provided by the embodiments of the present application;

[0055] Figure 2 is Figure 1 is a flowchart of step S130 in

[0056] Figure 3 is Figure 2 is a flowchart of step S230 in

[0057] Figure 4 is Figure 1 is a flowchart of step S150 in

[0058] Figure 5 is Figure 4 is a flowchart of step S440 in

[0059] Figure 6 is Figure 1 is a flowchart of step S160 in

[0060] Figure 7 is Figure 6the flowchart of step S630 in FIG. 6;

[0061] Figure 8 is a structural schematic diagram of an image description generation apparatus provided by an embodiment of the present application;

[0062] Figure 9 is a hardware structural schematic diagram of an electronic device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application is further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0064] It should be noted that although the functional modules are divided in the apparatus schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in a manner different from the module division in the apparatus or the order in the flowchart. The terms "first", "second", etc. in the specification and claims and the above-described drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the present application belongs. The terms used herein are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0066] Image description generation is the process of converting an image into a text description, which can automatically generate a natural language description according to a given image, and is helpful for quickly understanding the main content of the image. For example, in the financial technology scenario, an insurance claim image can be converted into a text description to generate a preliminary insurance claim report, thereby improving the efficiency of claim settlement. For another example, in the digital medical scenario, a text description of a medical image can be automatically generated to assist medical diagnosis.

[0067] In the image description generation process, the conventional method models the relationship between the image and the text by cross-modal alignment between the image features and the text features through feature splicing or cross-attention. However, simple feature splicing cannot capture the complex relationship between the image features and the text features, and the cross-attention mechanism only focuses on a few regions of the image, which ignores other important information, resulting in low accuracy of image description generation.

[0068] Based on this, the embodiments of the present application provide an image description generation method, an image description generation apparatus, an electronic device and a computer readable storage medium, aiming at the accuracy of image description generation.

[0069] The image description generation method, the image description generation apparatus, the electronic device, and the computer readable storage medium provided by the embodiments of the present application are described in detail through the following embodiments. First, the image description generation method in the embodiments of the present application is described.

[0070] The image description generation method provided by the embodiments of the present application relates to the technical field of artificial intelligence. The image description generation method provided by the embodiments of the present application can be applied to a terminal, can be applied to a server end, and can also be software running in the terminal or the server end. In some embodiments, the terminal can be a smart phone, a tablet computer, a notebook computer, a desktop computer, or the like; the server end can be configured as a stand-alone physical server, can be configured as a server cluster or a distributed system formed by multiple physical servers, can also be configured as a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDNs, and big data and artificial intelligence platforms; and the software can be an application that implements the image description generation method, but is not limited to the above forms.

[0071] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld devices or portable devices, tablet devices, multi-processor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, and the like. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, and the like that perform specific tasks or implement specific abstract data types. The present application can also be practiced in a distributed computing environment in which tasks are performed by remote processing devices connected by a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media, including storage devices.

[0072] Figure 1 The method in the optional flowchart of the image description generation method provided by the embodiments of the present application can include but is not limited to steps S110 to S170. Figure 1

[0073] In step S110, a sample image is obtained, target detection is performed on the sample image, and object positions, object categories, and object visual features of target objects are obtained.

[0074] In step S120, an original image description of the sample image is obtained, keyword recognition is performed on the original image description, and description keywords and semantic features of the description keywords are obtained. ​

[0075] In step S130, graph structure data is generated according to the object position, the object category, the object visual feature, the description keyword, and the semantic feature; the graph structure data includes graph nodes and connecting edges between any two graph nodes, and the connecting edges have edge weights;

[0076] In step S140, an adjacency matrix is constructed according to the edge weights.

[0077] In step S150, feature extraction is performed on the graph nodes and the adjacency matrix by using a preset graph neural network to obtain target fusion features.

[0078] In step S160, network parameters of the graph neural network are updated according to the original image description, the adjacency matrix, and the target fusion features to obtain an image description generation model.

[0079] In step S170, image description generation is performed on a preset target image by using the image description generation model.

[0080] The steps S110 to S170 shown in the embodiments of the present application represent the complex semantic relationship between the image and the text through the graph structure data, and the graph neural network learns the structured information of the graph structure data, so that a more rich and accurate image description is generated.

[0081] In step S110 of some embodiments, a sample image is obtained from a database or collected by a shooting device such as a camera, the sample image is an image used for model training, and the sample image is a color image (RGB image) of any resolution, which can be in JPG, PNG, or the like. In the financial technology scenario, the sample image can be a vehicle image, and by generating a description of the vehicle damage, it is helpful to quickly handle vehicle insurance claims. In the digital medical scenario, the sample image can be a medical image, and by generating a detailed report for the medical image, it is possible to assist medical diagnosis and disease treatment based on the image report.

[0082] In order to understand the image content, the target object is detected in the sample image by using a target detection model to obtain the object position, the object category, and the object visual feature of the target object. The target detection model can be a YOLOv5 model, the target object is an element in the sample image, the object position is the position of the target object in the sample image, the object category is the category of the target object, and the object visual feature is a property feature that can be visually perceived and quantified, including color features, texture features, spatial position features, motion features, and depth features, etc. If the sample image is a vehicle image, the object category can be a vehicle, a road, and a traffic sign, and the object visual feature can be the color of the vehicle, the position of the vehicle parked on the road, and the damage condition, etc. If the sample image is a medical image, the object category can be a lesion type, and the object visual feature can be a lesion shape, a lesion size, and a lesion deterioration degree, etc.

[0083] The target detection model also outputs the confidence of the target object. In order to improve the accuracy of target detection, the repeated target objects are filtered out, and the object area of the remaining target objects is calculated. According to the confidence and the object area, the target objects are screened, and the target objects with a confidence greater than a preset confidence threshold and an object area greater than a preset area threshold are selected. The preset confidence threshold and the preset area threshold can be set according to actual conditions. The preset confidence threshold is in the range of [0, 1], such as 0.7. The preset area threshold is greater than 0 and less than the image area of the sample image. The preset area threshold can be set to 1% of the image area.

[0084] In step S120 of some embodiments, a text description matching the sample image is obtained, and an original image description is obtained. The original image description is a natural language sentence, such as “a red car is driving on the road”. In order to understand the text semantics of the original image description, the original image description is subjected to keyword recognition by a text analysis model, and description keywords and semantic features of the description keywords are obtained. The text analysis model can be a BERT model, the description keywords are text keywords of the original image description, such as “red” and “drive”, and the semantic features are embedding feature vectors of the description keywords, which can be used to represent the meaning and concept expressed by the description keywords.

[0085] In order to improve the efficiency of model training, the description keywords are screened to quickly locate the key information of the original image description, filter out unimportant keywords, and reduce information overload. Specifically, the parts of speech of the description keywords are obtained, which can be adjectives, nouns, verbs, adverbs, etc. The description keywords are subjected to a first screening according to the parts of speech. The parts of speech of the description keywords after the first screening can be nouns, verbs, and adjectives. The text analysis model also outputs attention weights of the description keywords, and the description keywords after the first screening are subjected to a second screening according to the attention weights. In the second screening process, the description keywords with attention weights greater than a preset weight threshold can be selected, or the description keywords after the first screening can be sorted in descending order of attention weights, and the first K description keywords can be selected. K is an integer greater than 0.

[0086] The traditional method implicitly models the association between the image and the text through feature splicing or attention mechanism. However, the feature distribution difference between the image feature and the text feature will make the association invalid, for example, the "vehicle" in the image and the "vehicle" in the text will lead to association failure due to the feature distribution difference. In order to solve the problem of semantic association loss, the embodiment of the application explicitly models the semantic association between the image and the text through the graph structure. The graph structure data includes graph nodes and connecting edges between any two graph nodes. The connecting edge has an edge weight, and the edge weight is used to represent the degree of semantic association between the two graph nodes connected by the connecting edge, and the edge weight is proportional to the degree of semantic association.

[0087] Please refer to Figure 2 In some embodiments, step S130 can include but is not limited to steps S210 to S250:

[0088] Step S210, constructing an image target node according to the object position, the object category and the object visual feature;

[0089] Step S220, constructing a text keyword node according to the description keyword and the semantic feature;

[0090] Step S230, determining a graph node according to the image target node and the text keyword node, and calculating an initial weight between any two graph nodes;

[0091] Step S240, generating a connecting edge between any two graph nodes according to the initial weight, and determining an edge weight of the connecting edge according to the initial weight;

[0092] Step S250, generating graph structure data according to the graph node, the connecting edge and the edge weight.

[0093] In step S210 of some embodiments, the node is constructed according to the object position, the object category and the object visual feature, and the image target node is obtained.

[0094] In step S220 of some embodiments, the node is constructed according to the description keyword and the semantic feature, and the text keyword node is obtained.

[0095] In step S230 of some embodiments, the image target node is taken as a graph node, and the text keyword node is taken as a graph node. In order to obtain the degree of association between any two graph nodes, the initial weight between any two graph nodes is calculated, and the value range of the initial weight is [0, 1]. It should be noted that any two graph nodes can be two image target nodes, two text keyword nodes or an image target node and a text keyword node.

[0096] In step S240 of some embodiments, if the initial weight is greater than the preset weight threshold, a connection edge connected between the two graph nodes corresponding to the initial weight is generated, and the initial weight is taken as the edge weight of the connection edge.

[0097] If there is a connection edge between the two graph nodes, it means that there is an association relationship between the two graph nodes. Since the two graph nodes can be two image target nodes, two text keyword nodes, or an image target node and a text keyword node, the association relationship between the image target nodes, the association relationship between the text keyword nodes, or the cross-modal association relationship between the image target node and the text keyword node can be established through the connection edge.

[0098] The edge type of the connection edge includes a semantic association edge, an attribute association edge, and an action association edge. The semantic association edge represents the semantic relationship between the two graph nodes, and is usually used to represent the relationship between categories or concepts, such as “car”→“vehicle”, which means that “car” is a kind of “vehicle”. The attribute association edge represents the attribute or feature of a graph node, such as “car”→“red”, which means that the color of the car is red. The action association edge represents the action or behavior of a node, such as “car”→“driving”, which means that the car is driving. The direction of the connection edge is determined by the semantic role labeling, such as the SPO triple “car (subject)→ driving (predicate)→ road (object)”.

[0099] There are often spatial, semantic, or functional relationships between target objects in sample images, such as spatial relationships “on”, semantic relationships “driving”, and functional relationships “driving” between “vehicle” and “road”. By establishing the association relationship between the image target nodes, the image content of the sample image can be more comprehensively understood. There are rich semantic relationships between keywords, such as dependency relationships, semantic roles, and synonym relationships. By establishing the association relationship between the text keyword nodes, the structure and semantics of the text can be better understood. Image and text are two different modalities of data. By dynamically constructing the association relationship between the image target nodes and the text keywords, the semantic alignment of the image and the text can be achieved, so that the semantic information between the image and the text can be complemented, the semantic association accuracy between the image and the text can be improved, and the accuracy of image description generation can be improved.

[0100] In step S250 of some embodiments, the graph nodes, connection edges, and edge weights are taken as graph structure data, and the graph structure data is a multi-modal graph structure for explicitly representing the semantic association relationship between image features, text features, and image features and text features.

[0101] To improve the efficiency of model training, graph pruning can be performed on the graph structure data to reduce redundant calculations. Specifically, for each graph node, the connection edges connected to the graph node are sorted in descending order of edge weight, and the first K connection edges and the other graph nodes connected to the connection edges are retained, and the other connection edges and the other graph nodes connected to the other connection edges are removed.

[0102] The target object in the sample image and the description keyword in the original image description are taken as graph nodes to construct a dynamic association graph to obtain graph structure data. The graph structure data can adaptively capture the complex semantic relationship between the image and the text. For example, the "tumor node" in the medical image is associated with "malignant" and "treatment plan" in the image description through a connection edge, and the "red car" in the vehicle image is associated with "red" and "vehicle" in the image description through a connection edge.

[0103] By explicitly modeling the multi-modal semantic association relationship through the above steps S210 to S250, the utilization rate of graph structure information can be improved, and the explainability and scalability of the association relationship are enhanced, avoiding the semantic association loss caused by implicit modeling, thereby improving the accuracy of image description generation.

[0104] Please refer to Figure 3 In some embodiments, step S230 can include but is not limited to steps S310 to S330:

[0105] In step S310, the matching probability between any two graph nodes is calculated by an image encoder;

[0106] In step S320, the node similarity between any two graph nodes is calculated;

[0107] In step S330, the matching probability and the node similarity are weighted and fused to obtain an initial weight.

[0108] In step S310 of some embodiments, any two graph nodes are input into an image encoder to calculate the matching probability between the graph nodes, and the matching probability is in the interval [0, 1]. The image encoder uses a CLIP-ViT / L-14 model.

[0109] In step S320 of some embodiments, the cosine similarity between any two graph nodes is calculated, and the cosine similarity is normalized to the interval [0, 1] to obtain the node similarity.

[0110] In step S330 of some embodiments, the matching probability and the node similarity are weighted and summed to obtain an initial weight. The calculation formula of the initial weight is represented as:

[0111] w i,j= 0.7 * CLIP_score(i, j) + 0.3 * cos(i, j),

[0112] wherein w i,j is the initial weight between the graph node i and the graph node j; CLIP_score is the matching probability; and cos is the node similarity.

[0113] The steps S310 to S330 above ensure the accuracy of the weight calculation by weighting and fusing the matching probability and the node similarity, thereby improving the accuracy of the connection edge construction.

[0114] In step S140 of some embodiments, the edge weight of the connection edge between the graph node i and the graph node j is taken as the element of the i-th row and the j-th column of the matrix to obtain an adjacency matrix. The adjacency matrix has a matrix size of N x N, where N is the number of graph nodes.

[0115] All the graph nodes and the adjacency matrix are input into a graph neural network for feature extraction to obtain target fusion features. The target fusion features are multi-modal fusion features extracted based on a multi-modal graph structure. The graph neural network performs bidirectional information propagation between the graph nodes and the connection edges. Taking the image target node and the text keyword node as an example, the image target node obtains the text semantics of the text keyword node through the connection edge to enhance the features, and the text keyword node obtains the image context information of the image target node through the connection edge. For example, the text keyword node “fast driving” can update the motion features of the image target node “car” through the connection edge, and the image target node “car” can feed back the car position information to the text keyword node through the connection edge to benefit the text understanding.

[0116] Please refer to Figure 4 In some embodiments, the graph neural network includes a graph convolution network and a graph attention network, and step S150 can include but is not limited to steps S410 to S440:

[0117] Step S410, performing first feature extraction on the graph nodes and the adjacency matrix through the graph convolution network to obtain initial node features;

[0118] Step S420, obtaining neighbor nodes of the graph nodes according to the adjacency matrix;

[0119] Step S430, calculating attention weights between the graph nodes and the neighbor nodes;

[0120] Step S440, performing second feature extraction on the attention weights and the initial node features through the graph attention network to obtain the target fusion features.

[0121] In step S410 of some embodiments, each graph node and the adjacency matrix are input into a graph convolution network for first feature extraction, and direct neighborhood information such as "car" -> "red" is aggregated by the graph convolution network to capture complex semantic relationships, to obtain an initial node feature of each graph node. Specifically, for each graph node, a linear transformation is performed on the graph node to obtain an intermediate feature representation. The edge weight of the graph node is obtained according to the adjacency matrix, and the graph node connected to the graph node is determined according to the edge weight to obtain a neighbor node. The connection edge between the neighbor node and the graph node has an edge weight greater than 0. The adjacency matrix is normalized. The intermediate feature representation of the neighbor node is aggregated according to the normalized adjacency matrix to obtain the initial node feature of the graph node. The calculation formula of aggregation is represented as:

[0122] h i =σ(∑ j∈N(i) A ij h′ j ),

[0123] where h i is the initial node feature of the graph node i; sigma is an activation function; N(i) is a set of neighbor nodes j of the graph node i; A is the normalized adjacency matrix; A ij is the element of the i-th row and the j-th column of the adjacency matrix, representing the normalized edge weight between the graph node i and the neighbor node j; h′ j is the intermediate feature representation of the neighbor node j.

[0124] In step S420 of some embodiments, for each graph node, the edge weight of the graph node is obtained according to the adjacency matrix, and the graph node connected to the graph node is determined according to the edge weight to obtain a neighbor node. The connection edge between the neighbor node and the graph node has an edge weight greater than 0.

[0125] In step S430 of some embodiments, a linear transformation is performed on the initial node feature to obtain a reference node representation. The attention coefficient is calculated according to the reference node representation of the graph node and the reference node representation of the neighbor node, and the attention coefficient is normalized to obtain the attention weight between the graph node and the neighbor node. The calculation formula of the attention coefficient is represented as:

[0126] e ij =LeakyReLU(a T [h″ i ||h″ j ]),

[0127] where e ij represents the attention coefficient between the graph node i and the neighbor node j; LeakyReLU is an activation function; a is a learnable attention vector; T represents a transpose operation; h″ i and h″ jrespectively represent the reference node representation of the graph node i and the reference node representation of the neighbor node j; || represents a splicing operation.

[0128] The calculation formula of the attention weight is represented as:

[0129]

[0130] wherein, α ij represents the attention weight between the graph node i and the neighbor node j; k represents the neighbor node; N(i) is a set composed of the neighbor nodes of the graph node i.

[0131] The attention weight and the edge weight of the connecting edge can be multiplied to fine-tune the edge weight and dynamically update the graph structure data. The calculation formula of the fine-tuning is represented as:

[0132]

[0133] wherein, t represents a time step; α ij represents the attention weight between the graph node i and the neighbor node j; and respectively represent the edge weight of the connecting edge between the graph node i and the neighbor node j before fine-tuning and the edge weight after fine-tuning.

[0134] In step S440 of some embodiments, the attention weight of each graph node and the initial node feature corresponding to the graph node are input into the graph attention network for second feature extraction, so as to capture the indirect correlation relationship such as “car”→“red”→“color”, and obtain the target fusion feature.

[0135] The number of networks of the graph convolution network and the graph attention network can be set according to actual conditions, for example, the number of networks of the graph convolution network is 3, and the number of networks of the graph attention network is 2. The graph convolution network and the graph attention network can be freely combined, for example, 3 graph convolution networks connected in sequence can be cascaded with 2 graph attention networks connected in sequence, or 1 graph convolution network and 1 graph attention network can be cascaded to obtain a network block, and a plurality of network blocks can be connected in sequence.

[0136] Through the above steps S410 to S440, through the architecture combined with the graph convolution network and the graph attention network, the graph node and the edge weight are dynamically adjusted to capture the complex semantic correlation between the graph nodes.

[0137] Please refer to Figure 5 In some embodiments, step S440 can include but is not limited to steps S510 to S530:

[0138] In step S510, the initial node feature is subjected to feature screening to obtain the neighborhood node feature of the neighbor node.

[0139] Step S520, performing feature aggregation on the attention weight and the neighborhood node feature by the graph attention network to obtain a target node feature;

[0140] Step S530, updating the initial node feature according to the target node feature to obtain a target fusion feature.

[0141] In step S510 of some embodiments, performing linear transformation on the initial node feature of each graph node to obtain a reference node representation of each graph node. From the reference node representations of all graph nodes, filtering out the initial node features of the neighbor nodes of the current graph node to obtain neighborhood node features.

[0142] In step S520 of some embodiments, performing feature aggregation on the attention weight and the neighborhood node feature of the current graph node by the graph attention network to obtain the target node feature of the current graph node. The calculation formula of the aggregation is represented as:

[0143] σ(∑ j∈N(i) α ij h″ j ),

[0144] wherein h″ j is the neighborhood node feature of the neighbor node j; σ is an activation function; N(i) is a set of neighbor nodes of the current graph node i; α ij is the attention weight between the current graph node i and the neighbor node j.

[0145] In step S530 of some embodiments, updating the initial node feature of each graph node to the target node feature of the corresponding graph node to obtain a target fusion feature of each graph node, and the target fusion feature is a global semantic representation.

[0146] The above steps S510 to S530 can capture the indirect association relationship between the graph nodes by the graph attention network, so as to generate the image description based on the indirect association relationship.

[0147] Referring to Figure 6 In some embodiments, step S160 can include but is not limited to steps S610 to S650:

[0148] Step S610, generating a reference image description of the sample image according to the target fusion feature;

[0149] Step S620, performing loss calculation according to the original image description and the reference image description to obtain a first loss;

[0150] Step S630, performing loss calculation according to the adjacency matrix and the target fusion feature to obtain a second loss;

[0151] In step S640, the first loss and the second loss are loss-weighted to obtain a target loss.

[0152] In step S650, the network parameters of the graph neural network are updated according to the target loss to obtain the image description generation model.

[0153] In step S610 of some embodiments, the target fusion feature of each graph node is input into a transformer decoder for feature decoding to obtain a reference image description of the sample image. When generating a text description, the target fusion feature obtained based on the graph structure data can constrain the appearance order of text keywords, such as "car" before "driving".

[0154] In step S620 of some embodiments, the original image description is feature-extracted by a feature extractor to obtain first text description features, and the reference image description is feature-extracted by the feature extractor to obtain second text description features. The similarity of the first text description features and the second text description features is calculated. A cross-entropy loss is calculated according to the original image description and the reference image description, and the similarity and the cross-entropy loss are added to obtain the first loss. The first loss is used to measure the difference between the original image description and the reference image description. The calculation formula of the cross-entropy loss is represented as:

[0155]

[0156] wherein, L CE is the cross-entropy loss; M is the number of words in the original image description; y i is the i-th word in the original image description; y <i is all words from the 1st word to the i-1th word in the reference image description that have been generated before the i-th word is generated; I represents the sample image; P(y i |y ＜i ,I) represents the probability of predicting the i-th word given the first i-1 words and the sample image.

[0157] In step S630 of some embodiments, the embodiments of the present application introduce a graph structure in the image description generation task, and the second loss is a graph structure regularization loss used to constrain the reference image description to be consistent with the graph structure. The second loss is a loss value calculated according to the adjacency matrix and the target fusion feature.

[0158] In step S640 of some embodiments, the first loss and the second loss can be summed to obtain the target loss. Alternatively, a first weight of the first loss and a second weight of the second loss can be determined, the first weight and the first loss are multiplied to obtain a first value, the second weight and the second loss are multiplied to obtain a second value, and the first value and the second value are added to obtain the target loss.

[0159] In step S650 of some embodiments, the target loss is minimized to optimize the graph neural network, the network parameters of the graph neural network are updated, and an image description generation model is obtained.

[0160] Through the above steps S610 to S650, the graph neural network can be optimized to obtain a model with better text description generation performance, thereby improving the accuracy of image description generation.

[0161] Please see Figure 7 In some embodiments, step S630 may include, but is not limited to, steps S710 to S740:

[0162] Step S710: Calculate the degree matrix of the adjacency matrix;

[0163] Step S720: Determine the Laplacian matrix based on the adjacency matrix and degree matrix, and calculate the loss based on the Laplacian matrix and the target fusion features to obtain the sub-loss;

[0164] Step S730: Calculate the feature similarity between target fused features based on the adjacency matrix;

[0165] Step S740: Determine the second loss based on the sub-loss and feature similarity.

[0166] In step S710 of some embodiments, the number of connecting edges to a graph node is determined based on the adjacency matrix, thus obtaining the degree of the graph node. The degree of the graph node is used as the diagonal element to construct a diagonal matrix, resulting in the degree matrix of the adjacency matrix. The diagonal elements represent the degrees of the corresponding graph nodes. For example, the element in the i-th row and i-th column of the diagonal matrix represents the degree of graph node i.

[0167] In step S720 of some embodiments, the degree matrix is ​​subtracted from the adjacency matrix to obtain the Laplacian matrix. The similarity between the target fusion features of any two graph nodes is calculated, and the similarity is weighted according to the Laplacian matrix to obtain the sub-loss. The formula for calculating the sub-loss is expressed as:

[0168] L GL =∑ i,j∈V L ij ||h i -h j || 2 ,

[0169] Among them, L GL The sub-loss is V; V is the set of graph nodes; h i and h j Represent the target fusion features of graph node i and graph node j, respectively; ||·|| 2 Indicates similarity; Lij This represents the element in the i-th row and j-th column of the Laplace matrix.

[0170] In step S730 of some embodiments, for each graph node, the neighboring nodes of the graph node are determined according to the adjacency matrix, the similarity between the target fusion feature of the graph node and the target fusion feature of the neighboring nodes is calculated, and the similarity between the graph node and each neighboring node is added together to obtain the feature similarity. The formula for calculating the feature similarity is expressed as:

[0171] L GS =∑ j∈N(i) ||h i -h j || 2 ,

[0172] Among them, L GS For feature similarity; N(i) is the set of neighboring nodes of graph node i; h i and h j Let i and j represent the target fusion features of their respective neighbors; ||·|| 2 Indicates similarity; L ij This represents the element in the i-th row and j-th column of the Laplace matrix.

[0173] In step S740 of some embodiments, regularization weights are determined to balance the first loss and the second loss. The sub-losses and the feature similarities of each graph node are summed to obtain a sum value, and the regularization weights are multiplied by the sum value to obtain the second loss.

[0174] Through the above steps S710 to S740, a second loss can be obtained, which is used to ensure that the generated text description is consistent with the graph structure, thereby improving the accuracy of image description generation.

[0175] In step S170 of some embodiments, a target image is acquired. The target image is an image from which text descriptions are to be generated, such as a vehicle image or a medical image. The target image is input into an image description generation model for feature extraction to obtain fused features. The fused features are then decoded using a feature decoder to obtain a text description of the target image. By using a trained image description generation model with optimized network parameters, the accuracy of image description generation is improved.

[0176] Please see Figure 8 This application also provides an image description generation apparatus that can implement the above-described image description generation method. The image description generation apparatus includes:

[0177] The detection module 810 is used to acquire sample images, perform target detection on the sample images, and obtain the object location, object category, and object visual features of the target object;

[0178] The recognition module 820 is configured to acquire an original image description of the sample image, perform keyword recognition on the original image description, and obtain a description keyword and a semantic feature of the description keyword.

[0179] The graph structure generation module 830 is configured to generate graph structure data according to the object position, the object category, the object visual feature, the description keyword, and the semantic feature, wherein the graph structure data includes graph nodes and connecting edges between any two graph nodes, and the connecting edges have edge weights.

[0180] The matrix construction module 840 is configured to construct an adjacency matrix according to the edge weights.

[0181] The feature extraction module 850 is configured to perform feature extraction on the graph nodes and the adjacency matrix by using a preset graph neural network, and obtain a target fusion feature.

[0182] The update module 860 is configured to update network parameters of the graph neural network according to the original image description, the adjacency matrix, and the target fusion feature, and obtain an image description generation model.

[0183] The description generation module 870 is configured to perform image description generation on a preset target image by using the image description generation model.

[0184] The specific implementation of the image description generation apparatus is basically the same as the specific implementation of the above-described image description generation method, and will not be described here again.

[0185] The embodiments of the present application further provide an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor implements the above-described image description generation method when executing the computer program. The electronic device can be any intelligent terminal, such as a tablet computer or a vehicle-mounted computer.

[0186] Please refer to Figure 9 , Figure 9 The hardware structure of the electronic device of another embodiment is illustrated, which includes:

[0187] The processor 910 can be implemented in a general-purpose central processing unit (CPU), a microprocessor, an application specific integrated circuit (ASIC), or one or more integrated circuits, and is configured to execute related programs to implement the technical solutions provided by the embodiments of the present application.

[0188] The memory 920 can be implemented in the form of a read only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM), etc. The memory 920 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 920 and are called and executed by the processor 910 to implement the image description generation method of the embodiments of the present application;

[0189] The input / output interface 930 is configured to realize information input and output.

[0190] The communication interface 940 is configured to realize the communication interaction between the device and other devices, and can realize the communication through a wired manner (for example, a USB, a network cable, etc.) or a wireless manner (for example, a mobile network, WIFI, Bluetooth, etc.).

[0191] The bus 950 is configured to transmit information between various components (for example, the processor 910, the memory 920, the input / output interface 930, and the communication interface 940) of the device.

[0192] The processor 910, the memory 920, the input / output interface 930, and the communication interface 940 are connected to each other through the bus 950 to realize the communication connection between the device.

[0193] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is executed by a processor to implement the above-mentioned image description generation method.

[0194] The memory is a non-transitory computer readable storage medium, which can be used to store a non-transitory software program and a non-transitory computer executable program. In addition, the memory can include a high-speed random access memory, and can also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state memory device. In some embodiments, the memory can optionally include a memory remotely arranged relative to the processor, and these remote memories can be connected to the processor through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0195] The image description generation method, the image description generation device, the electronic device, and the computer storage medium provided by the embodiments of the present application can represent the complex semantic relationship between the image and the text through the graph structure data, and the graph neural network can learn the structured information of the graph structure data, so as to generate more rich and accurate image descriptions.

[0196] The embodiments described in the specification are for more clearly illustrating the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions provided by the embodiments of the present application. Those skilled in the art can know that, with the evolution of technology and the appearance of new application scenarios, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0197] Those skilled in the art can understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and can include more or fewer steps than shown in the figures, or combine certain steps, or different steps.

[0198] The device embodiments described above are only schematic, and the units described as separate components can or can not be physically separate, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments of the present application.

[0199] Those skilled in the art can understand that all or some of the steps in the above disclosed method, the functional modules / units in the system and the device can be implemented as software, firmware, hardware and their appropriate combinations.

[0200] The terms "first", "second", "third", "fourth" and the like (if any) in the specification of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, system, product or device that includes a series of steps or units does not have to be limited to those steps or units clearly listed, but can include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0201] It should be understood that, in the application, "at least one" refers to one or more, and "multiple" refers to two or more. "And / or" is used to describe the association relationship of the associated objects, which means that there can be three relationships, for example, "A and / or B" can represent three cases of only A, only B, and A and B existing at the same time, wherein A and B can be singular or plural. The character " / " generally represents an "or" relationship between the associated objects before and after it. "At least one of the following" or similar expressions means any combination of these items, including any combination of single or multiple items. For example, at least one of a, b or c can represent a, b, c, "a and b", "a and c", "b and c", or "a and b and c", wherein a, b, and c can be single or multiple.

[0202] In several embodiments provided in the application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the above units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed mutual ones can be indirect coupling or communication connection through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0203] The units described above as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. According to actual needs, some or all of the units can be selected to achieve the purpose of the embodiment scheme.

[0204] In addition, each functional unit in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0205] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, and includes multiple instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods of the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various program storage media.

[0206] The preferred embodiments of the embodiments of the present application are described above with reference to the accompanying drawings, and are not limited to the scope of the embodiments of the present application. Any modifications, equivalent replacements and improvements made by those skilled in the art without departing from the scope and essence of the embodiments of the present application shall be within the scope of the embodiments of the present application.

Claims

1. An image description generation method characterized by, The method comprises: acquiring a sample image, performing target detection on the sample image to obtain an object position, an object category and an object visual feature of a target object; acquiring an original image description of the sample image, performing keyword recognition on the original image description to obtain a description keyword and a semantic feature of the description keyword; generating graph structure data according to the object position, the object category, the object visual feature, the description keyword and the semantic feature; wherein the graph structure data comprises graph nodes and connecting edges between any two of the graph nodes, and the connecting edges have edge weights; constructing an adjacency matrix according to the edge weights; extracting features of the graph nodes and the adjacency matrix through a preset graph neural network to obtain target fusion features; updating network parameters of the graph neural network according to the original image description, the adjacency matrix and the target fusion features to obtain an image description generation model; generating an image description of a preset target image through the image description generation model.

2. The method of claim 1, wherein, The generating of the graph structure data according to the object position, the object category, the object visual feature, the description keyword and the semantic feature comprises: constructing an image target node according to the object position, the object category and the object visual feature; constructing a text keyword node according to the description keyword and the semantic feature; determining the graph nodes according to the image target node and the text keyword node, and calculating an initial weight between any two of the graph nodes; generating the connecting edges between any two of the graph nodes according to the initial weight, and determining the edge weights of the connecting edges according to the initial weight; generating the graph structure data according to the graph nodes, the connecting edges and the edge weights.

3. The method of claim 2, wherein, The calculation of the initial weight between any two of the graph nodes comprises: calculating a matching probability between any two of the graph nodes through an image encoder; calculating a node similarity between any two of the graph nodes; performing weighted fusion on the matching probability and the node similarity to obtain the initial weight.

4. The method of claim 1, wherein, The graph neural network comprises a graph convolution network and a graph attention network, and the feature extraction of the graph nodes and the adjacency matrix through the preset graph neural network to obtain the target fusion features comprises: performing first feature extraction on the graph nodes and the adjacency matrix through the graph convolution network to obtain initial node features; obtaining neighbor nodes of the graph nodes according to the adjacency matrix; calculating attention weights between the graph nodes and the neighbor nodes; performing second feature extraction on the attention weights and the initial node features through the graph attention network to obtain the target fusion features.

5. The method of claim 4, wherein, The second feature extraction on the attention weights and the initial node features through the graph attention network to obtain the target fusion features comprises: performing feature screening on the initial node features to obtain neighborhood node features of the neighbor nodes; The graph attention network is used for feature aggregation on the attention weight and the neighborhood node feature, to obtain a target node feature; The initial node feature is updated according to the target node feature, to obtain the target fusion feature.

6. The method according to any one of claims 1 to 5, characterized in that, The network parameter of the graph neural network is updated according to the original image description, the adjacency matrix and the target fusion feature, to obtain an image description generation model, which comprises: A reference image description of the sample image is generated according to the target fusion feature; A first loss is obtained by loss calculation according to the original image description and the reference image description; A second loss is obtained by loss calculation according to the adjacency matrix and the target fusion feature; A target loss is obtained by loss weighting on the first loss and the second loss; The network parameter of the graph neural network is updated according to the target loss, to obtain the image description generation model.

7. The method of claim 6, wherein, The second loss is obtained by loss calculation according to the adjacency matrix and the target fusion feature, which comprises: A degree matrix of the adjacency matrix is calculated; A Laplacian matrix is determined according to the adjacency matrix and the degree matrix, and a sub-loss is obtained by loss calculation according to the Laplacian matrix and the target fusion feature; A feature similarity between the target fusion features is calculated according to the adjacency matrix; The second loss is determined according to the sub-loss and the feature similarity.

8. An image description generation apparatus characterized by comprising: The device comprises: A detection module is configured to acquire a sample image, perform target detection on the sample image, and obtain object positions, object categories and object visual features of target objects; An identification module is configured to acquire an original image description of the sample image, perform keyword identification on the original image description, and obtain description keywords and semantic features of the description keywords; A graph structure generation module is configured to generate graph structure data according to the object positions, the object categories, the object visual features, the description keywords and the semantic features; the graph structure data comprises graph nodes and connecting edges between any two graph nodes, and the connecting edges have edge weights; A matrix construction module is configured to construct an adjacency matrix according to the edge weights; A feature extraction module is configured to perform feature extraction on the graph nodes and the adjacency matrix by a preset graph neural network, to obtain target fusion features; An update module is configured to update network parameters of the graph neural network according to the original image description, the adjacency matrix and the target fusion features, to obtain an image description generation model; A description generation module is configured to generate an image description of a preset target image by the image description generation model.

9. An electronic device, comprising: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method of any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the method of any one of claims 1 to 7.

Images