Image processing method, scene graph generation model training method, and electronic device

Abstract

The present disclosure relates to an image processing method, a scene graph generation model training method and an electronic device, and relates to the field of image processing. The method of the present disclosure comprises: extracting one or more target pairs and information of each target pair from an image, wherein the information of each target pair comprises: feature information, position information and classification information of a first target, feature information, position information and classification information of a second target; determining a visual relationship feature of each target pair according to the information of each target pair; determining a relationship saliency of each target pair according to the visual relationship feature of each target pair; and generating a scene graph with relationship saliency corresponding to the image according to the feature information of each target, the visual relationship feature of each target pair and the relationship saliency of each target pair.

Description

Technical Field

[0001] This disclosure relates to the field of image processing, and in particular to an image processing method, a training method for a scene graph generation model, and an electronic device. Background Technology

[0002] Scene graphs are symbolic representations of visual scenes. They abstract objects in the scene as graph nodes and the relationships between objects as edges connecting the nodes. Scene graph generation techniques take a given image as input and generate a scene graph representation corresponding to the image scene through an algorithmic model. The challenge lies not only in detecting all objects in the image but also in recognizing the visual relationships between them. Visual reasoning based on scene graphs is crucial for understanding the rich semantics in image data and can support numerous downstream vision-language tasks.

[0003] The basic idea of ​​the known scene graph generation method is as follows: First, detect all targets in the image, then enumerate the combinations of pairs of targets and construct the relationship features of the target pairs; next, strengthen the features of targets and relationships; finally, classify targets and relationships based on these features and output the scene graph. Summary of the Invention

[0004] The inventors discovered that existing scene graph generation techniques focus on capturing the relationships between objects as completely as possible, inevitably leading to scene graphs that fail to focus on the most important or salient visual relationships. This differs from human intelligence's perception of visual scenes—we always quickly focus our attention on the most salient objects and relationships in an image. This deficiency also hinders the effectiveness of scene graphs in supporting downstream tasks. For example, when describing an image, people typically focus on the events constituted by the most salient objects and relationships, while scene graphs generated based on existing technologies often fail to focus on the key content of the image and cannot accurately describe the visual relationships within it.

[0005] One of the technical problems this disclosure aims to solve is: how to generate more accurate scene maps from images.

[0006] According to some embodiments of this disclosure, an image processing method is provided, comprising: extracting one or more target pairs and information of each target pair from an image, wherein the information of each target pair includes: feature information, location information and classification information of a first target, and feature information, location information and classification information of a second target; determining visual relationship features of each target pair based on the information of each target pair; determining the relationship salience of each target pair based on the visual relationship features of each target pair; and generating a scene map with relationship salience corresponding to the image based on the feature information of each target, the visual relationship features of each target pair and the relationship salience of each target pair.

[0007] In some embodiments, determining the visual relationship features of each target pair based on the information of each target pair includes: for each target pair, determining, based on the position information of the first target and the position information of the second target, the smallest rectangular region containing the first target and the second target as a union region, and determining the intersection region of the first target and the second target as an intersection region; determining the feature information and position information of the union region, and the feature information and position information of the intersection region; and determining the visual relationship features of the target pair based on the information of the target pair, the feature information and position information of the union region, and the feature information and position information of the intersection region.

[0008] In some embodiments, determining the visual relationship features of the target pair based on the information of the target pair, the feature information and location information of the union region, and the feature information and location information of the intersection region includes: determining the appearance features corresponding to the target pair based on the feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region; determining the spatial features corresponding to the target pair based on the location information of the first target, the location information of the second target, the location information of the union region, and the location information of the intersection region; determining the semantic features corresponding to the target pair based on the classification information of the first target and the classification information of the second target; and determining the visual relationship features of the target pair based on the appearance features, spatial features, and semantic features corresponding to the target pair.

[0009] In some embodiments, determining the appearance features corresponding to the target pair based on the feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region includes: inputting the feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region into an appearance feature extractor, mapping them into feature vectors of a preset dimension, and obtaining the appearance features of the first target, the appearance features of the second target, the appearance features of the union region, and the appearance features of the intersection region; and concatenating the appearance features of the first target, the appearance features of the second target, the appearance features of the union region, and the appearance features of the intersection region to obtain the appearance features corresponding to the target pair.

[0010] In some embodiments, for each region of the bounding box region of the first target, the bounding box region of the second target, the union region, and the intersection region, features of each region are extracted from the feature map of the image and used as feature information of the first target, feature information of the second target, feature information of the union region, and feature information of the intersection region, respectively.

[0011] In some embodiments, the location information of the first target includes the vertex coordinates of the bounding box of the first target, the location information of the second target includes the vertex coordinates of the bounding box of the second target, the location information of the union region includes the vertex coordinates of the union region, and the location information of the intersection region includes the vertex coordinates of the intersection region. Determining the spatial features corresponding to the target pair based on the location information of the first target, the second target, the union region, and the intersection region includes: for each region in the bounding box region of the first target, the bounding box region of the second target, the union region, and the intersection region, determining the coordinates of the center point, the width, and the height; determining spatial features representing the position and proportion of the region in the image based on the coordinates of the center point, vertex coordinates, width, height, and the width and height of the image; and concatenating the spatial features of the bounding box region of the first target, the bounding box region of the second target, the intersection region, and the union region to obtain the spatial features corresponding to the target pair.

[0012] In some embodiments, the classification information includes: category name. Determining the semantic features corresponding to the target pair based on the classification information of the first target and the classification information of the second target includes: encoding the category name of the first target and inputting it into the word embedding matrix to obtain the word embedding vector of the first target; encoding the category name of the second target and inputting it into the word embedding matrix to obtain the word embedding vector of the second target; concatenating the word embedding vector of the first target and the word embedding vector of the second target to obtain the semantic features corresponding to the target pair.

[0013] In some embodiments, determining the relation saliency of each target pair based on the visual relation features of each target pair includes: for each target pair, inputting the visual relation features of the target pair into each sub-classifier in the relation saliency estimator to obtain the predicted probability of the relation saliency level output by each sub-classifier, wherein each sub-classifier corresponds to a relation saliency level; and determining the relation saliency of the target pair based on the predicted probability corresponding to each relation saliency level.

[0014] In some embodiments, generating a scene graph with relational saliency corresponding to an image based on the feature information of each target, the visual relationship features of each target pair, and the relational saliency of each target pair includes: treating each target as a node in the scene graph; for each node, updating the feature information of the node based on the feature information of the node, the feature information of each neighboring node of the node, and the relational saliency of the target pair formed by the node and each neighboring node; for each edge in the scene graph, determining the feature information of the edge based on the feature information of the two nodes connected by the edge and the visual relationship features of the target pair formed by the two nodes connected by the edge; determining the target category of each node based on the feature information of each node, and determining the predicate category of each edge based on the feature information of each edge, so as to generate a scene graph with relational saliency corresponding to the image.

[0015] In some embodiments, for each node, updating the node's feature information based on the node's feature information, the feature information of each of the node's neighboring nodes, and the salience of the relationship between the node and the target pairs formed by the node and each of the neighboring nodes includes: in each round of updating, for each node, selecting a preset number of neighboring nodes as associated nodes based on the salience of the relationship between the node and the target pairs formed by the node and each of the neighboring nodes; updating the node's feature information based on the updated feature information of the node in the previous round of updating and the updated feature information of the associated nodes of the node, wherein the feature information of each node in the first round of updating is feature information extracted from the image.

[0016] In some embodiments, updating the feature information of a node based on the updated feature information of the node in the previous update process and the updated feature information of the associated nodes of the node includes: for each associated node of the node, determining a first weight for message aggregation from the node to the associated node and a second weight for message aggregation from the associated node to the node based on an attention mechanism; fusing the feature information of each associated node updated in the previous update process according to the updated feature information of each associated node in the previous update process and the first and second weights corresponding to each associated node to obtain the fused feature information of the node in the current update process; and inputting the updated feature information of the node in the previous update process and the fused feature information of the node in the current update process into a first gated recurrent unit (GRU) to obtain the updated feature information of the node in the current update process.

[0017] In some embodiments, for each edge in the scene graph, determining the feature information of the edge based on the feature information of the two nodes connected by the edge and the visual relationship features of the target pair formed by the two nodes connected by the edge includes: in each round of update, for each edge in the scene graph, determining the fusion feature information corresponding to the edge in the current round of update based on the feature information of the two nodes connected by the edge after the update in the previous round of update; inputting the feature information of the edge determined in the previous round of update and the fusion feature information corresponding to the edge in the current round of update into the second gated recurrent unit (GRU) to obtain the feature information of the edge determined in the current round of update, wherein the feature information of each edge in the first round of update is the visual relationship features of the target pair formed by the two nodes connected by the edge.

[0018] In some embodiments, during each round of updates, for each edge in the scene graph, determining the fusion feature information corresponding to the edge in the current round of updates, based on the feature information of the two nodes connected by the edge after the update in the previous round of updates, includes: during each round of updates, for each edge in the scene graph, concatenating the feature information of the start node and the end node along the direction of the edge, and inputting the concatenated feature information into a fully connected layer to determine the fusion feature information corresponding to the edge in the current round of updates, wherein the fully connected layer includes a non-linear activation function.

[0019] In some embodiments, determining the target category of each node based on the feature information of each node, and determining the predicate category of each edge based on the feature information of each edge, to generate a scene graph with relational saliency corresponding to the image includes: inputting the feature information of each node into a target classifier to obtain a first probability that each node belongs to each target category; inputting the feature information of each edge into a predicate classifier to obtain a second probability that each edge belongs to each predicate category; for each triplet consisting of the target category of two nodes and the predicate category of the edge between the two nodes, determining the probability of the triplet based on the first probability, the second probability, and the prediction probability that the relational saliency of the target pair consisting of the two nodes is greater than a preset saliency level; selecting triplets based on the probabilities of all triplets, using the target category and predicate category in the selected triplet as the target category of the corresponding node and the predicate category of the corresponding edge, respectively, and adding the relational saliency of the target pair consisting of the two nodes to the corresponding edge to generate a scene graph with relational saliency.

[0020] In some embodiments, the method further includes at least one of the following: when the target in the image includes a person, determining the behavior or intention of the person based on a scene graph; when the target in the image includes both people and objects, determining the relationship between the people and objects based on a scene graph, and sorting the objects based on the relationship between the people and objects; responding to visual questions based on a scene graph; performing cross-modal retrieval based on a scene graph; and generating descriptive text for the image based on a scene graph.

[0021] According to other embodiments of this disclosure, a training method for a scene graph generation model is provided, comprising: acquiring multiple training sample images, wherein each training sample image is labeled with the target category of each target, the relational saliency of each target pair in one or more target pairs, and the predicate category corresponding to each target pair; for each training sample image, inputting the training sample image into the target detector of the scene graph generation model, extracting target pairs and information of each target pair, wherein the information of each target pair in the sample image includes: feature information, location information, and classification information of a third target, and feature information, location information, and classification information of a fourth target; and inputting the information of each target pair into the target interaction encoder of the scene graph generation model. The visual relationship features of each target pair are determined; the visual relationship features of each target pair are input into the relationship saliency estimator of the scene graph generation model to determine the relationship saliency of each target pair; the feature information of each target, the visual relationship features of each target pair, and the relationship saliency of each target pair are input into the scene graph generator of the scene graph generation model to generate a scene graph with relationship saliency corresponding to the image; the parameters of the scene graph generation model are adjusted according to the target category of each node in the scene graph, the predicate category of each edge, the relationship saliency of any two nodes forming a target pair, the target category of each labeled target, the predicate category corresponding to each target pair, and the relationship saliency of each target pair to complete the training.

[0022] In some embodiments, adjusting the parameters of the scene graph generation model based on the target category of each node in the scene graph, the predicate category of each edge, the relation salience of any two nodes forming a target pair, the target category of each labeled target, the predicate category of each target pair, and the relation salience of each target pair includes: determining a target classification loss function based on the difference between the target categories of each node in the scene graph and the target categories of each labeled target; determining a predicate classification loss function based on the difference between the predicate categories of each edge in the scene graph and the predicate categories of each labeled target pair; determining a relation salience loss function based on the difference between the relation salience of any two nodes forming a target pair in the scene graph and the relation salience of each labeled target pair; determining a total loss function based on the target classification loss function, the predicate classification loss function, and the relation salience loss function; and adjusting the parameters of the scene graph generation model based on the total loss function.

[0023] In some embodiments, inputting the visual relationship features of each target pair into the relationship saliency estimator of the scene graph generation model to determine the relationship saliency of each target pair includes: for each target pair, inputting the visual relationship features of the target pair into each sub-classifier in the relationship saliency estimator to obtain the predicted probability of the relationship saliency level output by each sub-classifier, wherein each sub-classifier corresponds to a relationship saliency level; and determining the relationship saliency of the target pair based on the predicted probability corresponding to each relationship saliency level.

[0024] In some embodiments, determining the relationship saliency loss function based on the difference between the relationship saliency of any two nodes forming a target pair in the scene graph and the relationship saliency of each labeled target pair includes: for any two nodes forming a target pair in the scene graph, determining a first sub-loss function based on the difference between the predicted probabilities of the relationship saliency levels output by each sub-classifier corresponding to the target pair and the labeled relationship saliency levels of the target pair; determining a second sub-loss function based on the difference between the predicted probabilities of two adjacent relationship saliency levels corresponding to the target pair, wherein the more instances where the predicted probability of the higher-level relationship saliency level is greater than the predicted probability of the lower-level relationship saliency level, the larger the value of the second sub-loss function; and determining the relationship saliency loss function based on the first and second sub-loss functions corresponding to any two nodes forming a target pair.

[0025] According to further embodiments of this disclosure, a scene graph generation model is provided, comprising: a target detector, configured to receive an input image and extract one or more target pairs and information of each target pair from the image, wherein the information of each target pair includes: feature information, location information, and classification information of a first target, and feature information, location information, and classification information of a second target; a target interaction encoder, configured to determine the visual relationship features of each target pair based on the information of each target pair; a relationship saliency estimator, configured to determine the relationship saliency of each target pair based on the visual relationship features of each target pair; and a scene graph generator, configured to generate a scene graph with relationship saliency corresponding to the image based on the feature information of each target, the visual relationship features of each target pair, and the relationship saliency of each target pair.

[0026] According to further embodiments of this disclosure, an image processing apparatus is provided, comprising: a target detection module, configured to extract one or more target pairs and information of each target pair from an image, wherein the information of each target pair includes: feature information, location information, and classification information of a first target, and feature information, location information, and classification information of a second target; a feature determination module, configured to determine visual relationship features of each target pair based on the information of each target pair; a relationship saliency determination module, configured to determine the relationship saliency of each target pair based on the visual relationship features of each target pair; and a generation module, configured to generate a scene map with relationship saliency corresponding to the image based on the feature information of each target, the visual relationship features of each target pair, and the relationship saliency of each target pair.

[0027] According to further embodiments of this disclosure, a training apparatus for a scene graph generation model is provided, comprising: an acquisition module, configured to acquire multiple training sample images, wherein each training sample image is labeled with the target category of each target, the relational saliency of each target pair in one or more target pairs, and the predicate category corresponding to each target pair; a target detection module, configured to, for each training sample image, input the training sample image into the target detector of the scene graph generation model, and extract the target pairs and information of each target pair, wherein the information of each target pair in the sample image includes: feature information, location information, and classification information of a third target, and feature information, location information, and classification information of a fourth target; and a feature determination module, configured to input the information of each target pair into the target interaction encoder of the scene graph generation model, and determine the target pair. The system comprises four modules: a visual relationship feature determination module for each target pair; a relationship saliency determination module for inputting the visual relationship features of each target pair into the relationship saliency estimator of the scene graph generation model to determine the relationship saliency of each target pair; a generation module for inputting the feature information of each target, the visual relationship features of each target pair, and the relationship saliency of each target pair into the scene graph generator of the scene graph generation model to generate a scene graph with relationship saliency corresponding to the image; and a parameter adjustment module for adjusting the parameters of the scene graph generation model based on the target category of each node in the scene graph, the predicate category of each edge, the relationship saliency of any two nodes forming a target pair, the target category of each labeled target, the predicate category corresponding to each target pair, and the relationship saliency of each target pair, to complete the training.

[0028] According to some further embodiments of the present disclosure, an electronic device is provided, including: a processor; and a memory coupled to the processor for storing instructions, which, when executed by the processor, cause the processor to perform an image processing method as described in any of the foregoing embodiments or a training method for a scene graph generation model as described in any of the foregoing embodiments.

[0029] According to some other embodiments of the present disclosure, a non-transitory computer-readable storage medium is provided, on which a computer program is stored, wherein when the program is executed by a processor, it implements the image processing method of any of the foregoing embodiments or the training method of the scene graph generation model of any of the foregoing embodiments.

[0030] The scheme disclosed herein extracts information from each target pair in an image. The information for each target pair includes feature information, location information, and classification information of both the first and second targets. Then, visual relationship features of each target pair are determined based on this information. These visual relationship features integrate the features, location, and category information of both the first and second targets, playing a crucial role in determining the salience of visual relationships. Furthermore, the relationship salience of each target pair is determined based on its visual relationship features. Based on the feature information of each target, the visual relationship features of each target pair, and the relationship salience of each target pair, a scene graph with relationship salience corresponding to the image is generated. This scheme aligns scene graph representation with human visual perception by adding a relationship salience attribute to the scene graph to represent the relative importance of visual relationships to the image scene, rather than treating all visual relationships indiscriminately, thereby generating a more accurate scene graph for the image.

[0031] Other features and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 A schematic flowchart illustrating some embodiments of the image processing method of this disclosure is shown.

[0034] Figure 2 A schematic diagram illustrating a scene graph generation model of some embodiments of this disclosure is shown.

[0035] Figure 3 The diagram illustrates a flowchart of a training method for a scene graph generation model according to some embodiments of the present disclosure.

[0036] Figure 4 A schematic diagram of the structure of an image processing apparatus according to some embodiments of the present disclosure is shown.

[0037] Figure 5A schematic diagram of the structure of a training apparatus for a scene graph generation model according to some embodiments of the present disclosure is shown.

[0038] Figure 6 A schematic diagram of the structure of an electronic device according to some embodiments of the present disclosure is shown.

[0039] Figure 7 Schematic diagrams of the structure of electronic devices according to other embodiments of the present disclosure are shown. Detailed Implementation

[0040] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of at least one exemplary embodiment is merely illustrative and is in no way intended to limit this disclosure or its application or use. All other embodiments obtained by those skilled in the art based on the embodiments of this disclosure without creative effort are within the scope of protection of this disclosure.

[0041] This disclosure proposes an image processing method, which is described below in conjunction with... Figures 1-3 Describe it.

[0042] Figure 1 Flowcharts are shown for some embodiments of the image processing method of this disclosure. For example... Figure 1 As shown, the method of this embodiment includes steps S102 to S108.

[0043] In step S102, one or more target pairs and information for each target pair are extracted from the image.

[0044] The scene graph generation model disclosed herein is a relation saliency-guided message passing framework, which can be called SMP (Saliency Message Passing), used to generate scene graphs with relation saliency. For example... Figure 2As shown, the SMP includes: an object detector, an object interaction encoder, a relation saliency estimator, and a scene graph generator. The object detector receives an input image and extracts one or more object pairs and information for each object pair from the image. The information for each object pair includes: feature information, location information, and classification information of a first object, and feature information, location information, and classification information of a second object. The object interaction encoder determines the visual relation features of each object pair based on the information of each object pair. The relation saliency estimator determines the relation saliency of each object pair based on the visual relation features of each object pair. The scene graph generator generates a scene graph with relation saliency corresponding to the image based on the feature information of each object, the visual relation features of each object pair, and the relation saliency of each object pair. The specific functions of the object detector, object interaction encoder, relation saliency estimator, and scene graph generator can be found in subsequent embodiments.

[0045] First, the image is input into the object detector of the scene graph generation model, which detects one or more objects in the image and pairs each pair of objects. The object detector can be a neural network model such as Faster R-CNN, but is not limited to the examples given. The object detector can output the location information of each object, such as the coordinates of its bounding box, and also the classification information, i.e., the category of the object. The feature extraction network in the object detector can further extract the image's feature information (e.g., feature maps), and then extract the feature information of each object from the image's feature information. For example, for each object... i It can be represented as o i =(l i ,b i ), where l i The category label of the target. It represents the spatial location of the target bounding box.

[0046] Each target pair includes a first target and a second target. The information in each target pair includes: feature information, location information, and classification information of the first target; and feature information, location information, and classification information of the second target. In the visual relationship description, the first target can be the subject, the second target can be the object, and the visual relationship between them can be the predicate. For example, each visual relationship r... m ∈R is a <subject-verb-object> triple, which can be represented as r m =(o i ,p ij ,o j ), where p ij These are labels for predicate categories, used to describe the target. i to oj Visual interaction.

[0047] In step S104, the visual relationship features of each target pair are determined based on the information of each target pair.

[0048] The information of each target pair is input into the target interactive encoder to determine the visual relationship of each target pair. In order to more accurately describe the visual relationship features of each target pair, the union region and intersection region of the first target and the second target are determined based on the position information of the first target and the second target, and the visual relationship features of the target pair are generated using the feature information of the union region and the intersection region and the position information.

[0049] In some embodiments, for each target pair, based on the position information of the first target and the position information of the second target, a minimum rectangular region containing the first target and the second target is determined as a union region, and the intersection region of the first target and the second target is determined as an intersection region; feature information and position information of the union region and the intersection region are determined; based on the information of the target pair, the feature information and position information of the union region, and the feature information and position information of the intersection region, the visual relationship features of the target pair are determined. For example, the minimum rectangular region containing the bounding box of the first target and the bounding box of the second target is taken as the union region, and the intersection region of the bounding boxes of the first target and the second target is taken as the intersection region.

[0050] To improve the accuracy of visual relation features for each target pair, the designed target interaction encoder is capable of learning relation representations from intersection and union, as well as from three different perspectives (i.e., appearance, semantics, and space). Specifically, the target interaction encoder uses three feature extractors (i.e., appearance, semantic, and spatial feature extractors) to learn relation representations from different perspectives.

[0051] In some embodiments, based on the feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region, the appearance features corresponding to the target pair are determined; based on the location information of the first target, the location information of the second target, the location information of the union region, and the location information of the intersection region, the spatial features corresponding to the target pair are determined; based on the classification information of the first target and the classification information of the second target, the semantic features corresponding to the target pair are determined; and based on the appearance features, spatial features, and semantic features corresponding to the target pair, the visual relationship features of the target pair are determined.

[0052] In some embodiments, for each region in the bounding box region of the first target, the bounding box region of the second target, the union region, and the intersection region, features of each region are extracted from the image's feature map, and these features are used as feature information for the first target, the second target, the union region, and the intersection region, respectively. The ROIAlign (Region of Interest Alignment) algorithm can be used to extract appearance feature maps for each region from the image feature map (e.g., extracting a 7×7×D feature map, where D = 512 or 256 depending on the backbone network, and D is the number of feature channels). The module for extracting features for each region can be set in the object detector or the object interactive encoder, or partially in the object detector and partially in the object interactive encoder, or it can be set independently without affecting the overall model's usability and performance.

[0053] In some embodiments, for the appearance feature extractor, the feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region are respectively input into the appearance feature extractor and mapped into feature vectors of a preset dimension to obtain the appearance features of the first target, the appearance features of the second target, the appearance features of the union region, and the appearance features of the intersection region. The appearance features of the first target, the appearance features of the second target, the appearance features of the union region, and the appearance features of the intersection region are concatenated to obtain the appearance features corresponding to the target pair.

[0054] For example, two fully connected layers with ReLU activation functions are used to map each appearance feature map into a low-dimensional vector, i.e., the appearance feature of a certain region. In the special case where the intersection region of two targets is an empty set, a vector of all zeros can be used to represent the appearance features of the intersection. By concatenating the appearance features of the four regions, the appearance features corresponding to the target pair are obtained.

[0055] For the spatial feature extractor, in some embodiments, the location information of the first target includes the vertex coordinates of the bounding box of the first target, the location information of the second target includes the vertex coordinates of the bounding box of the second target, the location information of the union region includes the vertex coordinates of the union region, and the location information of the intersection region includes the vertex coordinates of the intersection region. For each region in the bounding box region of the first target, the bounding box region of the second target, the union region, and the intersection region, the coordinates of the center point, the width, and the height are determined. Based on the coordinates of the center point, the vertex coordinates, the width, the height, and the width and height of the region, spatial features representing the position and proportion of the region in the image are determined. The spatial features of the bounding box region of the first target, the bounding box region of the second target, the intersection region, and the union region are concatenated to obtain the spatial features corresponding to the target pair.

[0056] For example, the ratios of the x and y coordinates of the center point to the width and height of the image can be determined, the ratios of the width and height of the region to the width and height of the image can be determined, the ratios of the x and y coordinates of the vertices to the width and height of the image can be determined, and the ratio of the area of ​​the region to the area of ​​the image can be determined. One or more of the above ratios can be used as the original spatial features of each region, and the spatial features of the region can be generated based on the original spatial features.

[0057] For example, for each region, the bounding box coordinates of that region are represented as (x1, y1, x2, y2), and the original spatial features of each region are... This can be expressed using the following formula:

[0058]

[0059] in, The coordinates of the center point of the bounding box are represented by (w, h), where (x2-x1, y2-y1) represents the width and height of the bounding box, and (W, H) represents the width and height of the overall image. For cases where the intersection region is empty, a 9-dimensional zero vector can be used to represent the original spatial features. Then, a fully connected layer maps the original spatial features of each region to a 64-dimensional vector, i.e., the spatial features. These spatial features describe the relative position and proportion of each region in the image. Finally, the spatial features corresponding to the target pair can be obtained by concatenating the spatial features of the four regions.

[0060] For the semantic feature extractor, in some embodiments, the classification information includes: category name; the category name of the first target is encoded and input into the word embedding matrix to obtain the word embedding vector of the first target; the category name of the second target is encoded and input into the word embedding matrix to obtain the word embedding vector of the second target; the word embedding vector of the first target and the word embedding vector of the second target are concatenated to obtain the semantic features corresponding to the target pair.

[0061] For example, the category names can be encoded using one-hot encoding, and the encoded input word embedding matrix can be used to obtain word embedding vectors. An embedding layer can be used to encode the category names of the first and second targets, i.e. Among them W e For example, the word embedding matrix is ​​initialized by GloVe. However, this is not limited to the given example; l is a one-hot vector based on the entire tag vocabulary. By concatenating the two embedding vectors of the first and second targets, the semantic features corresponding to the target pair are obtained.

[0062] Finally, for each target pair, the appearance features, spatial features, and semantic features corresponding to the target pair can be concatenated to obtain the visual relationship features of the target pair.

[0063] In step S106, the salience of the relationship between each target pair is determined based on the visual relationship features of each target pair.

[0064] The visual relationship features of each target pair are input into the relationship saliency estimator to obtain the relationship saliency of each target pair. The goal of the relationship saliency estimator is to identify visually salient relationships in an image, making them naturally aligned with human perception.

[0065] Relationship saliency can be divided into ordered levels (e.g., 0 represents no relation, 1 represents a perceived minor relation, and 2 represents a salient relation). Therefore, the relation saliency estimation task can be modeled as an ordered regression problem. To understand this ordered regression problem, it can be decomposed into several binary classification subproblems. Given a visual relation r... i The feature is represented as u i The goal is to predict its corresponding significance. Which of the S levels (denoted as {0, 1, ..., S-1}) does it belong to? This problem can be transformed into S-1 subtasks, each with the goal of determining... Does it hold true, where k∈{0,1,…,S-2}?

[0066] Therefore, in some embodiments, for each target pair, the visual relationship features of the target pair are input into each sub-classifier in the relationship saliency estimator to obtain the predicted probability of the relationship saliency level output by each sub-classifier, wherein each sub-classifier corresponds to a relationship saliency level; and the relationship saliency of the target pair is determined based on the predicted probability corresponding to each relationship saliency level.

[0067] The relation saliency estimator can be implemented using a multilayer perceptron (MLP), and is not limited to the example given. i As input, there are S-1 sub-classifiers, each containing, for example, a sigmoid activation function and an output unit. The output value of the k-th output unit is denoted as... express The probability value. For example, the predicted probabilities corresponding to each relation saliency level are summed to determine the relation saliency of the target pair. For example, for a visual relation r′, the expected value of the saliency can be obtained by summing all the output probabilities as the relation saliency of the target pair, which can be expressed by the following formula:

[0068]

[0069] In step S108, a scene map with relational saliency is generated corresponding to the image based on the feature information of each target, the visual relational features of each target pair, and the relational saliency of each target pair.

[0070] The scene graph data structure G = (O, R) is formed by combining all targets O = {o1, ..., o}. N Abstracting this into graph nodes, where N is the number of targets, the visual relationship between any two targets is R = {r1, ..., r...} M This can be abstracted as edges in a scene graph, where M represents the number of target pairs (M = N * (N-1)). Considering that traditional scene graphs cannot express the salience of visual relationships, a relationship salience s can be attached to each edge in the graph. ij This allows it to better align with human intelligence's perception of scenes. Accordingly, the edges in the graph can be represented as r. m =(o i ,p ij ,s ij ,o j ). s ij The higher the value, the more important the visual relationship is from a perceptual perspective.

[0071] In the scene graph generation process, for each target, contextual information is incorporated through message passing between the target and its neighboring nodes to improve target and predicate category prediction. However, message passing between the target and all its neighboring nodes not only incurs significant computational overhead but also inevitably affects the stability of the message passing process due to the involvement of secondary visual relationships in the computation. Therefore, a saliency-guided message passing module is designed in the scene graph generator to facilitate the interaction between the two subtasks of relation saliency estimation and scene graph generation. The design idea of ​​this module is to filter out insignificant relationships and then perform message passing along salient relationship connections, continuously iteratively strengthening the prediction of targets and relationships.

[0072] In some embodiments, each target is treated as a node in the scene graph. For each node, the feature information of the node is updated based on the feature information of the node, the feature information of each of the node's neighboring nodes, and the saliency of the relationship between the node and the target pairs formed by the node and each of the neighboring nodes. For each edge in the scene graph, the feature information of the edge is determined based on the feature information of the two nodes connected by the edge and the visual relationship features of the target pairs formed by the two nodes connected by the edge. The target category of each node is determined based on the feature information of each node, and the predicate category of each edge is determined based on the feature information of each edge, so as to generate a scene graph with relationship saliency corresponding to the image.

[0073] Use v in the scene diagram i and e ij Let represent the feature information of graph node i and edge i→j, respectively, where edge i→j represents the directed edge from node i to node j, and the corresponding relation salience value is denoted as s. ij v i and e ijIt is initialized with the feature information of the target extracted from the image and the visual relationship features output by the target interaction encoder.

[0074] The message passing (update) process can be executed in multiple rounds. In each round of update, the feature information of each node and the feature information of each edge are updated. In some embodiments, in each round of update, for each node, a preset number of neighboring nodes are selected as associated nodes based on the salience of the relationship between the node and each of its neighboring nodes; the feature information of the node is updated based on the feature information of the node updated in the previous round of update and the feature information of the associated nodes of the node updated in the previous round of update, wherein the feature information of each node in the first round of update is the feature information extracted from the image.

[0075] In some embodiments, each round of message passing (update) includes two steps: message pooling and feature fine-tuning. The goal of message pooling is to aggregate messages from neighboring nodes. For each node, the salience of the relationship between that node and each of its neighboring nodes can be sorted from highest to lowest, and a preset number of neighboring nodes are selected as associated nodes according to the sorting order. That is, for each node, messages only need to be obtained from the K (preset number) most associated neighboring nodes (sorted according to relationship salience).

[0076] In some embodiments, for each associated node of a node, a first weight for message aggregation from the node to the associated node and a second weight for message aggregation from the associated node to the node are determined based on an attention mechanism. Based on the feature information of each associated node after the update in the previous round of update and the first and second weights corresponding to each associated node, the feature information of each associated node after the update in the previous round of update is fused to obtain the fused feature information of the node in the current round of update.

[0077] The first weight and the second weight correspond to edges with different directions. The first weight corresponds to connections outward from the node, while the second weight corresponds to connections inward from other nodes to the node. For example, for each associated node, the updated feature information of the associated node in the previous update is multiplied by the first weight and the second weight respectively to obtain the first message and the second message. The first messages corresponding to each associated node are accumulated, and the second messages corresponding to each associated node are accumulated. Then, the accumulated first messages and second messages are fused using a mapping function to obtain the fused feature information corresponding to the node in the current update.

[0078] For example, for the message of node i (fused feature information, denoted as m) i This is obtained by concatenating node messages from outward and inward connections, and using the mapping function φ. vThe following formula can be used to determine the fusion process:

[0079]

[0080] Where, topK represents the set of salience values ​​of the top K most significant visual relationships with node i as the start / end point, f out and f in f is a function that learns attention weights during message aggregation. out (v i ,v j ) and f in (v i ,v j () represents the first weight and the second weight, respectively. f can be implemented using a fully connected layer with a Softmax normalization function. out and f in φ can be achieved using a fully connected layer with a LeakyReLU nonlinear activation function. v .

[0081] For the feature fine-tuning process, in some embodiments, the feature information of the node after the update in the previous round of update and the fused feature information of the node corresponding to the current round of update are input into the first gated recurrent unit (GRU) to obtain the feature information of the node after the update in the current round of update.

[0082] The feature fine-tuning step enriches the node representation by utilizing the contextual information embedded in the pooling messages. The updated feature information of the node during this round of updates can be determined using the following formula:

[0083] v i =GRU v (v i ,m i (4)

[0084] Among them, GRU v This represents the first GRU network used for memory and fine-tuning.

[0085] For each edge, the message pooling process is as follows: In some embodiments, during each round of updates, for each edge in the scene graph, based on the feature information of the two nodes connected by the edge after the update in the previous round of updates, the fused feature information corresponding to the edge in the current round of updates is determined.

[0086] In some embodiments, during each round of updates, for each edge in the scene graph, the feature information of the start node and the end node along the direction of the edge is concatenated, and the concatenated feature information is input into a fully connected layer to determine the fused feature information corresponding to the edge in this round of updates. The fully connected layer includes a non-linear activation function.

[0087] For example, for the message of edge i→j (fused feature information, denoted as m) ij The feature of node i and node j is obtained by concatenating and mapping the features, and can be determined using the following formula:

[0088] m ij =φ e ([v i ,v j (5)

[0089] Where, φ e It is a mapping function that can be implemented by a fully connected layer with a LeakyReLU nonlinear activation function.

[0090] The feature fine-tuning process is as follows: In some embodiments, the feature information of the edge determined in the previous update process and the fused feature information of the edge corresponding to the current update process are input into the second gated recurrent unit (GRU) to obtain the feature information of the edge determined in the current update process. In the first update process, the feature information of each edge is the visual relationship feature of the target pair composed of the two nodes connected by the edge.

[0091] For example, the following formula can be used to determine the updated feature information of the edge during the current update process:

[0092] e ij =GRU e (e ij ,m ij (6)

[0093] Among them, GRU e These represent the second GRU network used for memorization and edge fine-tuning, respectively.

[0094] Finally, after T rounds of message passing iterations, we obtain representations of nodes and edges reinforced with context messages, which are used to predict the target category of nodes and the predicate category of edges, respectively.

[0095] In some embodiments, the feature information of each node is input into a target classifier to obtain a first probability that each node belongs to each target category; the feature information of each edge is input into a predicate classifier to obtain a second probability that each edge belongs to each predicate category; for each triple consisting of the target category of two nodes and the predicate category of the edge between the two nodes, the probability of the triple is determined based on the first probability, the second probability, and the predicted probability that the relational saliency of the target pair formed by the two nodes is greater than a preset saliency level; triples are selected based on the probabilities of all triples, and the target category and predicate category in the selected triples are used as the target category of the corresponding node and the predicate category of the corresponding edge, respectively, and the relational saliency of the target pair formed by the two nodes is added to the corresponding edge to generate a scene graph with relational saliency. Both the target classifier and the predicate classifier can be implemented using a two-layer perceptron with Softmax activation, and are not limited to the examples given.

[0096] For example, ranking all predicted relation triples is used for selection, and the ranking score (probability) of each triple r′=(s,p,o) is calculated using the following formula:

[0097] C r′ =P(s)·P(o)·P(p)·P(r′is salient) (7)

[0098] Where P(s), P(o), and P(p) represent the predicted probabilities for the subject, object, and predicate categories, respectively: the first probability of a node being the subject, the first probability of a node being the object, and the second probability of the predicate between the two. For example, for the triple (women, holding, bat), P(s) = 0.8 means the probability that the subject is women is 0.8, P(o) = 0.5 means the probability that the object is bat is 0.5, and P(p) = 0.7 means the probability that the predicate is holding is 0.7. Taking the saliency level S = 3 as an example, P(r′is salient) = P(S r′ >1), then the probability of the triple (women, holding, bat) is C r′ =0.8*0.5*0.7*P(S) r′ >1).

[0099] In the above embodiments, a unified model is proposed for joint saliency estimation and visual relationship detection to generate scene graphs with relational saliency. A novel saliency-guided message passing architecture (SMP) is designed to facilitate scene graph generation using estimated relational saliency, and the proposed SMP architecture can be jointly optimized end-to-end. First, an object detector is used to find all target regions in the image. Then, for each pair of targets, a target interaction encoder is used to construct a feature representation of their visual relationship from appearance, semantics, and spatial perspectives. Next, the feature representation of each visual relationship is input into a relational saliency estimator to estimate the saliency of the visual relationship. Further, relatively insignificant relationships are discarded, and message passing is only performed along salient relationships, continuously iteratively enhancing the prediction of targets and relationships, ultimately improving the algorithm's performance in generating scene graphs with relational saliency.

[0100] In the above embodiments, information for each target pair is extracted from the image. This information includes feature information, location information, and classification information for both the first and second targets. Then, visual relationship features for each target pair are determined based on this information. These visual relationship features integrate the features, location, and category information of both the first and second targets, playing a crucial role in determining the salience of visual relationships. Furthermore, the relationship salience of each target pair is determined based on its visual relationship features. Based on the feature information of each target, the visual relationship features of each target pair, and the relationship salience of each target pair, a scene graph with relationship salience corresponding to the image is generated. The above embodiments align scene graph representation with human visual perception, adding a relationship salience attribute to the scene graph to represent the relative importance of visual relationships to the image scene, rather than treating all visual relationships indiscriminately, thereby generating a more accurate scene graph for the image.

[0101] The generated scene graph can be used for downstream tasks. In some embodiments, any of the following can be performed: if the target in the image includes a person, determine the behavior or intention of the person based on the scene graph; if the target in the image includes both people and objects, determine the relationship between the people and objects based on the scene graph, and sort the objects based on the relationship between the people and objects; answer visual questions based on the scene graph; perform cross-modal retrieval based on the scene graph; generate descriptive text for the image based on the scene graph.

[0102] Because scene graphs include relationships and saliency between various objects, when an image includes people, the main actions or intentions of the people can be determined, and the main relationships between people and objects can be identified. Scene graphs can be used to answer visual questions; for example, if the question is "What are the people in the image doing?", the scene graph can be used to determine that the people in the image are playing ball. Cross-modal retrieval is also possible, such as searching for similar text based on scene graphs, or searching for similar images based on text. Descriptive text for images can also be generated from scene graphs.

[0103] The following is combined Figure 3 Describe the training method for the scene graph generation model.

[0104] Figure 3 Flowcharts are shown for some embodiments of the image processing method of this disclosure. For example... Figure 3 As shown, the method of this embodiment includes steps S302 to S312.

[0105] In step S302, multiple training sample images are acquired.

[0106] Each training sample image is labeled with the target category of each object, the salience of the relationship between each target pair in one or more target pairs, and the predicate category corresponding to each target pair.

[0107] In step S304, for each training sample image, the training sample image is input into the target detector of the scene graph generation model to extract target pairs and information for each target pair.

[0108] The information for each target pair in the sample image includes: feature information, location information, and classification information for the third target, and feature information, location information, and classification information for the fourth target.

[0109] In step S306, the information of each target pair is input into the target interaction encoder of the scene graph generation model to determine the visual relationship features of each target pair.

[0110] In step S308, the visual relationship features of each target pair are input into the relationship saliency estimator of the scene graph generation model to determine the relationship saliency of each target pair.

[0111] In step S310, the feature information of each target, the visual relationship features of each target pair, and the relationship salience of each target pair are input into the scene graph generator of the scene graph generation model to generate a scene graph with relationship salience corresponding to the image.

[0112] Steps S304 to S310 can refer to the method of the foregoing embodiments.

[0113] In step S312, the parameters of the scene graph generation model are adjusted according to the target category of each node in the scene graph, the predicate category of each edge, the saliency of the relationship between any two nodes forming a target pair, the target category of each labeled target, the predicate category of each target pair, and the saliency of the relationship between each target pair, in order to complete the training.

[0114] In some embodiments, a target classification loss function is determined based on the difference between the target categories of each node in the scene graph and the target categories of each labeled target; a predicate classification loss function is determined based on the difference between the predicate categories of each edge in the scene graph and the predicate categories corresponding to each labeled target pair; a relation saliency loss function is determined based on the difference between the relation saliency of any two nodes forming a target pair in the scene graph and the relation saliency of each labeled target pair; a total loss function is determined based on the target classification loss function, the predicate classification loss function, and the relation saliency loss function; and the parameters of the scene graph generation model are adjusted based on the total loss function.

[0115] In some embodiments, for each target pair, the visual relationship features of the target pair are input into each sub-classifier in the relationship saliency estimator to obtain the predicted probability of the relationship saliency level output by each sub-classifier, wherein each sub-classifier corresponds to a relationship saliency level; the relationship saliency of the target pair is determined based on the predicted probability corresponding to each relationship saliency level.

[0116] In some embodiments, for any two nodes forming a target pair in the scene graph, a first sub-loss function is determined based on the difference between the predicted probabilities of the relation saliency levels output by each sub-classifier corresponding to the target pair and the labeled relation saliency levels of the target pair; a second sub-loss function is determined based on the difference between the predicted probabilities of the two adjacent relation saliency levels corresponding to the target pair, wherein the more instances where the predicted probability of the higher-level relation saliency level is greater than the predicted probability of the lower-level relation saliency level, the larger the value of the second sub-loss function; and a relation saliency loss function is determined based on the first and second sub-loss functions corresponding to any two nodes forming a target pair.

[0117] For example, there are k subclassifiers, corresponding to k output units, given visual relation features u. i and the corresponding true (annotated) relationship significance level The significance loss function of the relationship can be determined using the following formula:

[0118]

[0119] Where 1{·} represents the indicator function, and BCEloss(·) represents the binary cross-entropy loss function. The first term on the right-hand side of the equation aggregates the classification losses of all sub-tasks, i.e., the first sub-loss function, while the second term suppresses contradictory cases in saliency estimation (e.g., These unreasonable situations), namely the second sub-loss function.

[0120] The target classification loss function and the predicate classification loss function can be determined using the cross-entropy loss function. For example, the target classification loss function is denoted as L. object The predicate classification loss function is denoted as L. relation The overall loss function of the SMP framework is expressed as:

[0121] L SMP =L saliency +L object +L relation (9)

[0122] The total loss function enables end-to-end training of SMP (Scene Graph Generation Model). The specific training process can be found in existing technologies and will not be elaborated further.

[0123] This disclosure also provides an image processing apparatus, which is described below in conjunction with... Figure 4 Describe it.

[0124] Figure 4 These are structural diagrams of some embodiments of the image processing apparatus of this disclosure. For example... Figure 4 As shown, the apparatus 40 in this embodiment includes: a target detection module 410, a feature determination module 420, a relationship saliency determination module 430, and a generation module 440.

[0125] The target detection module 410 is used to extract one or more target pairs and information of each target pair from an image, wherein the information of each target pair includes: feature information, location information and classification information of a first target, and feature information, location information and classification information of a second target.

[0126] The feature determination module 420 is used to determine the visual relationship features of each target pair based on the information of each target pair.

[0127] In some embodiments, the feature determination module 420 is configured to, for each target pair, determine, based on the position information of the first target and the position information of the second target, the smallest rectangular region containing the first target and the second target as a union region, and determine the intersection region of the first target and the second target as an intersection region; determine the feature information and position information of the union region, and the feature information and position information of the intersection region; and determine the visual relationship features of the target pair based on the information of the target pair, the feature information and position information of the union region, and the feature information and position information of the intersection region.

[0128] In some embodiments, the feature determination module 420 is configured to determine the appearance features corresponding to the target pair based on the feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region; determine the spatial features corresponding to the target pair based on the location information of the first target, the location information of the second target, the location information of the union region, and the location information of the intersection region; determine the semantic features corresponding to the target pair based on the classification information of the first target and the classification information of the second target; and determine the visual relationship features of the target pair based on the appearance features, spatial features, and semantic features corresponding to the target pair.

[0129] In some embodiments, the feature determination module 420 is used to input the feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region into the appearance feature extractor, respectively, and map them into feature vectors of a preset dimension to obtain the appearance features of the first target, the appearance features of the second target, the appearance features of the union region, and the appearance features of the intersection region; and to concatenate the appearance features of the first target, the appearance features of the second target, the appearance features of the union region, and the appearance features of the intersection region to obtain the appearance features corresponding to the target pair.

[0130] In some embodiments, for each region of the bounding box region of the first target, the bounding box region of the second target, the union region, and the intersection region, features of each region are extracted from the feature map of the image and used as feature information of the first target, feature information of the second target, feature information of the union region, and feature information of the intersection region, respectively.

[0131] In some embodiments, the location information of the first target includes the vertex coordinates of the bounding box of the first target, the location information of the second target includes the vertex coordinates of the bounding box of the second target, the location information of the union region includes the vertex coordinates of the union region, and the location information of the intersection region includes the vertex coordinates of the intersection region. The feature determination module 420 is used to determine the coordinates of the center point, the width, and the height of each region in the bounding box region of the first target, the bounding box region of the second target, the union region, and the intersection region; based on the coordinates of the center point, vertex coordinates, width, and height of the region, and the width and height of the image, determine the spatial features representing the position and proportion of the region in the image; and concatenate the spatial features of the bounding box region of the first target, the bounding box region of the second target, the intersection region, and the union region to obtain the spatial features corresponding to the target pair.

[0132] In some embodiments, the feature determination module 420 is used to encode the category name of the first target and input it into the word embedding matrix to obtain the word embedding vector of the first target; encode the category name of the second target and input it into the word embedding matrix to obtain the word embedding vector of the second target; and concatenate the word embedding vector of the first target and the word embedding vector of the second target to obtain the semantic features corresponding to the target pair.

[0133] The relation saliency determination module 430 is used to determine the relation saliency of each target pair based on the visual relation features of each target pair.

[0134] In some embodiments, the relation saliency determination module 430 is used to input the visual relation features of each target pair into each sub-classifier in the relation saliency estimator to obtain the predicted probability of the relation saliency level output by each sub-classifier, wherein each sub-classifier corresponds to a relation saliency level; and determine the relation saliency of the target pair based on the predicted probability corresponding to each relation saliency level.

[0135] The generation module 440 is used to generate a scene map with relational saliency corresponding to the image based on the feature information of each target, the visual relational features of each target pair, and the relational saliency of each target pair.

[0136] In some embodiments, the generation module 440 is used to treat each target as a node in the scene graph. For each node, the feature information of the node is updated according to the feature information of the node, the feature information of each neighboring node of the node, and the saliency of the relationship between the node and the target pair formed by the node and each neighboring node. For each edge in the scene graph, the feature information of the edge is determined according to the feature information of the two nodes connected by the edge and the visual relationship features of the target pair formed by the two nodes connected by the edge. The target category of each node is determined according to the feature information of each node, and the predicate category of each edge is determined according to the feature information of each edge, so as to generate a scene graph with relationship saliency corresponding to the image.

[0137] In some embodiments, the generation module 440 is used to select a preset number of neighboring nodes as associated nodes for each node in each round of update, based on the saliency of the relationship between the node and each neighboring node in the target pair; and to update the feature information of the node based on the updated feature information of the node in the previous round of update and the updated feature information of the associated nodes of the node, wherein the feature information of each node in the first round of update is the feature information extracted from the image.

[0138] In some embodiments, the generation module 440 is configured to, for each associated node of the node, determine a first weight for message aggregation from the node to the associated node and a second weight for message aggregation from the associated node to the node based on an attention mechanism; fuse the feature information of each associated node updated in the previous update process and the first and second weights corresponding to each associated node according to the feature information of each associated node updated in the previous update process to obtain the fused feature information of the node in the current update process; input the feature information of the node updated in the previous update process and the fused feature information of the node in the current update process into the first gated recurrent unit (GRU) to obtain the feature information of the node updated in the current update process.

[0139] In some embodiments, the generation module 440 is used to determine the fusion feature information corresponding to each edge in the scene graph in each round of update, based on the feature information of the two nodes connected by the edge after the update in the previous round of update; input the feature information of the edge determined in the previous round of update and the fusion feature information corresponding to the edge in the current round of update into the second gated recurrent unit (GRU) to obtain the feature information of the edge determined in the current round of update, wherein the feature information of each edge in the first round of update is the visual relationship feature of the target pair composed of the two nodes connected by the edge.

[0140] In some embodiments, the generation module 440 is used to, in each round of update, for each edge in the scene graph, concatenate the feature information of the start node and the end node along the direction of the edge, and input the concatenated feature information into the fully connected layer to determine the fused feature information corresponding to the edge in the current round of update, wherein the fully connected layer includes a non-linear activation function.

[0141] In some embodiments, the generation module 440 is used to input the feature information of each node into the target classifier to obtain the first probability that each node belongs to each target category; input the feature information of each edge into the predicate classifier to obtain the second probability that each edge belongs to each predicate category; for each triplet consisting of the target category of two nodes and the predicate category of the edge between the two nodes, the probability of the triplet is determined according to the first probability, the second probability and the prediction probability that the relational saliency of the target pair consisting of the two nodes is greater than a preset saliency level; select triplets according to the probabilities of all triplets, and use the target category and predicate category in the selected triplet as the target category of the corresponding node and the predicate category of the corresponding edge, respectively, and add the relational saliency of the target pair consisting of the two nodes to the corresponding edge to generate a scene graph with relational saliency.

[0142] In some embodiments, the apparatus further includes an application module 450 for performing at least one of the following: determining the behavior or intention of a person based on a scene graph when the target in the image includes a person; determining the relationship between a person and an object based on a scene graph when the target in the image includes both a person and an object, and sorting the object based on the relationship between the person and the object; responding to a visual question based on a scene graph; performing cross-modal retrieval based on a scene graph; and generating descriptive text for the image based on a scene graph.

[0143] This disclosure also provides a training device for a scene graph generation model, which is described below in conjunction with... Figure 5 Describe it.

[0144] Figure 5 Structural diagrams of some embodiments of the training apparatus for the scene graph generation model of this disclosure. For example... Figure 5 As shown, the apparatus 50 in this embodiment includes: an acquisition module 510, a target detection module 520, a feature determination module 530, a relationship saliency determination module 540, a generation module 550, and a parameter adjustment module 560.

[0145] The acquisition module 510 is used to acquire multiple training sample images, wherein each training sample image is labeled with the target category of each target, the relational saliency of each target pair in one or more target pairs, and the predicate category corresponding to each target pair;

[0146] The target detection module 520 is used to input the training sample image into the target detector of the scene graph generation model for each training sample image, and extract the target pairs and the information of each target pair. The information of each target pair in the sample image includes: feature information, location information and classification information of the third target, and feature information, location information and classification information of the fourth target.

[0147] The feature determination module 530 is used to input the information of each target pair into the target interaction encoder of the scene graph generation model to determine the visual relationship features of each target pair;

[0148] The relation saliency determination module 540 is used to input the visual relation features of each target pair into the relation saliency estimator of the scene graph generation model to determine the relation saliency of each target pair.

[0149] The generation module 550 is used to input the feature information of each target, the visual relationship features of each target pair, and the relationship saliency of each target pair into the scene graph generator of the scene graph generation model to generate a scene graph with relationship saliency corresponding to the image.

[0150] The parameter adjustment module 560 is used to adjust the parameters of the scene graph generation model based on the target category of each node in the scene graph, the predicate category of each edge, the saliency of the relationship between any two nodes forming a target pair, the target category of each labeled target, the predicate category of each target pair, and the saliency of the relationship between each target pair, in order to complete the training.

[0151] In some embodiments, the parameter adjustment module 560 is used to determine a target classification loss function based on the difference between the target category of each node in the scene graph and the target category of each labeled target; determine a predicate classification loss function based on the difference between the predicate category of each edge in the scene graph and the predicate category corresponding to each labeled target pair; determine a relation saliency loss function based on the difference between the relation saliency of any two nodes forming a target pair in the scene graph and the relation saliency of each labeled target pair; determine a total loss function based on the target classification loss function, the predicate classification loss function, and the relation saliency loss function; and adjust the parameters of the scene graph generation model based on the total loss function.

[0152] In some embodiments, the parameter adjustment module 560 is used to input the visual relationship features of each target pair into each sub-classifier in the relationship saliency estimator to obtain the predicted probability of the relationship saliency level output by each sub-classifier, wherein each sub-classifier corresponds to a relationship saliency level; and determine the relationship saliency of the target pair based on the predicted probability corresponding to each relationship saliency level.

[0153] In some embodiments, the parameter adjustment module 560 is used to determine a first sub-loss function for any two nodes forming a target pair in the scene graph, based on the difference between the predicted probabilities of the relationship saliency levels output by each sub-classifier corresponding to the target pair and the labeled relationship saliency levels of the target pair; and to determine a second sub-loss function based on the difference between the predicted probabilities of the two adjacent relationship saliency levels corresponding to the target pair, wherein the more instances where the predicted probability of the higher-level relationship saliency level is greater than the predicted probability of the lower-level relationship saliency level, the larger the value of the second sub-loss function; and to determine a relationship saliency loss function based on the first and second sub-loss functions corresponding to any two nodes forming a target pair.

[0154] The electronic devices in the embodiments of this disclosure can each be implemented by various computing devices or computer systems, as described below. Figure 6 as well as Figure 7 Describe it.

[0155] Figure 6 This is a structural diagram of some embodiments of the electronic device disclosed herein. For example... Figure 6 As shown, the electronic device 60 of this embodiment includes a memory 610 and a processor 620 coupled to the memory 610. The processor 620 is configured to execute an image processing method or a scene graph generation model training method in any of the embodiments of this disclosure based on instructions stored in the memory 610.

[0156] The memory 610 may include, for example, system memory, fixed non-volatile storage media, etc. The system memory stores, for example, the operating system, application programs, boot loader, database, and other programs.

[0157] Figure 7 These are structural diagrams of some other embodiments of the electronic device disclosed herein. For example... Figure 7 As shown, the device 70 of this embodiment includes a memory 710 and a processor 720, which are similar to the memory 610 and processor 620, respectively. It may also include an input / output interface 730, a network interface 740, a storage interface 750, etc. These interfaces 730, 740, 750, and the memory 710 and processor 720 can be connected, for example, via a bus 760. The input / output interface 730 provides a connection interface for input / output devices such as a display, mouse, keyboard, and touchscreen. The network interface 740 provides a connection interface for various networked devices, such as connecting to a database server or cloud storage server. The storage interface 750 provides a connection interface for external storage devices such as SD cards and USB flash drives.

[0158] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable non-transitory storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0159] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0160] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0161] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0162] The above description is only a preferred embodiment of this disclosure and is not intended to limit this disclosure. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this disclosure should be included within the protection scope of this disclosure.

Claims

1. An image processing method, comprising: Extract one or more target pairs and information for each target pair from the image, wherein the information for each target pair includes: feature information, location information and classification information of the first target, and feature information, location information and classification information of the second target; Determine the visual relationship features of each target pair based on the information of each target pair; For each target pair, the visual relationship features of the target pair are input into each sub-classifier in the relationship saliency estimator to obtain the predicted probability of the relationship saliency level output by each sub-classifier. Based on the predicted probability corresponding to each relationship saliency level, the relationship saliency of the target pair is determined. Each sub-classifier corresponds to a relationship saliency level, and the relationship saliency of each target pair represents the saliency of the visual relationship of each target pair. Based on the feature information of each target, the visual relationship features of each target pair, and the relationship saliency of each target pair, a scene map with relationship saliency corresponding to the image is generated.

2. The image processing method according to claim 1, wherein, The step of determining the visual relationship features of each target pair based on the information of each target pair includes: For each target pair, based on the location information of the first target and the location information of the second target, a minimum rectangular region containing the first target and the second target is determined as the union region, and the intersection region of the first target and the second target is determined as the intersection region; Determine the feature information and location information of the union region, and the feature information and location information of the intersection region; Based on the information of the target pair, the feature information and location information of the union region, and the feature information and location information of the intersection region, the visual relationship features of the target pair are determined.

3. The image processing method according to claim 2, wherein, The step of determining the visual relationship features of the target pair based on the information of the target pair, the feature information and location information of the union region, and the feature information and location information of the intersection region includes: Based on the feature information of the first target, the feature information of the second target, the feature information of the union region and the feature information of the intersection region, the corresponding appearance features of the target are determined. Based on the location information of the first target, the location information of the second target, the location information of the union region, and the location information of the intersection region, the spatial features corresponding to the target pair are determined. Based on the classification information of the first target and the classification information of the second target, determine the corresponding semantic features of the target pair; Based on the appearance features, spatial features, and semantic features of the target pair, determine the visual relationship features of the target pair.

4. The image processing method according to claim 3, wherein, The step of determining the corresponding appearance features of the target based on the feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region includes: The feature information of the first target, the feature information of the second target, the feature information of the union region, and the feature information of the intersection region are respectively input into the appearance feature extractor and mapped into feature vectors of a preset dimension to obtain the appearance features of the first target, the appearance features of the second target, the appearance features of the union region, and the appearance features of the intersection region. The appearance features of the first target, the appearance features of the second target, the appearance features of the union region, and the appearance features of the intersection region are spliced ​​together to obtain the appearance features corresponding to the target pair.

5. The image processing method according to claim 2, wherein, For each region in the bounding box region of the first target, the bounding box region of the second target, the union region, and the intersection region, features of each region are extracted from the feature map of the image and used as feature information of the first target, the second target, the union region, and the intersection region, respectively.

6. The image processing method according to claim 3, wherein, The location information of the first target includes the vertex coordinates of the bounding box of the first target; the location information of the second target includes the vertex coordinates of the bounding box of the second target; the location information of the union region includes the vertex coordinates of the union region; the location information of the intersection region includes the vertex coordinates of the intersection region; determining the spatial features corresponding to the target pair based on the location information of the first target, the location information of the second target, the location information of the union region, and the location information of the intersection region includes: For each region in the bounding box region of the first target, the bounding box region of the second target, the union region, and the intersection region, determine the coordinates, width, and height of the center point; Based on the coordinates of the center point, vertex coordinates, width, height, and the width and height of the image, determine the spatial features representing the position and proportion of the region in the image; The spatial features of the bounding box region of the first target, the spatial features of the bounding box region of the second target, the spatial features of the intersection region, and the spatial features of the union region are concatenated to obtain the spatial features corresponding to the target pair.

7. The image processing method according to claim 3, wherein, The classification information includes: category name; determining the corresponding semantic features of the target based on the classification information of the first target and the classification information of the second target includes: The category name of the first target is encoded and then input into the word embedding matrix to obtain the word embedding vector of the first target; The category name of the second target is encoded and then input into the word embedding matrix to obtain the word embedding vector of the second target; The word embedding vectors of the first target and the second target are concatenated to obtain the semantic features corresponding to the target pair.

8. The image processing method according to claim 1, wherein, The step of generating a scene map with relational saliency corresponding to the image based on the feature information of each target, the visual relational features of each target pair, and the relational saliency of each target pair includes: Each target is treated as a node in the scene graph. For each node, the feature information of the node is updated based on the feature information of the node, the feature information of each neighboring node of the node, and the salience of the relationship between the node and the target pairs formed by each neighboring node. For each edge in the scene graph, the feature information of the edge is determined based on the feature information of the two nodes connected by the edge and the visual relationship features of the target pair formed by the two nodes connected by the edge. The target category of each node is determined based on the feature information of each node, and the predicate category of each edge is determined based on the feature information of each edge, so as to generate a scene graph with relational saliency corresponding to the image.

9. The image processing method according to claim 8, wherein, For each node, updating the node's feature information based on its feature information, the feature information of each of its neighboring nodes, and the salience of the relationship between the node and the target pairs formed by the node and its neighboring nodes includes: In each round of updates, for each node, a preset number of neighboring nodes are selected as associated nodes based on the salience of the relationship between the node and each of its neighboring nodes in the target pair. Based on the updated feature information of the node in the previous update process, and the updated feature information of the associated nodes of the node, the feature information of the node is updated. In the first update process, the feature information of each node is the feature information extracted from the image.

10. The image processing method according to claim 9, wherein, The step of updating the node's feature information based on the updated feature information of the node in the previous update process, and the updated feature information of the node's associated nodes, includes: For each associated node of the node, a first weight for message aggregation from the node to the associated node and a second weight for message aggregation from the associated node to the node are determined based on an attention mechanism. Based on the feature information of each associated node after the update in the previous round of updates, as well as the first weight and second weight corresponding to each associated node, the feature information of each associated node after the update in the previous round of updates is fused to obtain the fused feature information of the node in the current round of updates. The feature information of the node after the previous update and the fused feature information of the node in the current update are input into the first gated recurrent unit (GRU) to obtain the feature information of the node after the current update.

11. The image processing method according to claim 9, wherein, For each edge in the scene graph, based on the feature information of the two nodes connected by the edge and the visual relationship features of the target pair formed by the two nodes connected by the edge, the feature information of the edge is determined to include: In each round of updates, for each edge in the scene graph, based on the feature information of the two nodes connected by the edge after the update in the previous round, the fusion feature information corresponding to the edge in the current round of updates is determined. The feature information of the edge determined in the previous update process and the fused feature information of the edge in the current update process are input into the second gated recurrent unit (GRU) to obtain the feature information of the edge determined in the current update process. In the first update process, the feature information of each edge is the visual relationship feature of the target pair composed of the two nodes connected by the edge.

12. The image processing method according to claim 11, wherein, In each round of updates, for each edge in the scene graph, based on the feature information of the two nodes connected by the edge after the previous round of updates, the fused feature information corresponding to the edge in the current round of updates is determined, including: In each round of updates, for each edge in the scene graph, the feature information of the starting node and the ending node along the direction of the edge are concatenated, and the concatenated feature information is input into the fully connected layer to determine the fused feature information corresponding to the edge in this round of updates. The fully connected layer includes a non-linear activation function.

13. The image processing method according to claim 8, wherein, The step of determining the target category of each node based on the feature information of each node, and determining the predicate category of each edge based on the feature information of each edge, to generate a scene graph with relational saliency corresponding to the image, includes: Input the feature information of each node into the target classifier to obtain the first probability of each node belonging to each target category; Input the feature information of each edge into the predicate classifier to obtain the second probability of each edge belonging to each predicate category; For each triplet consisting of the target category of two nodes and the predicate category of the edge between the two nodes, the probability of the triplet is determined based on the first probability, the second probability, and the prediction probability that the relational significance of the target pair consisting of the two nodes is greater than the preset significance level. Triples are selected based on the probabilities of all triples. The target category and predicate category in the selected triples are used as the target category of the corresponding node and the predicate category of the corresponding edge, respectively. The relation saliency of the target pair formed by the two nodes is added to the corresponding edge to generate a scene graph with relation saliency.

14. The image processing method according to claim 1, further comprising at least one of the following: If the target in the image includes a person, the behavior or intention of the person is determined based on the scene diagram; If the targets in the image include people and objects, the relationship between the people and objects is determined according to the scene diagram, and the objects are sorted according to the relationship between the people and objects. Answer the visual questions based on the scene diagram; Perform cross-modal retrieval based on the scene graph; Generate descriptive text for the image based on the scene diagram.

15. A method for training a scene graph generation model, comprising: Multiple training sample images are acquired, wherein each training sample image is labeled with the target category of each target, the relation salience of each target pair in one or more target pairs, and the predicate category corresponding to each target pair, wherein the relation salience of each target pair represents the salience of the visual relation of each target pair; For each training sample image, the training sample image is input into the target detector of the scene graph generation model to extract the target pair and the information of each target pair. The information of each target pair in the sample image includes: feature information, location information and classification information of the third target, and feature information, location information and classification information of the fourth target. The information of each target pair is input into the target interaction encoder of the scene graph generation model to determine the visual relationship features of each target pair; The visual relationship features of each target pair are input into the relationship saliency estimator of the scene graph generation model to determine the relationship saliency of each target pair; The feature information of each target, the visual relationship features of each target pair, and the relationship saliency of each target pair are input into the scene graph generator of the scene graph generation model to generate a scene graph with relationship saliency corresponding to the image. Based on the target category of each node in the scene graph, the predicate category of each edge, the saliency of the relationship between any two nodes forming a target pair, the target category of each labeled target, the predicate category corresponding to each target pair, and the saliency of the relationship between each target pair, the parameters of the scene graph generation model are adjusted to complete the training.

16. The training method according to claim 15, wherein, The step of adjusting the parameters of the scene graph generation model based on the target category of each node in the scene graph, the predicate category of each edge, the saliency of the relationship between any two nodes forming a target pair, the target category of each labeled target, the predicate category corresponding to each target pair, and the saliency of the relationship between each target pair includes: Based on the difference between the target category of each node in the scene graph and the target category of each labeled target, determine the target classification loss function; Based on the difference between the predicate category of each edge in the scene graph and the predicate category of each labeled target pair, the predicate classification loss function is determined. Based on the difference between the relation saliency of any two target pairs formed by any two nodes in the scene graph and the relation saliency of each labeled target pair, determine the relation saliency loss function; Based on the target classification loss function, the predicate classification loss function, and the relation saliency loss function, determine the total loss function; The parameters of the scene graph generation model are adjusted based on the total loss function.

17. The training method according to claim 16, wherein, The step of inputting the visual relationship features of each target pair into the relationship saliency estimator of the scene graph generation model to determine the relationship saliency of each target pair includes: For each target pair, the visual relationship features of the target pair are input into each sub-classifier in the relationship saliency estimator to obtain the predicted probability of the relationship saliency level output by each sub-classifier, where each sub-classifier corresponds to a relationship saliency level; The significance of the target pair is determined based on the predicted probability corresponding to each significance level of the relationship.

18. The training method according to claim 17, wherein, The step of determining the relation saliency loss function based on the difference between the relation saliency of any two target pairs formed by any two nodes in the scene graph and the relation saliency of each labeled target pair includes: For any two nodes forming a target pair in the scene graph, the first sub-loss function is determined based on the difference between the predicted probability of the relation saliency level output by each sub-classifier corresponding to the target pair and the labeled relation saliency level of the target pair. The second sub-loss function is determined based on the difference between the predicted probabilities of the target and the corresponding two adjacent relationship salience levels. The more times the predicted probability of the higher level is greater than the predicted probability of the lower level among the two adjacent relationship salience levels, the larger the value of the second sub-loss function will be. The significance loss function is determined based on the first and second sub-loss functions corresponding to the target pair composed of any two nodes.

19. A scene graph generation model includes: A target detector is used to receive an input image and extract one or more target pairs and information of each target pair from the image, wherein the information of each target pair includes: feature information, location information and classification information of a first target, and feature information, location information and classification information of a second target; A target interaction encoder is used to determine the visual relationship features of each target pair based on the information of each target pair; A relation saliency estimator is used to determine the relation saliency of each target pair based on the visual relation features of each target pair. The relation saliency estimator includes multiple sub-classifiers. For each target pair, the visual relation features of the target pair are input into each sub-classifier to obtain the predicted probability of the relation saliency level output by each sub-classifier. The relation saliency of the target pair is determined based on the predicted probability corresponding to each relation saliency level. Each sub-classifier corresponds to a relation saliency level, and the relation saliency of each target pair represents the saliency of the visual relation of each target pair. A scene graph generator is used to generate a scene graph with relational saliency corresponding to the image based on the feature information of each target, the visual relational features of each target pair, and the relational saliency of each target pair.

20. An image processing apparatus, comprising: The target detection module is used to extract one or more target pairs and information of each target pair from an image, wherein the information of each target pair includes: feature information, location information and classification information of a first target, and feature information, location information and classification information of a second target; The feature determination module is used to determine the visual relationship features of each target pair based on the information of each target pair; The relation saliency determination module is used to input the visual relation features of each target pair into each sub-classifier in the relation saliency estimator for each target pair, obtain the predicted probability of the relation saliency level output by each sub-classifier, and determine the relation saliency of the target pair based on the predicted probability corresponding to each relation saliency level. Each sub-classifier corresponds to a relation saliency level, and the relation saliency of each target pair represents the saliency of the visual relation of each target pair. The generation module is used to generate a scene map with relational saliency corresponding to the image based on the feature information of each target, the visual relational features of each target pair, and the relational saliency of each target pair.

21. A training device for a scene graph generation model, comprising: The acquisition module is used to acquire multiple training sample images, wherein each training sample image is labeled with the target category of each target, the relation saliency of each target pair in one or more target pairs, and the predicate category corresponding to each target pair; The target detection module is used to input the training sample image into the target detector of the scene graph generation model for each training sample image, and extract the target pair and the information of each target pair. The information of each target pair in the sample image includes: feature information, location information and classification information of the third target, and feature information, location information and classification information of the fourth target. The feature determination module is used to input the information of each target pair into the target interaction encoder of the scene graph generation model to determine the visual relationship features of each target pair; The relation saliency determination module is used to input the visual relation features of each target pair into the relation saliency estimator of the scene graph generation model to determine the relation saliency of each target pair, wherein the relation saliency of each target pair represents the saliency of the visual relation of each target pair; The generation module is used to input the feature information of each target, the visual relationship features of each target pair, and the relationship saliency of each target pair into the scene graph generator of the scene graph generation model, and generate a scene graph with relationship saliency corresponding to the image. The parameter adjustment module is used to adjust the parameters of the scene graph generation model based on the target category of each node in the scene graph, the predicate category of each edge, the saliency of the relationship between any two nodes forming a target pair, the target category of each labeled target, the predicate category corresponding to each target pair, and the saliency of the relationship between each target pair, in order to complete the training.

22. An electronic device, comprising: processor; as well as A memory coupled to the processor is used to store instructions that, when executed by the processor, cause the processor to perform the image processing method as described in any one of claims 1-14 or the training method for the scene graph generation model as described in any one of claims 15-18.

23. A non-transitory computer-readable storage medium having a computer program stored thereon, wherein, When executed by a processor, the program implements the steps of the image processing method according to any one of claims 1-14 or the training method of the scene graph generation model according to any one of claims 15-18.

Images