Method and apparatus for training a relation identification model, performing graph analysis

Abstract

Embodiments of the present specification provide a method for training a relationship identification model, a method and device for performing graph analysis using the model. The training method comprises dividing a total set of relationship categories into K groups, and correspondingly setting K classifiers, wherein the Kth classifier corresponds to a Kth classification space composed of the relationship categories in the Kth group superimposed on the classification space corresponding to the previous classifier. Then, from the entire set of picture samples, K sample subsets are obtained by sampling for the K classifiers respectively, and each is input into the corresponding classifier to obtain the relationship category prediction result between the objects in the picture. Next, a first loss is determined based on the classification loss of each of the K classifiers, and a second loss is determined based on the similarity between the relationship category prediction results of different classifiers in the K classifiers for the same pair of objects. Thus, the relationship identification model is updated according to the first loss and the second loss.

Description

Technical Field

[0001] This specification relates to image data processing in artificial intelligence in one or more embodiments, and more particularly to a method and apparatus for training a relation recognition model and performing graph analysis based on the relation recognition model. Background Technology

[0002] Image recognition and analysis have been widely applied in various technological scenarios. To better understand the semantic information in images, several technical solutions have been proposed. These solutions go beyond simply identifying objects in an image; they further identify the relationships between objects to better interpret the image's content. This is also known as scene graph generation. The task of scene graph generation is to detect all object categories and pairwise relationships in a given image, using the detected objects as nodes and the relationships between paired objects as edges, to further organize them into a semantic graph structure that highly summarizes the visual information in the image.

[0003] Scene graph generation is a midstream computer vision task that bridges lower-level object recognition tasks, such as object detection and semantic recognition, while simultaneously enabling higher-level vision-semantic tasks, such as visual question answering and image captioning. Therefore, scene graph generation plays a crucial connecting role, organically combining image detection and image understanding, and is a significant contributor to achieving more advanced image understanding.

[0004] Therefore, we hope to find an improved solution that can enhance the accuracy of object relationship recognition in scene graphs. Summary of the Invention

[0005] In view of the above problems, this specification provides a method for training a relationship recognition model, a graph analysis method, and a corresponding apparatus to improve the accuracy of object relationship recognition.

[0006] According to a first aspect, a method for training a relation recognition model is provided, the relation recognition model comprising an encoding network and a classification network, wherein the encoding network is used to identify and encode multiple objects contained in an image; and the classification network is used to identify relation categories between the multiple objects from a preset relation category set, the method comprising:

[0007] The set of relation categories is divided into K groups, and K classifiers are set accordingly to form the classification network. The classification space of the first classifier in the K classifiers corresponds to the relation category in the first group. The classification space of any subsequent k-th classifier is formed by superimposing the classification space of the previous classifier with the relation category in the k-th group.

[0008] From the complete set of image samples, samples are taken from each of the K classifiers to obtain K subsets of samples;

[0009] The sample images in each sample subset are input into the corresponding classifiers via the coding network to obtain the category prediction results of the relationship between objects in the sample images;

[0010] The first loss is determined based on the classification loss of each of the K classifiers, wherein the classification loss of any classifier is determined based on the relation category prediction result obtained by the classifier and the corresponding category label.

[0011] The second loss is determined based on the similarity between the prediction results of the relationship category of the same pair of objects by different classifiers among K classifiers.

[0012] The total prediction loss is determined based on the first loss and the second loss, and the encoding network and K classifiers are updated based on the total prediction loss.

[0013] According to the second aspect, an image analysis method is provided, including:

[0014] Obtain a relation recognition model trained according to the method of the first aspect, which includes an encoding network and a classification network, wherein the classification network includes K classifiers;

[0015] The target image to be identified is input into the Kth classifier via the encoding network to predict the relationship categories between multiple objects contained in the target image.

[0016] According to a third aspect, a method for analyzing images using a relation recognition model is provided, the relation recognition model comprising an encoding network and a classification network, the encoding network comprising a detection subnetwork and a first encoder; the method includes:

[0017] The target image is input into the detection sub-network to obtain the image features, bounding box information, and initial object categories of several objects contained in the target image;

[0018] The first encoder performs attention-based encoding based on the image features, bounding box information, and initial object categories to obtain the encoded features of each object.

[0019] The encoded features of each object are input into the classification network to identify the relationship categories between the objects;

[0020] The first encoder includes several hybrid attention layers, each including a self-attention module and a cross-attention module set for each of the two types of features; the attention-based encoding operation includes, in a single hybrid attention layer:

[0021] By using the self-attention module corresponding to any one of the two types of features, the input feature of that type is transformed based on self-attention to obtain the first transformed feature.

[0022] By using the cross-attention module corresponding to any one of the two types of features, and based on the attention of the other type of feature relative to the arbitrary one type of feature, the two types of features are combined and transformed to obtain the second transformed feature.

[0023] The first transformation feature and the second transformation feature are combined to form the output feature of any one of the categories.

[0024] The two types of features input to the first encoder include a first type of features obtained based on the image features and bounding box information of each first object, and a second type of features obtained based on the initial object category of each first object.

[0025] According to the fourth aspect, a method for analyzing images using a relation recognition model is provided, wherein the relation recognition model includes an encoding network and a classification network, and the encoding network includes a detection subnetwork, a first encoder, a first decoder, and a second encoder; the method includes:

[0026] The target image is input into the detection sub-network to obtain the image features, bounding box information, and initial object categories of several objects contained in the target image;

[0027] The first encoder performs attention-based encoding based on the image features, bounding box information, and initial object category of each object to obtain the first encoded features of each object.

[0028] The modified object category of each object is obtained by using the first decoder based on the first encoded features of each object.

[0029] The second encoder performs attention-based encoding based on the image features, first encoded features, and corrected object categories of each object to obtain the second encoded features of each object.

[0030] The second encoded features of each object are input into the classification network to identify the categories of relationships between objects;

[0031] Wherein, the first encoder and / or the second encoder include several hybrid attention layers, each hybrid attention layer including a self-attention module and a cross-attention module respectively set for each of the two types of features; the attention-based encoding operation includes, in a single hybrid attention layer:

[0032] By using the self-attention module corresponding to any one of the two types of features, the input feature of that type is transformed based on self-attention to obtain the first transformed feature.

[0033] By using the cross-attention module corresponding to any one of the two types of features, and based on the attention of the other type of feature relative to the arbitrary one type of feature, the two types of features are combined and transformed to obtain the second transformed feature.

[0034] The first transformation feature and the second transformation feature are combined to form the output feature of any one of the categories.

[0035] The two types of features input to the first encoder include a first type of features obtained based on the image features and bounding box information of each first object, and a second type of features obtained based on the initial object category of each first object.

[0036] The two types of features input to the second encoder include a first type of features obtained based on the image features and first coding features of each first object, and a second type of features obtained based on the modified object category of each first object.

[0037] According to a fifth aspect, an apparatus for training a relation recognition model is provided, the relation recognition model including an encoding network and a classification network, the encoding network being used to recognize and encode multiple objects contained in an image; the classification network being used to recognize the relation categories between the multiple objects from a preset relation category set, the apparatus comprising:

[0038] The grouping unit is configured to divide the relation category set into K groups and set K classifiers to form the classification network. The classification space of the first classifier in the K classifiers corresponds to the relation category in the first group, and the k-th classification space of any subsequent k-th classifier is formed by superimposing the classification space of the previous classifier with the relation category in the k-th group.

[0039] The sampling unit is configured to sample from the entire set of image samples, targeting K classifiers respectively, to obtain K sample subsets;

[0040] The prediction unit is configured to input the sample images from each sample subset into the corresponding classifiers via the coding network to obtain the prediction results of the relationship categories between objects in the sample images;

[0041] The first loss determination unit is configured to determine the first loss based on the classification loss of each of the K classifiers, wherein the classification loss of any classifier is determined based on the relationship category prediction result obtained by the classifier and the corresponding category label.

[0042] The second loss determination unit is configured to determine the second loss based on the similarity between the prediction results of different classifiers for the relationship category of the same pair of objects among the K classifiers.

[0043] The update unit is configured to determine the total prediction loss based on the first loss and the second loss, and update the encoding network and K classifiers based on the total prediction loss.

[0044] According to a sixth aspect, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed in a computer, causes the computer to perform the method described in the first aspect.

[0045] According to a seventh aspect, a computing device is provided, including a memory and a processor, characterized in that the memory stores executable code, and when the processor executes the executable code, it implements the method described in the first aspect.

[0046] In the embodiments of this specification, during the training of the relationship recognition model, the training dataset is divided into several relatively balanced subsets, and multiple classifiers are introduced to learn from each of these balanced subsets. Then, through knowledge accumulation and transfer among these classifiers, the training of multiple classifiers is carried out collaboratively, promoting the unbiased prediction of the final relationship. Furthermore, a hybrid attention layer is proposed to be introduced into the encoding network of the relationship recognition model, thereby better fusing visual and textual features, improving feature representation capabilities, and ultimately enhancing relationship recognition performance. Attached Figure Description

[0047] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0048] Figure 1 This illustrates the process of generating the scene graph;

[0049] Figure 2 The overall architecture of a relationship identification model according to one embodiment is shown;

[0050] Figure 3 This shows the distribution of each relation category in a typical training dataset;

[0051] Figure 4 This diagram illustrates a method for training a relation recognition model according to one embodiment.

[0052] Figure 5 This illustrates K relation category groupings formed in one embodiment;

[0053] Figure 6 A schematic diagram illustrating the structure and processing of a relationship recognition model according to one embodiment is shown.

[0054] Figure 7 This diagram illustrates the processing flow of a hybrid attention layer.

[0055] Figure 8 An apparatus for training a relation recognition model according to one embodiment is shown. Detailed Implementation

[0056] The solution provided in this specification will now be described with reference to the accompanying drawings.

[0057] Figure 1 This diagram illustrates the process of generating a scene graph. As shown, given an image, the scene graph generation task includes first detecting all objects in the image and identifying their categories. In the left image, each object is bounded by a border and labeled with its category, such as "dog," "ear," and "nose." Then, the relationships between the objects are identified. Using the detected objects as nodes and the relationships between them as edges, the scene graph on the right is generated. As you can see, in this scene graph, the nodes include the previously labeled objects, such as "dog," "ear," and "nose"; the edges between the nodes represent the relationships between the objects in the graph, such as "lying on," "has," etc. More specifically, for example, the relationship between the object "dog" and the object "bed" is "lying on," and the relationship between the object "dog" and the object "ear" is "has."

[0058] As can be understood, the core of the above scene graph generation process is identifying the relationships between objects, which is essentially a classification process. Therefore, a relationship recognition model can be built based on classifying the categories of relationships between objects.

[0059] Figure 2 The overall architecture of a relationship identification model according to one embodiment is shown. Figure 2 As shown, the relationship recognition model may include an encoding network 100 and a classification network 200. The encoding network 100 is used to identify multiple objects contained in an image and encode them, obtaining the encoded features of each object. The classification network 200, based at least on the encoded features of each object, identifies the relationship categories between the objects from a set of relationship categories. Such a relationship recognition model needs to be trained using a training dataset consisting of labeled images. These labels include not only annotations of object categories but also annotations of the relationship categories between objects.

[0060] The inventors discovered that the distribution of object relationship categories in the training dataset used to train the relationship recognition model is extremely biased, and it faces a serious long-tail distribution effect. Figure 3 This shows the distribution of each relation category in a typical training dataset. For example... Figure 3 As shown, a small number of head relation categories occupy the majority of the training samples. For example, "on" (ranked first) accounts for 31.56% of the entire training sample, while "flyingin" (ranked 50th) accounts for only 0.005%. This causes the model's convergence process to be severely biased towards head categories during training, resulting in predictions that only cover a limited number of head relation categories and ignore a large number of tail relation categories. Furthermore, compared to head relation categories (such as "on," "has," and "near"), tail relation categories (such as "riding," "eating," and "pulling") contain more information, and the latter are the ideal predictions because they provide more valuable prior input to the upstream visual understanding task. Therefore, the long-tail effect of relation category distribution in the training dataset poses a challenge to the training of relation recognition models and affects the performance of the trained models.

[0061] Therefore, in the embodiments of this specification, a technical concept of grouped collaborative training using multiple classifiers is proposed. According to this technical concept, the training dataset is divided into several relatively balanced subsets, and multiple classifiers are introduced to learn from each of these balanced subsets. Then, through knowledge accumulation and transfer among these classifiers, the unbiased prediction of the final relationship is collaboratively promoted. The specific implementation process of this technical concept is described below.

[0062] Figure 4 A flowchart illustrating a method for training a relation recognition model according to one embodiment is shown. This method can be executed by any computing unit, platform, server, device, etc., with computational and processing capabilities. The structure of the relation recognition model to be trained is as follows: Figure 2 As shown, this includes an encoding network and a classification network. The encoding network is used to identify and encode multiple objects contained in an image; the classification network is used to identify the relationship categories between multiple objects from a preset set of relationship categories. Figure 4 As shown, the training method may include the steps described in detail below.

[0063] First, in step 41, the relation category set is divided into K groups, and K classifiers are set up to form a classification network. This step aims to divide the imbalanced dataset into multiple relatively balanced relation category groups and set up multiple corresponding classifiers, thereby facilitating the subsequent formation of relatively balanced training data subsets.

[0064] like Figure 3 As shown, although the distribution of relation categories in the dataset exhibits a severe long-tail effect overall, relatively balanced ranges can still be found locally. Therefore, with a certain balance metric as the goal, the total set of relation categories can be divided into K groups, so that the distribution of relation categories within each group is relatively balanced.

[0065] Specifically, in one embodiment, in order to group relation categories, the estimated sample size for each relation category in the relation category set is first obtained, for example... Figure 3 As shown. In one example, this sample size estimate can be obtained by statistically analyzing all relation categories contained in the complete set of image samples to be used subsequently. In another example, this sample size estimate can be obtained from other prior distribution data.

[0066] Based on the estimated sample size values ​​for each relation category obtained above, the relation categories can be ranked. For example... Figure 3 The diagram illustrates the sorting of relation categories by estimated sample size, from largest to smallest. Then, K groups are formed based on this sorting, such that the sample size difference index between the relation category with the largest sample size and the relation category with the smallest sample size in each group is below a preset threshold μ. This sample size difference index can be the difference in estimated sample size values, or an index such as the difference ratio.

[0067] In one example, the grouping of relation categories can be achieved through the following process. Specifically, the relation categories are iterated sequentially according to the sorted sequence. For the current relation category i, it is determined whether the difference in sample size between it and the first relation category in the newly formed group k is lower than a preset threshold μ. If it is lower than the preset threshold μ, it means that the current relation category i is relatively balanced with the existing relation categories in group k, and the current relation category i can be added to group k. If the difference in sample size between the current relation category i and the first relation category in group k is not lower than the preset threshold μ, it means that the sample size of the current relation category i is no longer balanced with the existing relation categories in group k. In this case, a new group k+1 is generated, and the current relation category i is taken as the first relation category in the newly generated group k+1, and the newly generated group k+1 is taken as the newly formed group. Then, the next relation category is traversed. In this way, each relation category is assigned to its corresponding group.

[0068] In another example, the sample size estimate can be used as a feature to cluster relation categories with similar sample size estimates into a single cluster, thereby dividing the total set of relation categories into K groups.

[0069] Figure 5 This illustrates K relation category groupings formed in one embodiment. As can be seen, Figure 5 Relationship category grouping is based on Figure 3 The distribution of relational categories was carried out. Figure 5In the model, the 50 relation categories are divided into 5 groups. Group 1 includes the 4 relation categories ranked 1-4 in sample size estimates (e.g., c1-c4); Group 2 includes the 6 relation categories ranked 5-10 in sample size estimates (e.g., c5-c10); the other groups are shown in the figure. It can be seen that within each group, the sample size estimates are relatively balanced. More specifically, in... Figure 5 In this study, the balance index was set to be less than 4 times the difference in sample size between the relation category with the largest sample size and the relation category with the smallest sample size.

[0070] Based on dividing the set of relation categories into K groups, K classifiers are set accordingly. The K classifiers are set using an incremental accumulation method. Specifically, the classification space of the first classifier, that is, the subset of categories it identifies, corresponds to the relation categories in the first group. Any subsequent classifier k corresponds to the k-th classification space, which is formed by adding the classification space of the previous classifier (i.e., the (k-1)-th classifier) ​​to the relation categories in the k-th group. This accumulation continues until the K-th classifier, whose corresponding classification space is the total set of relation categories.

[0071] continue Figure 5 Examples. In Figure 5 In this example, the total set of 50 relation categories is divided into K=5 groups, corresponding to 5 classifiers. The classification space of the first classifier is the relation categories in the first group; that is, classifier 1 is used to classify the relation category subset c1-c4. The classification space of the second classifier is the classification space of the first classifier plus the relation categories in the second group; that is, classifier 2 is used to classify relation categories c1-c4 plus relation categories c5-c10, i.e., the relation category subset c1-c10. And so on. Finally, the first classifier, classifier 5, is used to classify all 50 relation categories.

[0072] Based on the setup of K groups and K classifiers, in step 42, samples are taken from the entire set of image samples for each of the K classifiers to obtain K sample subsets.

[0073] It is important to understand that since the sample size of relation categories in each group is relatively balanced, sampling for each classifier's classification space can ensure that the sample size for the corresponding classification space in each sample subset is relatively balanced.

[0074] Furthermore, considering that the classification space of subsequent classifiers actually includes relation categories from multiple groups, and the balance of relation categories in these groups is somewhat reduced, certain strategies can be employed during the sampling process to impose constraints on the sampling process. This ensures that the sampling distribution of any k-th sample subset for each relation category in the k-th classification space conforms to predetermined conditions related to the balanced distribution. These predetermined conditions may include, for example, that the difference index between the relation category with the largest number of samples and the relation category with the smallest number of samples is below a certain threshold; or that the proportion of the number of samples of the head relation category relative to the entire sample subset does not exceed a set threshold; or that the sampling distribution of each relation category in the classification space satisfies a preset distribution curve, and so on.

[0075] In one specific embodiment, a partial downsampling strategy can be employed to balance the sampling distribution for a classifier. Specifically, for any k-th classifier, a target value T is determined based on the estimated sample size of each relation category in the k-th classification space. This target value T can be the average or median of the estimated sample size of each relation category.

[0076] Then, based on the target value T, the sampling rate for each relation category is determined. Specifically, for relation categories whose estimated sample size is greater than the target value T, their sampling rate is set to less than 1; for relation categories whose estimated sample size is less than or equal to the target value T, their sampling rate is set to 1. Then, sampling is performed for each relation category according to the sampling rate for each relation category. This means that rich relation categories with estimated sample size greater than the target value are downsampled, while relation categories with estimated sample size less than or equal to the target value are sampled normally, thereby balancing the sampling distribution for a classifier.

[0077] In a specific example, the sampling rate for any classifier k in a partial downsampling strategy can be set as follows:

[0078]

[0079] Where, φ j For classifier k, the relation category c j The sampling rate, T is the target value in the aforementioned k-th classification space, count(c j ) represents the relation category c j The estimated sample size. According to equation (1) above, for relation categories whose estimated sample size is greater than the target value T (T <count(c j In the case of a target value T, the sampling rate can be set as the ratio of the target value T to the estimated sample size of the relation category. This ratio is obviously less than 1, thus achieving downsampling.

[0080] When sampling for each relation category based on the determined sampling rate, a random number comparison method can be used to ensure sampling is performed according to the sampling rate. Specifically, for relation category c... j When sampling, categories c that conform to this relationship can be obtained from the entire set of image samples. j An instance is a specific element in an image that matches the category c of the relation. j Given an object pair (O1, O2), generate a positive random number less than 1, meaning the random number is between (0, 1). If this positive random number is less than the sampling rate φ of the relation category... j Sample this instance as relation category φ j The corresponding sample; otherwise, the instance is discarded.

[0081] continue Figure 5 Examples will be described. As mentioned earlier, regarding... Figure 5 Five classifiers are set up for grouping. The classification space of classifier 1 includes c1-c4. According to... Figure 5 The sample size estimates for the example are distributed such that the target value for classifier 1 should lie between the sample size estimates for relation categories c2 and c3. Thus, the sample size estimates for relation categories c1 and c2 are both greater than the target value. Assuming the sampling rate is calculated using formula (1), for example, the sampling rate for category c1 is assumed to be 0.4, the sampling rate for category c2 is assumed to be 0.8, and the sampling rates for categories c3 and c4 are both 1.

[0082] When sampling for class c1, for each instance that matches the c1 relation, a random number between (0,1) is generated and compared to the sampling rate of 0.4. If the random number is less than 0.4, it is considered a sample of class c1; otherwise, the instance is discarded. Since random numbers are generally located between (0,1) with equal probability, the probability that the random number is less than 0.4 is 40%, which is equal to the sampling rate. This achieves sampling of instances matching c1 with probability based on the sampling rate. The sampling process for classes c2-c4 is similar. This achieves sampling for classifier 1.

[0083] Similarly, sampling can be performed on other classifiers. For example, for classifier 2, its classification space is c1-c10. Assume its target value is between c4 and c5. Then, c1-c4 needs to be downsampled at a sampling rate less than 1, while c5-c10 needs to be sampled normally.

[0084] By sampling K classifiers using the methods described above, we can obtain K corresponding sample subsets. In each sample subset, the sampling volume (number of instances) of relation categories is relatively evenly distributed.

[0085] Back Figure 4 Next, in step 43, the sample images from each sample subset are input into the corresponding classifiers via the encoding network to obtain the prediction results of the relationship categories between objects in the sample images.

[0086] Specifically, for a sample image in the sample subset k, it is input into the encoding network. The encoding network identifies and encodes the objects in the sample image, obtaining the encoded features of each object. The instances sampled for classifier k, i.e., object pairs (o... i ,o j The encoded features of the object pair are input into the k-th classifier to obtain the relationship category prediction result. The above operation is performed on each instance of each sample image in each sample subset, so that each classifier predicts the relationship category for the input object pair.

[0087] To better achieve collaborative learning and optimization among multiple classifiers, the total prediction loss is determined by combining the relation category prediction results output by each classifier from two aspects: parallel optimization of classifiers and collaborative knowledge distillation.

[0088] Accordingly, in step 44, a first loss is determined based on the classification losses of the K classifiers, wherein the classification loss of any classifier is determined based on the relation category prediction result obtained by that classifier and the corresponding category label. This first loss is the loss determined according to the classifier parallel optimization (PCO) scheme.

[0089] Specifically, the classification loss of the k-th classifier can be determined based on the prediction result of the relation category of any k-th classifier on the instance samples in the k-th sample subset and the corresponding category label; the classification loss of each of the k classifiers is summed to obtain the first loss mentioned above.

[0090] In one example, the first loss can be determined according to the following formula (2):

[0091]

[0092] Among them, D k It is the k-th sample subset, |D k | represents the number of instances in the k-th sample subset; L CE It is the cross-entropy loss function; y ij Represents instance sample (o) i ,o j The corresponding relationship category label, For the k-th classifier, for instance samples (o i ,o jThe relationship category prediction results obtained are generally... The form is a probability distribution vector, where each element represents an instance sample (o). i ,o j The probability that a relation belongs to a category in the k-th classification space.

[0093] According to equation (2), for a single instance (o) processed by classifier k, i ,o j The cross-entropy loss function L can be used. CE Calculate the prediction loss for the given instance sample. Then, sum the prediction losses of all instances in the k-th sample subset and divide by the number of instances to obtain the classification loss of the k-th classifier. Summing the classification losses of each of the K classifiers (i.e., summing k from 1 to K) yields the first loss L. PCO .

[0094] Furthermore, in step 45, a second loss is determined based on the similarity between the prediction results of different classifiers for the relationship category of the same pair of objects among the K classifiers. This second loss is used to establish a knowledge transfer mechanism and knowledge distillation between classifiers, so that later classifiers can learn the prediction capabilities of earlier classifiers. Therefore, it is called the loss measure for Collaborative Knowledge Distillation (CKD).

[0095] To determine the CKD loss, for an instance sample, i.e., an object pair (o i ,o j ), obtain the first prediction result of the first classifier m for the instance in the first classification space. The first prediction result is a probability distribution vector with the same dimension dm as the first classification space. Furthermore, a second prediction result is obtained from the second classifier n for this instance in the second classification space. The dimension of the second prediction result is the same as the dimension dn of the second classification space. Assuming the second classifier n is the later classifier, the second classification space contains the first classification space, and the dimension dn of the second classification space is greater than that of the first classification space dm, i.e., dn > dm. Therefore, the dimension of the second prediction result is greater than that of the first prediction result.

[0096] Therefore, we can proceed from the second prediction result. In the first prediction result, the portion corresponding to the dimension of the first prediction result is extracted to obtain the second truncation result. Its dimension is the same as the first prediction result. It can be understood that the part of the second prediction result corresponding to the dimension of the first prediction result corresponds to the prediction of the second classifier n for the relation categories contained in the first classification space, and it is comparable to the first prediction result in both dimension and meaning. Therefore, the first prediction result can be determined... Second truncation result The similarity between the vectors is used to determine the second loss mentioned above. In different embodiments, the similarity can be determined by methods such as cosine similarity, KL divergence, and vector dot product.

[0097] continue Figure 5 Let's illustrate this with an example. Assume the first classifier is classifier 1, and the first prediction result is... This is a 4-dimensional probability distribution vector, indicating the probability distribution of an instance belonging to the c1-c4 relationship category; the second classifier is assumed to be classifier 2, and the second prediction result... The vector is a 10-dimensional probability distribution, indicating the probability distribution of the instance belonging to relation categories c1-c10. The first four dimensions of the second prediction result indicate the probability distribution predicted by classifier 2 for the instance belonging to c1-c4. Therefore, the first four dimensions of the second prediction result can be extracted as the second truncation result. Based on the first prediction result and the second truncation result, the similarity of the prediction results of classifier 2 and classifier 1 for categories c1-c4 is determined.

[0098] In one embodiment, the object pairs (o) constituting the above instance samples i ,o j ), which are object pairs in the common sample images of the first sample subset corresponding to the first classifier m and the second sample subset corresponding to the second classifier n. Thus, the first and second prediction results can be read and reused from the relation category prediction results output by each classifier in step 43. In another embodiment, the above object pairs (o i ,o j ) is an instance in the sample subset corresponding to a certain classifier, so the corresponding prediction result can be read from the relation category prediction result in step 43. In order to determine the second loss, it can be manually input into another classifier whose classification space contains its label category to obtain another prediction result, which can be used as the first prediction result and the second prediction result mentioned above, respectively.

[0099] In a specific example, the second loss L CKD It can be determined according to the following formula (3):

[0100]

[0101] Where Q is the set of distillation learning pairs consisting of classifier m and classifier n; The classifier m is for object pairs (o) i ,o j The first prediction result obtained, L is the second truncation result obtained after truncating the second prediction result of classifier n. KL It is the KL divergence loss function, and its specific form is shown in the following formula (4).

[0102]

[0103] In different embodiments, the above set Q can be obtained in different ways, for example, by combining the K classifiers in pairs, or by combining each classifier with its next classifier, or by combining the Kth classifier with each of the preceding K-1 classifiers, and so on. In other embodiments, other similarity determination functions can be used to replace the KL divergence loss function in equation (3), which is not limited here.

[0104] Based on the determination of the first loss and the second loss, in step 46, the total prediction loss is determined according to the first loss and the second loss, and the encoding network and K classifiers are updated according to the total prediction loss.

[0105] In one embodiment, the first loss L can be adjusted according to a preset weighting factor. PCO Second loss L CKD The weighted sum is used to obtain the total prediction loss. Specifically, as shown in the following formula (5), the total prediction loss L of Group Collaborative Learning (GCL) is obtained. GCL :

[0106]

[0107] Where α is a preset weighting factor.

[0108] As can be seen, the total predicted loss is a combination of the PCO loss and the CKD loss. The first loss L is determined based on the PCO scheme. PCO The aim is to promote the convergence of the last classifier by calculating the loss of all classifiers and applying gradient descent. The PCO approach can be viewed as a weak constraint strategy on the subsequent classifiers, that is, by aggregating the classification losses of all classifiers, it implicitly improves the prediction performance of the last classifier.

[0109] The second loss L determined according to the CKD method CKDThis approach establishes a knowledge transfer and distillation mechanism, allowing subsequent classifiers to learn from the predictions of earlier classifiers. Because earlier classifiers have smaller classification spaces, a smaller range of predicted categories, and a larger sample size, they can learn strong predictive performance for a limited range of relation categories. Subsequent classifiers can improve their predictive ability for top-ranked relation categories by learning from the predictions of earlier classifiers. This can be viewed as a strong constraint strategy, forcing later classifiers to learn from the predictive behavior of earlier classifiers, thus improving their unbiased predictive performance. This ensures that the final classifier has strong predictive ability for both top-ranked relation categories and long-tail relation categories.

[0110] In summary, by grouping the relation category set and setting multiple classifiers, the multiple classifiers can learn collaboratively, overcoming or mitigating the adverse effects of the long-tail effect of sample distribution bias on the training of the relation category recognition model. Through knowledge accumulation and transfer among classifiers, the unbiasedness of the final relation prediction is promoted in a collaborative manner, thereby improving prediction performance.

[0111] It is important to understand that among the K classifiers trained according to the above embodiment, the Kth classifier is a classifier for the entire classification space, capable of recognizing all relation categories in the total set of relation categories. The other K-1 classifiers are essentially there to assist in the training of the Kth classifier. Therefore, after the training process is complete, when using the relation recognition model for image analysis, only the Kth classifier needs to be used.

[0112] Specifically, in the image analysis process, the target image to be identified can be input into the Kth classifier via an encoding network. The Kth classifier then classifies and predicts the relationship categories between multiple objects contained in the target image. Furthermore, in one embodiment, a scene graph can be generated based on the recognition results of the relationship recognition model. Specifically, a scene graph corresponding to the target image can be generated by using multiple objects in the image as multiple nodes and the identified relationship categories as edges. For example... Figure 1 As shown on the right.

[0113] On the other hand, the inventors discovered that the process of relation category recognition involves information from multiple modalities. One modality is visual feature information, including image features of the object and border position features; another modality is textual feature information, including textual descriptions of the identified object category, for example... Figure 1 The image shows a dog, ears, etc. Multiple modalities of information need to be fused together to jointly determine the relationship categories between objects. To improve the predictive performance of the relationship recognition model, an improvement is proposed from the perspective of promoting modal fusion.

[0114] Figure 6 A schematic diagram illustrating the structure and processing of a relationship recognition model according to one embodiment is shown. (As previously mentioned...) Figure 2 The relationship recognition model generally includes an encoding network and a classification network. The encoding network is used to identify and encode multiple objects contained in an image; the classification network identifies the relationship categories between objects based at least on the encoded features of each object. Figure 6 The structure of the coding network is shown in more detail. For example... Figure 6 As shown, the encoding network includes a detection subnetwork, a first encoder, a first decoder, and a second encoder, and its encoding process is as follows.

[0115] The detection subnetwork can be a pre-trained object detection model, such as Mask R-CNN or Faster CNN. When an image is input into the detection subnetwork, it identifies the objects in the image and outputs the image features v, bounding box information s, and initial object category l of the objects contained in the image.

[0116] Then, in the first encoder, attention-based encoding is performed based on the image features v, bounding box information s, and initial object category l of the above objects to obtain the first encoded features x of each object. These first encoded features x are more refined and comprehensive object features obtained based on the image features v, bounding box information s, and initial object category l.

[0117] Next, the first decoder obtains the corrected object category l' for each object based on the first encoded feature x of each object. Since the first encoded feature x is a more comprehensive feature obtained by fusing multiple features, the object category l' output by the first decoder based on the first encoded feature x may be different from the original object category l output by the detection sub-network, and can be used as a correction to the original object category.

[0118] Next, in the second encoder, an attention-based encoding operation is performed based on the image features v of each object, the first encoded features x output by the first encoder, and the corrected object category l' output by the first decoder, to obtain the second encoded features x' of each object. Therefore, the second encoded features x are a further refined and comprehensive representation of the object features.

[0119] Therefore, the second encoded features x' of each object can be input into the classification network, such as the corresponding classifier among the K classifiers mentioned above, to identify the relationship categories between objects.

[0120] To better fuse features from different modalities, in one embodiment, several hybrid attention layers are set in the first encoder and / or the second encoder. Self-attention processing and cross-attention processing are introduced through the hybrid attention layers to achieve better fusion results.

[0121] Figure 7 A schematic diagram of the processing flow of a hybrid attention layer is shown. As illustrated, this arbitrary l-th hybrid attention layer includes a self-attention module SA and a cross-attention module CA set for each of the two types of features (or two modal features). Specifically, the l-th hybrid attention layer includes a first self-attention module SA and a first cross-attention module CA set for the first type of features, and a second self-attention module SA and a second cross-attention module CA set for the second type of features. The l-th hybrid attention layer processes the first type of feature X as input. (I) Second type feature Y (I) Attention-based fusion encoding is performed. The specific processing steps are as follows.

[0122] In the first self-attention module, the first type of feature X input to the hybrid attention layer is... (I) Perform a self-attention-based transformation to obtain the first self-transformation feature SA(X). (I) In the first cross-attention module, based on the second type of feature Y input to the hybrid attention layer... (I) Compared to the first type of feature X (I) Attention to the first type of feature X (I) Second type feature Y (I) Perform a combination transformation to obtain the first cross-transformation feature CA(X). (I) ,Y (I) The first self-transformation feature and the first cross-transformation feature are combined to form the first type of feature X of the hybrid attention output. (I+1) In a specific example, the first type of feature output is represented as:

[0123] X (I+1) =SA(X) (I) )+CA(X (I) ,Y (I) (6)

[0124] It is understandable that the first type of feature X (I+1) That is, the first type of feature that serves as the input to the next hybrid attention layer (the (l+1)th layer).

[0125] Correspondingly, in the second self-attention module, the second type of feature Y input to the hybrid attention layer is... (I) Perform a self-attention-based transformation to obtain the second self-transformation feature SA(Y). (I)In the second cross-attention module, based on the first type of feature X input to the hybrid attention layer... (I) Compared to the second type of feature Y (I) Attention to the second type of feature Y (I) and the first type of feature X (I) Perform a combined transformation to obtain the second cross-transformation feature CA(Y). (I) ,X (I) The second self-transformation feature and the second cross-transformation feature are combined to form the second type of feature Y of the hybrid attention output. (I+1) In a specific example, the output of the second type of feature is represented as:

[0126] Y (I+1) =SA(Y) (I) )+CA(Y (I) ,X (I) (7)

[0127] It is understandable that the second type of feature Y (I+1) That is, the second type of feature that serves as the input to the next hybrid attention layer (layer l+1).

[0128] The self-attention-based transformation (SA) described above employs a known self-attention mechanism. Specifically, for the input feature matrix, an attention transformation function determined by parameter matrices Q, K, and V is applied to obtain the transformed feature matrix. The cross-attention-based combined transformation (CA) employs a known cross-attention mechanism. Specifically, for the ordered first and second feature matrices, an attention transformation function is applied, where parameter matrix Q is applied to the first feature matrix, and parameter matrices K and V are applied to the second feature matrix. Through these hybrid attention layers, the first and second types of features can be more fully integrated.

[0129] It should be noted that when the above-mentioned hybrid attention layer is used in the first encoder, the first type of feature X in the input corresponds to visual features, which consists of the image features v and bounding box features s of each object, for example, the concatenation of image features v and bounding box features s. The second type of feature Y corresponds to text features, which is the embedding vector representation of the initial object category l.

[0130] When the hybrid attention layer described above is used in the second encoder, the first type of feature X in the input consists of the image features v and the first encoded feature x of each object, and still corresponds to visual features. For example, X can be a concatenation of image features v and the first encoded feature x. The second type of feature Y corresponds to text features and is an embedding vector representation of the corrected object category l'.

[0131] exist Figure 6In the illustration, the aforementioned hybrid attention layer is used in both the first and second encoders to achieve a full fusion of visual and textual features. In practice, a variant implementation can also be used, introducing the aforementioned hybrid attention layer into one of the two encoders.

[0132] In one embodiment, it is also possible to Figure 6 The structure is simplified to obtain a simplified relationship recognition model. This simplification may include removing... Figure 6 The image processing involves a first decoder and a second encoder. The process includes inputting the target image into the detection subnetwork to obtain image features, bounding box information, and initial object categories for several objects within the image. Then, the first encoder performs attention-based encoding based on the image features, bounding box information, and initial object categories to obtain the encoded features of each object. The object encoded features obtained from the first encoder are directly output to the classification network, enabling the classification network to identify the relationships between objects. In this case, a hybrid attention layer can be used in the first encoder. The structure and processing mechanism of the hybrid attention layer have been described previously and will not be repeated here.

[0133] By introducing a hybrid attention layer in the encoder, the first and second type features of different modalities can be fully fused, optimizing the encoding network's representation of objects and thus improving the classification performance of subsequent classification networks in classifying the relationships between objects.

[0134] According to another embodiment, with Figure 4 Corresponding to the training method shown, this specification also provides an apparatus for training a relationship recognition model. This apparatus can be deployed in any computing unit, platform, server, or device with computing and processing capabilities. The relationship recognition model to be trained includes an encoding network and a classification network. The encoding network is used to identify and encode multiple objects contained in an image. The classification network is used to identify the relationship categories between the multiple objects from a preset set of relationship categories. Figure 8 An apparatus for training a relation recognition model according to one embodiment is shown. Figure 8 As shown, the device 800 includes:

[0135] The group setting unit 81 is configured to divide the relation category set into K groups and set K classifiers to form the classification network accordingly; the classification space of the first classifier in the K classifiers corresponds to the relation category in the first group, and the k-th classification space corresponding to any subsequent k-th classifier is formed by superimposing the classification space corresponding to the previous classifier with the relation category in the k-th group.

[0136] Sampling unit 82 is configured to sample from the entire set of image samples, targeting K classifiers respectively, to obtain K sample subsets;

[0137] The prediction unit 83 is configured to input the sample images in each sample subset into the corresponding classifiers via the coding network to obtain the prediction results of the relationship categories between objects in the sample images.

[0138] The first loss determination unit 84 is configured to determine the first loss based on the classification loss of each of the K classifiers, wherein the classification loss of any classifier is determined based on the relationship category prediction result obtained by the classifier and the corresponding category label.

[0139] The second loss determination unit 85 is configured to determine the second loss based on the similarity between the prediction results of the relationship category of the same pair of objects by different classifiers among the K classifiers.

[0140] The update unit 86 is configured to determine the total prediction loss based on the first loss and the second loss, and update the encoding network and K classifiers based on the total prediction loss.

[0141] Each unit of the above device can be configured to perform... Figure 4 The method and steps, and the specific execution process, can be referred to in conjunction with the previous steps. Figure 4 The description is omitted.

[0142] According to other embodiments, a computer-readable storage medium is also provided, on which a computer program is stored, which, when executed in a computer, causes the computer to perform the aforementioned combination. Figure 4 The training method described.

[0143] According to another embodiment, a computing device is also provided, including a memory and a processor, wherein the memory stores executable code, and the processor executes the executable code to achieve the foregoing combination. Figure 4 The training method described.

[0144] Those skilled in the art will recognize that, in one or more of the examples above, the functions described in this invention can be implemented using hardware, software, firmware, or any combination thereof. When implemented in software, these functions can be stored in a computer-readable medium or transmitted as one or more instructions or code on a computer-readable medium.

[0145] The specific embodiments described above further illustrate the purpose, technical solution, and beneficial effects of the present invention. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made on the basis of the technical solution of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for training a relation recognition model, the relation recognition model comprising an encoding network and a classification network, the encoding network being used to identify and encode multiple objects contained in an image; The classification network is used to identify the relationship categories between the multiple objects from a preset set of relationship categories, and the method includes: The set of relation categories is divided into K groups, and K classifiers are set accordingly to form the classification network. The classification space of the first classifier in the K classifiers corresponds to the relation category in the first group. The classification space of any subsequent k-th classifier is formed by superimposing the classification space of the previous classifier with the relation category in the k-th group. From the complete set of image samples, samples are taken from each of the K classifiers to obtain K subsets of samples; The sample images in each sample subset are input into the corresponding classifiers via the coding network to obtain the category prediction results of the relationship between objects in the sample images; The first loss is determined based on the classification loss of each of the K classifiers, wherein the classification loss of any classifier is determined based on the relation category prediction result obtained by the classifier and the corresponding category label. The second loss is determined based on the similarity between the prediction results of the relationship category of the same pair of objects by different classifiers among K classifiers. The total prediction loss is determined based on the first loss and the second loss, and the encoding network and K classifiers are updated based on the total prediction loss.

2. The method according to claim 1, wherein, The set of relation categories is divided into K groups, including: Based on the estimated sample size of each relation category in the relation category set, the relation categories are sorted. The K groups are formed according to the sorting, such that the sample size difference index between the relation category with the largest sample size and the relation category with the smallest sample size in each group is lower than a preset threshold.

3. The method according to claim 2, wherein, The estimated sample size is obtained by statistically analyzing all relation categories contained in the complete set of image samples.

4. The method according to claim 2, wherein, The process of forming the K groups according to the aforementioned sorting includes: According to the sorting, each relation category is traversed, wherein for the current relation category being traversed, it is determined whether the difference index between its sample size and the first relation category in the latest grouping is lower than a preset threshold. If the value is below a preset threshold, the current relationship category is added to the latest group. If the value is not lower than a preset threshold, a new group is generated, and the current relationship category is taken as the first relationship category in the newly generated group, and the newly generated group is taken as the latest group that has been formed.

5. The method according to claim 1, wherein, The K sample subsets include any k-th sample subset, and the sampling distribution of the k-th sample subset for each relation category in the k-th classification space conforms to a predetermined condition related to the equilibrium distribution.

6. The method according to claim 1, wherein, From the complete set of image samples, samples are taken for each of the K classifiers, including: For any k-th classifier, a target value is determined based on the estimated sample size of each relation category in the k-th classification space; the target value is the average or median of the estimated sample size of each relation category. Based on the target value, a sampling rate is determined for each relation category; this includes determining a first sampling rate corresponding to a first relation category, such that the first sampling rate is less than 1, wherein the estimated sample size of the first relation category is greater than the target value; Sampling is performed for each relation category based on the sampling rate of each relation category.

7. The method according to claim 6, wherein, Determining the sampling rate for each relation category also includes determining a second sampling rate of 1 for the second relation category, wherein the estimated sample size of the second relation category is less than or equal to the target value.

8. The method according to claim 6, wherein, Determine the first sampling rate corresponding to the first relation category, including: The first sampling rate is defined as the ratio of the target value to the estimated sample size of the first relation category.

9. The method according to claim 6, wherein, Based on the sampling rate of each relation category, sampling is performed for each relation category, including: Obtain a first sample that matches the first relationship category from the complete set of image samples; Generate positive random numbers less than 1; If the positive random number is less than the first sampling rate, the first sample is sampled as the sample corresponding to the first relation category; otherwise, the first sample is discarded.

10. The method according to claim 1, wherein, The first loss is determined based on the classification losses of the K classifiers, including: Based on the prediction results of the relation category of the samples in the k-th sample subset by the k-th classifier and the corresponding category labels, determine the classification loss of the k-th classifier; The sum of the classification losses of the K classifiers is determined as the first loss.

11. The method according to claim 1, wherein, The second loss is determined based on the similarity between the prediction results of different classifiers for the relationship category of the same pair of objects among K classifiers, including: Obtain the first prediction result of the first classifier for a certain object pair in the first classification space, and the second prediction result of the second classifier for the same object pair in the second classification space, wherein the dimension of the second classification space is greater than that of the first classification space. From the second prediction result, the part corresponding to the dimension of the first prediction result is extracted to obtain the second truncation result; Determine the first similarity between the first prediction result and the second truncation result; The second loss is determined based at least on the first similarity.

12. The method according to claim 11, wherein, The object pair is a pair of objects in a common sample image in the first sample subset corresponding to the first classifier and the second sample subset corresponding to the second classifier.

13. The method according to claim 11, wherein, Determining the first similarity between the first prediction result and the second truncation result includes: The KL divergence between the first prediction result and the second truncation result is determined as the first similarity.

14. The method according to claim 1, wherein, The total predicted loss is determined based on the first loss and the second loss, including: The first loss and the second loss are weighted and summed according to the preset weighting factors to obtain the total predicted loss.

15. The method according to claim 1, wherein, The encoding network includes a detection subnetwork and a first encoder; The sample images from each sample subset are input into the corresponding classifiers via the encoding network, including: The first sample image is input into the detection sub-network to obtain the image features, border information and initial object category of several first objects contained in the first sample image; The first encoder performs attention-based encoding based on the image features, bounding box information, and initial object category of each first object to obtain the first encoded features of each first object.

16. The method according to claim 15, wherein, The first encoder includes several hybrid attention layers, each including a self-attention module and a cross-attention module set for each of the two types of features; the attention-based encoding operation includes, in a single hybrid attention layer: By using the self-attention module corresponding to any one of the two types of features, the input feature of that type is transformed based on self-attention to obtain the first transformed feature. By using the cross-attention module corresponding to any one of the two types of features, and based on the attention of the other type of feature relative to the arbitrary one type of feature, the two types of features are combined and transformed to obtain the second transformed feature. The first transformation feature and the second transformation feature are combined to form the output feature of any one of the categories. The two types of features input to the first encoder include a first type of features obtained based on the image features and bounding box information of each first object, and a second type of features obtained based on the initial object category of each first object.

17. The method according to claim 15, wherein, The encoding network also includes a first decoder and a second encoder; The sample images from each sample subset are input into the corresponding classifiers via the encoding network, and the method further includes: The corrected object category of each first object is obtained by using the first decoder based on the first encoded features of each first object. The second encoder performs attention-based encoding based on the image features, first encoding features, and corrected object categories of each first object to obtain the second encoding features of each first object, which are then input into the corresponding i-th classifier.

18. The method according to claim 17, wherein, The first encoder and / or the second encoder include several hybrid attention layers, each hybrid attention layer including a self-attention module and a cross-attention module respectively set for each of the two types of features; the attention-based encoding operation includes, in a single hybrid attention layer: By using the self-attention module corresponding to any one of the two types of features, the input feature of that type is transformed based on self-attention to obtain the first transformed feature. By using the cross-attention module corresponding to any one of the two types of features, and based on the attention of the other type of feature relative to the arbitrary one type of feature, the two types of features are combined and transformed to obtain the second transformed feature. The first transformation feature and the second transformation feature are combined to form the output feature of any one of the categories. The two types of features input to the first encoder include a first type of features obtained based on the image features and bounding box information of each first object, and a second type of features obtained based on the initial object category of each first object. The two types of features input to the second encoder include a first type of features obtained based on the image features and first coding features of each first object, and a second type of features obtained based on the modified object category of each first object.

19. An image analysis method, comprising: A relation recognition model trained by the method of claim 1 is obtained, comprising an encoding network and a classification network, wherein the classification network comprises K classifiers; The target image to be identified is input into the Kth classifier via the encoding network to predict the relationship categories between multiple objects contained in the target image.

20. The method of claim 19, further comprising: Using the multiple objects as multiple nodes and the relationship categories as edges, a scene graph corresponding to the target image is generated.

21. A method for analyzing images using a relation recognition model, wherein the relation recognition model includes an encoding network and a classification network, the encoding network including a detection sub-network and a first encoder; the method includes: The target image is input into the detection sub-network to obtain the image features, bounding box information, and initial object categories of several objects contained in the target image; The first encoder performs attention-based encoding based on the image features, bounding box information, and initial object categories to obtain the encoded features of each object. The encoded features of each object are input into the classification network to identify the relationship categories between the objects; The first encoder includes several hybrid attention layers, each including a self-attention module and a cross-attention module set for each of the two types of features; the attention-based encoding operation includes, in a single hybrid attention layer: By using the self-attention module corresponding to any one of the two types of features, the input feature of that type is transformed based on self-attention to obtain the first transformed feature. By using the cross-attention module corresponding to any one of the two types of features, and based on the attention of the other type of feature relative to the arbitrary one type of feature, the two types of features are combined and transformed to obtain the second transformed feature. The first transformation feature and the second transformation feature are combined to form the output feature of any one of the categories. The two types of features input to the first encoder include a first type of features obtained based on the image features and bounding box information of each first object, and a second type of features obtained based on the initial object category of each first object.

22. A method for analyzing images using a relation recognition model, wherein the relation recognition model includes an encoding network and a classification network, and the encoding network includes a detection sub-network, a first encoder, a first decoder, and a second encoder; the method includes: The target image is input into the detection sub-network to obtain the image features, bounding box information, and initial object categories of several objects contained in the target image; The first encoder performs attention-based encoding based on the image features, bounding box information, and initial object category of each object to obtain the first encoded features of each object. The modified object category of each object is obtained by using the first decoder based on the first encoded features of each object. The second encoder performs attention-based encoding based on the image features, first encoded features, and corrected object categories of each object to obtain the second encoded features of each object. The second encoded features of each object are input into the classification network to identify the categories of relationships between objects; Wherein, the first encoder and / or the second encoder include several hybrid attention layers, each hybrid attention layer including a self-attention module and a cross-attention module respectively set for each of the two types of features; the attention-based encoding operation includes, in a single hybrid attention layer: By using the self-attention module corresponding to any one of the two types of features, the input feature of that type is transformed based on self-attention to obtain the first transformed feature. By using the cross-attention module corresponding to any one of the two types of features, and based on the attention of the other type of feature relative to the arbitrary one type of feature, the two types of features are combined and transformed to obtain the second transformed feature. The first transformation feature and the second transformation feature are combined to form the output feature of any one of the categories. The two types of features input to the first encoder include a first type of features obtained based on the image features and bounding box information of each first object, and a second type of features obtained based on the initial object category of each first object. The two types of features input to the second encoder include a first type of features obtained based on the image features and first coding features of each first object, and a second type of features obtained based on the modified object category of each first object.

23. An apparatus for training a relation recognition model, the relation recognition model comprising an encoding network and a classification network, the encoding network being used to recognize and encode multiple objects contained in an image; The classification network is used to identify the relationship categories between the multiple objects from a preset set of relationship categories, and the device includes: The grouping unit is configured to divide the relation category set into K groups and set K classifiers to form the classification network. The classification space of the first classifier in the K classifiers corresponds to the relation category in the first group, and the k-th classification space of any subsequent k-th classifier is formed by superimposing the classification space of the previous classifier with the relation category in the k-th group. The sampling unit is configured to sample from the entire set of image samples, targeting K classifiers respectively, to obtain K sample subsets; The prediction unit is configured to input the sample images from each sample subset into the corresponding classifiers via the coding network to obtain the prediction results of the relationship categories between objects in the sample images; The first loss determination unit is configured to determine the first loss based on the classification loss of each of the K classifiers, wherein the classification loss of any classifier is determined based on the relationship category prediction result obtained by the classifier and the corresponding category label. The second loss determination unit is configured to determine the second loss based on the similarity between the prediction results of different classifiers for the relationship category of the same pair of objects among the K classifiers. The update unit is configured to determine the total prediction loss based on the first loss and the second loss, and update the encoding network and K classifiers based on the total prediction loss.

24. A computing device, comprising a memory and a processor, characterized in that, The memory stores executable code, and when the processor executes the executable code, it implements the method of any one of claims 1-18.

Images