A scene graph generation method based on collaborative learning and IoST data

Abstract

The present application relates to a scene graph generation method based on collaborative learning and IoST data. The method performs target detection on an image in an IoST scene to obtain visual features of a subject target / object target in the image; for the visual features of the subject target, a difference guidance hint vector between the subject target and the object target is calculated; the initial visual block features and the difference guidance hint vector obtained by adding position encoding are passed through a multi-layer first Transformer, and any visual block feature input to each layer is fused with the corresponding difference guidance hint vector based on the attention mechanism in each layer, and the last layer outputs several subject visual feature blocks; the visual features of the object target are processed in the same way to obtain several object visual feature blocks; a relationship classification result is obtained based on each subject visual feature block and each object visual feature block; finally, a scene graph is constructed based on the relationship classification result, the subject target, and the object target.

Description

Technical Field

[0001] The present application relates to the technical field of scene graph generation, and in particular to a scene graph generation method based on collaborative learning and IoST data. Background Art

[0002] In the context of Intelligence of Social Things (IoST), interactions between social entities (such as people and devices, and devices and environments) are dynamically evolving. Traditional scene graph generation methods for IoST scenarios face challenges such as the difficulty of adapting existing tools to dynamic social interactions, insufficient training data, and a lack of predefined relationship templates. This leads to complex model construction and low relationship prediction accuracy. Furthermore, existing scene graph generation methods focus more on feature extraction or a single classification stage rather than a complete scene graph generation system. Summary of the Invention

[0003] Based on this, it is necessary to provide a scene graph generation method based on collaborative learning and IoST data, which includes:

[0004] S1: Obtain the image to be processed in the IoST scenario, use the object detector to detect the object in the image, and obtain the category, bounding box and visual features of each object in the image. The objects include subject objects and object objects.

[0005] S2: Add position encoding to the visual features of the subject target to obtain the initial visual block features; calculate the differences in category and bounding box between the subject target and the object target to obtain the difference guidance hint vector; pass the initial visual block features through multiple layers of first Transformer. Starting from the second layer, the input of each first Transformer layer is the visual block features output by the previous layer; based on the S²P_MSA mechanism in the first Transformer layer of each layer, the initial visual block features / visual block features of each layer are fused with the difference guidance hint vector. The last first Transformer layer outputs several subject visual feature blocks;

[0006] S3: Replace the visual features of the subject target in step S2 with the visual features of the object target, and execute step S2 again to obtain several object visual feature blocks;

[0007] S4: Connect each subject visual feature block and each object visual feature block along the channel direction and pass them through the fully connected layer respectively to obtain several subject-object semantic visual features; pass the subject-object semantic visual features through the relation decoder, and map the decoding results into relation classification results, and construct a scene graph based on the relation classification results, subject targets, and object targets.

[0008] Preferably, the target detector comprises a Faster R-CNN model.

[0009] Preferably, the process of obtaining the initial visual features includes:

[0010] The visual features of the subject target are divided into several feature blocks, and position encoding vectors are added after the several feature blocks to obtain the initial visual features.

[0011] Preferably, the process of obtaining the difference guidance hint vector includes:

[0012] Based on the semantic vector difference between the subject target and the object target, the subject-object semantic difference feature is obtained;

[0013] Get the coordinates of the upper left corner, lower right corner, and center point of the bounding box of the subject target;

[0014] Get the coordinates of the upper left corner, lower right corner, and center point of the bounding box of the object target;

[0015] Based on the horizontal coordinates of the upper left corner, lower right corner, and center point of the two bounding boxes, calculate the relative position difference of the corresponding horizontal coordinates;

[0016] Based on the upper left corner ordinate, lower right corner ordinate, and center point ordinate of the two bounding boxes, calculate the relative position difference of the corresponding ordinates;

[0017] Based on the width and height of the two bounding boxes, calculate the relative size difference;

[0018] Based on the coordinates of the upper left corner and lower right corner of the two bounding boxes, calculate the proportional difference between the intersection area and the bounding box of the subject target;

[0019] The relative position difference of each horizontal coordinate between the two bounding boxes, the relative position difference of each vertical coordinate, the relative size difference, and the ratio difference between the intersection area and the subject-target bounding box are input into the fully connected layer and mapped into the subject-object spatial difference feature;

[0020] The subject-object semantic difference feature and the subject-object spatial difference feature are concatenated and then activated by the ReLU function. The activation result is regularized by the Dropout function to obtain the spatial cue vector.

[0021] Multiplying the normalized spatial cue vector with any normalized initial visual block feature / visual block feature to obtain a mask vector, and performing a matrix element product operation on the mask vector and the initial visual block feature / visual block feature to obtain the corresponding visual block feature after spatial cueing;

[0022] The spatial difference features of the host and the guest are mapped through the fully connected layer to obtain the channel prompt vector;

[0023] The corresponding spatially cued visual block features are multiplied by the channel cue vector, and the resulting product is passed through a multi-layer perceptron to obtain the difference-guided cue vector corresponding to the initial visual block features / visual block features.

[0024] Preferably, the process of obtaining the subject-object semantic difference feature includes:

[0025] The categories of the subject target and the object target are respectively passed through the Glove model to obtain the subject semantic vector and the object semantic vector;

[0026] The difference between the subject semantic vector and the object semantic vector is calculated, and the obtained difference is passed through the fully connected layer to obtain the subject-object semantic difference feature.

[0027] Preferably, the calculation process of the relative position difference includes:

[0028] Calculate the upper left corner horizontal coordinate difference, upper left corner vertical coordinate difference, lower right corner horizontal coordinate difference, lower right corner vertical coordinate difference, center point horizontal coordinate difference, and center point vertical coordinate difference between the two bounding boxes respectively;

[0029] Divide the horizontal coordinate difference of the upper left corner, the horizontal coordinate difference of the lower right corner, and the horizontal coordinate difference of the center point by the width of the image to get the relative position difference of the corresponding horizontal coordinates;

[0030] Divide the upper left corner vertical coordinate difference, the lower right corner vertical coordinate difference, and the center point vertical coordinate difference by the height of the image to obtain the relative position difference of the corresponding vertical coordinates.

[0031] Preferably, the calculation process of the relative size difference includes:

[0032] Relative size differences include relative width differences, relative height differences, and relative proportion differences;

[0033] Calculate the width difference of the two bounding boxes and divide the width difference by the width of the image to get the relative width difference;

[0034] Calculate the height difference between the two bounding boxes and divide the height difference by the height of the image to get the relative height difference;

[0035] Divide the width of the bounding box of the object by the width of the image to obtain a first quotient; divide the height of the bounding box of the object by the height of the image to obtain a second quotient; multiply the first quotient by the second quotient to obtain a first product;

[0036] Divide the width of the subject target's bounding box by the width of the image to obtain a third quotient; divide the height of the subject target's bounding box by the height of the image to obtain a fourth quotient; multiply the third quotient by the fourth quotient to obtain a second product;

[0037] Subtract the second product from the first product to get the relative proportional difference.

[0038] Preferably, the calculation process of the ratio difference between the intersection area and the subject target bounding box is:

[0039] Determine whether there is an intersection between the bounding box of the subject target and the bounding box of the object target,

[0040] If there is no intersection area, the ratio difference is 0; if there is an intersection area:

[0041] Apply the minimum function to the horizontal coordinates of the lower right corners of the two bounding boxes to obtain the first minimum value; apply the maximum function to the horizontal coordinates of the upper left corners of the two bounding boxes to obtain the first maximum value; subtract the first maximum value from the first minimum value to obtain the intersection of the horizontal coordinates;

[0042] Apply the minimum function to the ordinates of the lower right corners of the two bounding boxes to obtain the second minimum value; apply the maximum function to the ordinates of the upper left corners of the two bounding boxes to obtain the second maximum value; subtract the second maximum value from the second minimum value to obtain the intersection of the ordinates;

[0043] Divide the intersection of the horizontal coordinates by the width of the image to get the fifth quotient; divide the intersection of the vertical coordinates by the height of the image; multiply the fifth quotient and the sixth quotient to get the third product;

[0044] Subtracting the second product from the third product gives the proportional difference between the intersection region and the bounding box of the subject target.

[0045] Preferably, fusing the initial visual block features / visual block features of each layer input with the difference guidance hint vector through the S²P_MSA mechanism includes:

[0046] In any layer of the first Transformer,

[0047] Calculate the sum of the initial visual block feature / the visual block feature output by the previous layer and the difference guidance hint vector, and multiply the result by the query projection matrix to obtain the query vector;

[0048] Multiply the initial visual block feature / the visual block feature output by the previous layer with the key projection matrix to obtain the key vector;

[0049] Multiply the initial visual block feature / the visual block feature outputted by the previous layer by the value projection matrix to obtain a value vector;

[0050] The attention score is calculated based on the query vector, key vector, and value vector, and the attention scores in all attention heads are concatenated. The concatenated results are fused into the visual block features / several subject visual feature blocks output by the corresponding layer through a fully connected layer.

[0051] Preferably, the relation decoder includes a multi-layer second Transformer, where the number of layers of the second Transformer is three more than that of the first Transformer; the classification patch features, semantic patch features, spatial patch features, and several subject-object semantic visual features are fused through the relation decoder, the classification patch features in the last layer of the second Transformer are output, and the output classification patch features are mapped into relation classification results through a linear classifier;

[0052] The classification patch features are learning parameters;

[0053] After connecting the subject semantic vector and the object semantic vector, they pass through the fully connected layer and output the semantic patch feature;

[0054] The bounding box features of the subject target and the object target are concatenated and passed through a fully connected layer to output the spatial patch features.

[0055] The length, width, and center point coordinates of the subject target / object target's bounding box are passed through the fully connected layer to obtain the subject target / object target's bounding box features.

[0056] Beneficial effects: This method performs target detection on images in IoST scenarios to obtain the visual features of the subject target / object target in the image; for the visual features of the subject target, the difference guidance cue vector between the subject target and the object target is calculated; the initial visual block features and the difference guidance cue vector obtained by adding position encoding are passed through multiple layers of the first Transformer, and based on the attention mechanism in the first Transformer of each layer, any visual block feature input in each layer is fused with the corresponding difference guidance cue vector, and the last layer outputs several subject visual feature blocks; the visual features of the object target are processed in the same way to obtain several object visual feature blocks; the relationship classification results are obtained based on each subject visual feature block and each object visual feature block; finally, a scene graph is constructed based on the relationship classification results, the subject target, and the object target. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0058] Figure 1 This is a flowchart of a method for generating a scene graph based on collaborative learning and IoST data in an embodiment of the present application. DETAILED DESCRIPTION

[0059] To make the above-mentioned objects, features, and advantages of the present application more clearly understood, the specific embodiments of the present application are described in detail below with reference to the accompanying drawings. The following description sets forth many specific details to facilitate a full understanding of the present application. However, the present application can be implemented in many other ways than those described herein, and those skilled in the art can make similar improvements without violating the scope of the present application. Therefore, the present application is not limited to the specific embodiments disclosed below.

[0060] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of the technical features being referred to. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of such features. Throughout the description of this application, "plurality" means at least two, for example, two, three, etc., unless otherwise specifically defined.

[0061] like Figure 1 As shown, this embodiment provides a method for generating a scene graph based on collaborative learning and IoST data, the method comprising:

[0062] S1: Obtain the image to be processed in the IoST scenario, use the object detector to detect the object in the image, and obtain the category, bounding box, and visual features of each object in the image. The objects include subject objects and object objects.

[0063] In this embodiment, the object detector includes a Faster R-CNN model. In this embodiment, the object detector Faster R-CNN is used to detect objects in the image and obtain the bounding box, category, and feature information of the object.

[0064] In this embodiment, the subject features and object features of the target interaction prompts are fully utilized to strengthen the collaborative learning strategy of the target detector, and the training effect is improved by combining a small amount of labeled data and automatically generated pseudo-labels, thereby strengthening the learning of feature extraction, thereby improving the encoding ability of the image and the subsequent classification ability.

[0065] S2: Add position encoding to the visual features of the subject target to obtain the initial visual block features; calculate the differences in categories and bounding boxes between the subject target and the object target to obtain the difference guidance hint vector; pass the initial visual block features through multiple layers of first Transformer, and starting from the second layer, the input of each first Transformer layer is the visual block features output by the previous layer; based on the S²P_MSA mechanism in the first Transformer layer of each layer, the initial visual block features / visual block features inputted in each layer are fused with the difference guidance hint vector, and the last layer of first Transformer layer outputs several subject visual feature blocks.

[0066] In this step, the process of obtaining the initial visual features includes:

[0067] The visual features of the subject target are divided into several feature blocks, and position encoding vectors are added after the several feature blocks to obtain the initial visual features.

[0068] Furthermore, the process of obtaining the difference guidance hint vector includes:

[0069] Based on the semantic vector difference between the subject target and the object target, a subject-object semantic difference feature is obtained. In this embodiment, the process of obtaining the subject-object semantic difference feature includes:

[0070] The categories of the subject target and the object target are respectively passed through the Glove model to obtain the subject semantic vector and the object semantic vector;

[0071] The difference between the subject semantic vector and the object semantic vector is calculated, and the obtained difference is passed through the fully connected layer to obtain the subject-object semantic difference feature.

[0072] Get the upper left corner coordinates, lower right corner coordinates, and center point coordinates of the subject target's bounding box (the subject target's bounding box information);

[0073] Get the upper left corner coordinates, lower right corner coordinates, and center point coordinates of the bounding box of the object target (bounding box information of the object target);

[0074] When calculating the difference, the bounding box information of the subject target is subtracted from the bounding box information of the object target;

[0075] Based on the horizontal coordinates of the upper left corner, lower right corner, and center point of the two bounding boxes, calculate the relative position difference of the corresponding horizontal coordinates;

[0076] Based on the upper left corner ordinate, lower right corner ordinate, and center point ordinate of the two bounding boxes, the relative position difference of the corresponding ordinates is calculated.

[0077] In this embodiment, the calculation process of the relative position difference includes:

[0078] Calculate the upper left corner horizontal coordinate difference, upper left corner vertical coordinate difference, lower right corner horizontal coordinate difference, lower right corner vertical coordinate difference, center point horizontal coordinate difference, and center point vertical coordinate difference between the two bounding boxes respectively;

[0079] Divide the horizontal coordinate difference of the upper left corner, the horizontal coordinate difference of the lower right corner, and the horizontal coordinate difference of the center point by the width of the image to get the relative position difference of the corresponding horizontal coordinates;

[0080] Divide the upper left corner vertical coordinate difference, the lower right corner vertical coordinate difference, and the center point vertical coordinate difference by the height of the image to obtain the relative position difference of the corresponding vertical coordinates.

[0081] Based on the width and height of the two bounding boxes, the relative size difference is calculated.

[0082] In this embodiment, the calculation process of the relative size difference includes:

[0083] Relative size differences include relative width differences, relative height differences, and relative proportion differences;

[0084] Calculate the width difference of the two bounding boxes and divide the width difference by the width of the image to get the relative width difference;

[0085] Calculate the height difference between the two bounding boxes and divide the height difference by the height of the image to get the relative height difference;

[0086] Divide the width of the bounding box of the object by the width of the image to obtain a first quotient; divide the height of the bounding box of the object by the height of the image to obtain a second quotient; multiply the first quotient by the second quotient to obtain a first product;

[0087] Divide the width of the subject target's bounding box by the width of the image to obtain a third quotient; divide the height of the subject target's bounding box by the height of the image to obtain a fourth quotient; multiply the third quotient by the fourth quotient to obtain a second product;

[0088] Subtract the second product from the first product to get the relative proportional difference.

[0089] Based on the upper left corner coordinates and lower right corner coordinates of the two bounding boxes, the proportional difference between the intersection area and the bounding box of the subject target is calculated.

[0090] In this embodiment, the calculation process of the ratio difference between the intersection area and the bounding box of the subject target is:

[0091] Determine whether there is an intersection between the bounding box of the subject target and the bounding box of the object target,

[0092] If there is no intersection area, the ratio difference is 0; if there is an intersection area:

[0093] Apply the minimum function to the horizontal coordinates of the lower right corners of the two bounding boxes to obtain the first minimum value; apply the maximum function to the horizontal coordinates of the upper left corners of the two bounding boxes to obtain the first maximum value; subtract the first maximum value from the first minimum value to obtain the intersection of the horizontal coordinates;

[0094] Apply the minimum function to the ordinates of the lower right corners of the two bounding boxes to obtain the second minimum value; apply the maximum function to the ordinates of the upper left corners of the two bounding boxes to obtain the second maximum value; subtract the second maximum value from the second minimum value to obtain the intersection of the ordinates;

[0095] Divide the intersection of the horizontal coordinates by the width of the image to get the fifth quotient; divide the intersection of the vertical coordinates by the height of the image; multiply the fifth quotient and the sixth quotient to get the third product;

[0096] Subtracting the second product from the third product gives the proportional difference between the intersection region and the bounding box of the subject target.

[0097] The relative position difference of each horizontal coordinate between the two bounding boxes, the relative position difference of each vertical coordinate, the relative size difference, and the ratio difference between the intersection area and the subject-target bounding box are input into the fully connected layer and mapped into the subject-object spatial difference feature;

[0098] The subject-object semantic difference feature and the subject-object spatial difference feature are concatenated and then activated by the ReLU function. The activation result is regularized by the Dropout function to obtain the spatial cue vector.

[0099] Multiplying the normalized spatial cue vector with any normalized initial visual block feature / visual block feature to obtain a mask vector, and performing a matrix element product operation on the mask vector and the initial visual block feature / visual block feature to obtain the corresponding visual block feature after spatial cueing;

[0100] The spatial difference features of the host and the guest are mapped through the fully connected layer to obtain the channel prompt vector;

[0101] The corresponding spatially cued visual block features are multiplied by the channel cue vector, and the resulting product is passed through a multi-layer perceptron to obtain the difference-guided cue vector corresponding to the initial visual block features / visual block features.

[0102] In this embodiment, the process of obtaining the mask vector includes:

[0103] Multiplying each element of the normalized spatial cue vector with each element of any normalized initial visual block feature / visual block feature to obtain a plurality of fourth products;

[0104] It is determined whether each fourth product is greater than a preset threshold; if so, the value of the position corresponding to the fourth product is set to 1; otherwise, the value of the position corresponding to the fourth product is set to 0.

[0105] In this embodiment, the preset threshold is 0.5, which can be set according to actual needs.

[0106] The visual features used in this embodiment are target-related features obtained by the target detection module. However, these visual features contain information irrelevant to relationship classification, resulting in a scattered feature space for similar relationships and low classification accuracy. Because the relationship classification task is a multi-target interactive task, the subject's visual attention area is related to the object. Therefore, this embodiment designs a subject-object semantic difference calculation module and a subject-object spatial difference calculation module to respectively obtain subject-object semantic difference features and subject-object spatial difference features; and designs a difference-guided prompt generation module that uses the difference-guided prompt vector to update the query portion, thereby guiding the model to focus on the relationship-related target visual features.

[0107] In this embodiment, the S²P_MSA mechanism is used to replace the multi-head self-attention mechanism MSA in the conventional Transformer. Specifically, the S²P_MSA mechanism is used to fuse the initial visual block features / visual block features of each layer input with the difference guidance hint vector, including:

[0108] In any layer of the first Transformer,

[0109] Calculate the sum of the initial visual block feature / the visual block feature output by the previous layer and the difference guidance hint vector, and multiply the result by the query projection matrix to obtain the query vector;

[0110] Multiply the initial visual block feature / the visual block feature output by the previous layer with the key projection matrix to obtain the key vector;

[0111] Multiply the initial visual block feature / the visual block feature outputted by the previous layer by the value projection matrix to obtain a value vector;

[0112] The attention score is calculated based on the query vector, key vector, and value vector, and the attention scores in all attention heads are concatenated. The concatenated results are fused into the visual block features / several subject visual feature blocks output by the corresponding layer through a fully connected layer.

[0113] The difference-guided cue vector is used to update the query part of the S²P_MSA mechanism, guiding the model to focus on the visual area related to relation classification rather than the entire target visual area, thereby obtaining relation-intrinsic features (subject visual feature block / object visual feature block) for relation classification.

[0114] S3: Replace the visual features of the subject target in step S2 with the visual features of the object target, and execute step S2 again to obtain several object visual feature blocks.

[0115] Taking into account the different characteristics of the target when serving as a subject and an object, this embodiment designs a dual feature enhancement module based on the Transformer architecture, which is used to refine the subject visual features and the object visual features respectively; it fully exploits the differences in semantic and spatial position features between the subject and the object, and obtains the subject visual feature block and the object visual feature block for relationship classification, providing a basis for subsequent relationship classification.

[0116] S4: Connect each subject visual feature block and each object visual feature block along the channel direction and pass them through the fully connected layer respectively to obtain several subject-object semantic visual features; pass the subject-object semantic visual features through the relation decoder, and map the decoding results into relation classification results, and construct a scene graph based on the relation classification results, subject targets, and object targets.

[0117] Furthermore, the relation decoder includes a multi-layer second Transformer, where the number of layers of the second Transformer is three more than that of the first Transformer; the classification patch features, semantic patch features, spatial patch features, and several subject-object semantic visual features are fused through the relation decoder, and the classification patch features in the last layer of the second Transformer are output, and the output classification patch features are mapped into the relation classification results through a linear classifier;

[0118] The classification patch features are learning parameters;

[0119] After connecting the subject semantic vector and the object semantic vector, they pass through the fully connected layer and output the semantic patch feature;

[0120] The bounding box features of the subject target and the object target are concatenated and passed through a fully connected layer to output the spatial patch features.

[0121] The length (calculated by the horizontal coordinates of the upper left corner and the lower right corner of the bounding box), width (calculated by the vertical coordinates of the upper left corner and the lower right corner of the bounding box), and center point coordinates of the subject target / object target are passed through the fully connected layer to obtain the bounding box features of the subject target / object target.

[0122] Furthermore, triples are constructed based on the subject target, relation classification results, and object target. The set of all triples in the image is the scene graph.

[0123] This embodiment designs a relationship classification module based on the Transformer architecture, which realizes the classification of relationships by mining the association between subject-object semantic visual features, semantic patch features, and spatial patch features.

[0124] The scene graph generation method based on collaborative learning and IoST data provided in this embodiment has the following beneficial effects:

[0125] 1. This method performs object detection on images in IoST scenarios to obtain visual features of the subject and object targets in the image. For the visual features of the subject target, a difference-guided cue vector is calculated between the subject and object targets. The initial visual block features and difference-guided cue vectors obtained by adding positional encoding are passed through multiple layers of first-layer Transformers. The attention mechanism in each first-layer Transformer is used to fuse any input visual block features with the corresponding difference-guided cue vectors, and the final layer outputs several subject visual feature blocks. The visual features of the object targets are processed in the same way to obtain several object visual feature blocks. Relationship classification results are obtained based on each subject and object visual feature block. Finally, a scene graph is constructed based on the relationship classification results, the subject target, and the object target. This method achieves unbiased classification of relationships between objects in images in IoST scenarios with a long-tail distribution, improving the recall and precision of scene graph generation and significantly reducing the workload.

[0126] 2. After testing, the method achieved a recall rate of 44.5% mR@100 and 50.0% F@100 after training.

[0127] The technical features of the above-mentioned embodiments can be combined arbitrarily. In order to make the description concise, not all possible combinations of the technical features in the above-mentioned embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0128] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the patent application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present patent application shall be determined by the appended claims.

Claims

1. A scene graph generation method based on collaborative learning and IoST data, characterized in that: include: S1: Obtain the image to be processed in the IoST scenario, use the object detector to detect the object in the image, and obtain the category, bounding box and visual features of each object in the image. The objects include subject objects and object objects. S2: Add position encoding to the visual features of the subject target to obtain the initial visual block features; calculate the differences in category and bounding box between the subject target and the object target to obtain the difference guidance hint vector; pass the initial visual block features through multiple layers of first Transformer. Starting from the second layer, the input of each first Transformer layer is the visual block features output by the previous layer; based on the S²P_MSA mechanism in the first Transformer layer of each layer, the initial visual block features / visual block features of each layer are fused with the difference guidance hint vector. The last first Transformer layer outputs several subject visual feature blocks; The process of obtaining the difference guidance hint vector includes: Based on the semantic vector difference between the subject target and the object target, the subject-object semantic difference feature is obtained; The relative position difference of each horizontal coordinate between the two bounding boxes, the relative position difference of each vertical coordinate, the relative size difference, and the ratio difference between the intersection area and the subject-target bounding box are input into the fully connected layer and mapped into the subject-object spatial difference feature; The subject-object semantic difference feature and the subject-object spatial difference feature are concatenated and then activated by the ReLU function. The activation result is regularized by the Dropout function to obtain the spatial cue vector. Multiplying the normalized spatial cue vector with any normalized initial visual block feature / visual block feature to obtain a mask vector, and performing a matrix element product operation on the mask vector and the initial visual block feature / visual block feature to obtain the corresponding visual block feature after spatial cueing; The spatial difference features of the host and the guest are mapped through the fully connected layer to obtain the channel prompt vector; Multiply the corresponding spatially cued visual block features with the channel cue vector, and pass the resulting product through a multi-layer perceptron to obtain the difference guidance cue vector corresponding to the initial visual block features / visual block features; S3: Replace the visual features of the subject target in step S2 with the visual features of the object target, and execute step S2 again to obtain several object visual feature blocks; S4: Connect each subject visual feature block and each object visual feature block along the channel direction and pass them through the fully connected layer respectively to obtain several subject-object semantic visual features; pass the subject-object semantic visual features through the relation decoder, and map the decoding results into relation classification results, and construct a scene graph based on the relation classification results, subject targets, and object targets.

2. The scene graph generation method based on collaborative learning and IoST data according to claim 1 is characterized in that The object detector includes a Faster R-CNN model.

3. The scene graph generation method based on collaborative learning and IoST data according to claim 1 is characterized in that The process of obtaining initial visual features includes: The visual features of the subject target are divided into several feature blocks, and position encoding vectors are added after the several feature blocks to obtain the initial visual features.

4. The scene graph generation method based on collaborative learning and IoST data according to claim 1, characterized in that The process of obtaining the relative position difference of the horizontal coordinates, the relative position difference of the vertical coordinates, the relative size difference, and the ratio difference between the intersection area and the subject target bounding box between the two bounding boxes includes: Get the coordinates of the upper left corner, lower right corner, and center point of the bounding box of the subject target; Get the coordinates of the upper left corner, lower right corner, and center point of the bounding box of the object target; Based on the horizontal coordinates of the upper left corner, lower right corner, and center point of the two bounding boxes, calculate the relative position difference of the corresponding horizontal coordinates; Based on the upper left corner ordinate, lower right corner ordinate, and center point ordinate of the two bounding boxes, calculate the relative position difference of the corresponding ordinates; Based on the width and height of the two bounding boxes, calculate the relative size difference; Based on the upper left corner coordinates and lower right corner coordinates of the two bounding boxes, the proportional difference between the intersection area and the bounding box of the subject target is calculated.

5. The scene graph generation method based on collaborative learning and IoST data according to claim 4 is characterized in that The process of obtaining the subject-object semantic difference features includes: The categories of the subject target and the object target are respectively passed through the Glove model to obtain the subject semantic vector and the object semantic vector; The difference between the subject semantic vector and the object semantic vector is calculated, and the obtained difference is passed through the fully connected layer to obtain the subject-object semantic difference feature.

6. The scene graph generation method based on collaborative learning and IoST data according to claim 4, characterized in that The calculation process of relative position difference includes: Calculate the upper left corner horizontal coordinate difference, upper left corner vertical coordinate difference, lower right corner horizontal coordinate difference, lower right corner vertical coordinate difference, center point horizontal coordinate difference, and center point vertical coordinate difference between the two bounding boxes respectively; Divide the horizontal coordinate difference of the upper left corner, the horizontal coordinate difference of the lower right corner, and the horizontal coordinate difference of the center point by the width of the image to get the relative position difference of the corresponding horizontal coordinates; Divide the upper left corner vertical coordinate difference, the lower right corner vertical coordinate difference, and the center point vertical coordinate difference by the height of the image to obtain the relative position difference of the corresponding vertical coordinates.

7. The scene graph generation method based on collaborative learning and IoST data according to claim 4 is characterized in that: The calculation process of relative size difference includes: Relative size differences include relative width differences, relative height differences, and relative proportion differences; Calculate the width difference of the two bounding boxes and divide the width difference by the width of the image to get the relative width difference; Calculate the height difference between the two bounding boxes and divide the height difference by the height of the image to get the relative height difference; Divide the width of the bounding box of the object by the width of the image to obtain a first quotient; divide the height of the bounding box of the object by the height of the image to obtain a second quotient; multiply the first quotient by the second quotient to obtain a first product; Divide the width of the subject target's bounding box by the width of the image to obtain a third quotient; divide the height of the subject target's bounding box by the height of the image to obtain a fourth quotient; multiply the third quotient by the fourth quotient to obtain a second product; Subtract the second product from the first product to get the relative proportional difference.

8. The scene graph generation method based on collaborative learning and IoST data according to claim 7, characterized in that: The calculation process of the ratio difference between the intersection area and the bounding box of the subject target is: Determine whether there is an intersection between the bounding box of the subject target and the bounding box of the object target, If there is no intersection area, the ratio difference is 0; if there is an intersection area: Apply the minimum function to the horizontal coordinates of the lower right corners of the two bounding boxes to obtain the first minimum value; apply the maximum function to the horizontal coordinates of the upper left corners of the two bounding boxes to obtain the first maximum value; subtract the first maximum value from the first minimum value to obtain the intersection of the horizontal coordinates; Apply the minimum function to the ordinates of the lower right corners of the two bounding boxes to obtain the second minimum value; apply the maximum function to the ordinates of the upper left corners of the two bounding boxes to obtain the second maximum value; subtract the second maximum value from the second minimum value to obtain the intersection of the ordinates; Divide the intersection of the horizontal coordinates by the width of the image to get the fifth quotient; divide the intersection of the vertical coordinates by the height of the image; multiply the fifth quotient and the sixth quotient to get the third product; Subtracting the second product from the third product gives the proportional difference between the intersection region and the bounding box of the subject target.

9. The scene graph generation method based on collaborative learning and IoST data according to claim 1, characterized in that: The initial visual block features / visual block features of each layer input are fused with the difference-guided hint vector through the S²P_MSA mechanism, including: In any layer of the first Transformer, Calculate the sum of the initial visual block feature / the visual block feature output by the previous layer and the difference guidance hint vector, and multiply the result by the query projection matrix to obtain the query vector; Multiply the initial visual block feature / the visual block feature output by the previous layer with the key projection matrix to obtain the key vector; Multiply the initial visual block feature / the visual block feature outputted by the previous layer by the value projection matrix to obtain a value vector; The attention score is calculated based on the query vector, key vector, and value vector, and the attention scores in all attention heads are concatenated. The concatenated results are fused into the visual block features / several subject visual feature blocks output by the corresponding layer through a fully connected layer.

10. The scene graph generation method based on collaborative learning and IoST data according to claim 5, characterized in that: The relation decoder includes a multi-layer second Transformer, where the number of layers of the second Transformer is three more than that of the first Transformer; the classification patch features, semantic patch features, spatial patch features, and several subject-object semantic visual features are fused through the relation decoder, and the classification patch features in the last layer of the second Transformer are output. The output classification patch features are mapped into relation classification results through a linear classifier; The classification patch features are learning parameters; After connecting the subject semantic vector and the object semantic vector, they pass through the fully connected layer and output the semantic patch feature; The bounding box features of the subject target and the object target are concatenated and passed through a fully connected layer to output the spatial patch features. The length, width, and center point coordinates of the subject target / object target's bounding box are passed through the fully connected layer to obtain the subject target / object target's bounding box features.

Images