Scene graph generation method and device

Abstract

The present disclosure provides a scene graph generation method and device. The scene graph generation method includes: identifying an input image to identify all targets in the input image; generating a relationship set, wherein the relationship set includes the relationship between any two targets among all targets; selecting a second relationship r_dk associated with a first relationship r_ij in the relationship set, wherein the first relationship r_ij is a relationship from the i-th target to the j-th target, and the second relationship r_dk is a relationship from the d-th target to the k-th target, 1≤i,j,d,k≤N, N is the total number of targets, i≠j, d≠k, and the first relationship r_ij and the second relationship r_dk involve the same target; updating the relationship value of the first relationship r_ij by using the relationship value between the different targets involved in the first relationship r_ij and the second relationship r_dk; and generating a scene graph using all targets and the updated relationship values. The present disclosure can generate a robust scene graph.

Description

Technical Field

[0001] The present disclosure relates to the field of information processing, and in particular to a method and device for generating a scene graph. Background Art

[0002] A scene graph is a structured representation of the information contained in an image. It not only represents all objects in the image but also expresses the relationships between different objects. Therefore, scene graph generation facilitates visual understanding and explainable reasoning, and is widely used in tasks such as image description, visual question answering, and 3D scene synthesis. It also helps analyze user profiles based on their behavior in real-world scenarios. For example, it can be used to analyze and provide early warnings for pedestrian behavior through video surveillance.

[0003] Existing scene graph generation methods mainly include scene graph generation methods based on recurrent neural networks and scene graph generation methods based on graph neural networks. Summary of the Invention

[0004] The inventors noted that existing scene graph generation methods only assume homogeneous connections between objects and relationships. That is, they only consider dependencies between objects or relationships of the same type. However, objects and relationships in a scene graph often do not belong to the same type, i.e., they are highly heterogeneous. However, existing techniques suffer from the following three problems:

[0005] 1) How to simultaneously consider the heterogeneity between objects and between relations? However, existing scene graph generation methods do not explicitly characterize the heterogeneity between objects. In addition, existing neural network models also ignore the heterogeneity between relations.

[0006] 2) In real applications, objects in the same scene are prone to large occlusions, especially between two objects of different classes, which will make it difficult for graph neural networks to distinguish between homogeneity and heterogeneity. This will ultimately affect the recognition of objects and relationships. For example, Figure 1 The scene diagram shown is as follows Figure 2 However, as Figure 1 As shown in the figure, the lady (target 1) and the refrigerator (target 2) are blocked by a large area. The existing scene graph generation model cannot correctly identify the two objects and the relationship between the two objects. In this case, the problem of being unable to distinguish between target 1 and target 2 will occur.

[0007] 3) N objects involve N(N-1) potential relationships. Therefore, considering the connections between all potential relationships will lead to high computational complexity. Furthermore, not all pairs of objects have meaningful relationships. The presence of some redundant relationships in the scene graph may reduce the performance of scene graph generation.

[0008] Accordingly, the present disclosure provides a scene graph generation method that can effectively identify the heterogeneity and homogeneity between objects and relationships in complex scenes, thereby generating a robust scene graph.

[0009] According to a first aspect of an embodiment of the present disclosure, a scene graph generation method is provided, comprising: identifying an input image to identify all targets in the input image; generating a relationship set, wherein the relationship set includes the relationship between any two targets among all the targets; selecting a second relationship r_dk associated with a first relationship r_ij from the relationship set, wherein the first relationship r_ij is a relationship from the i-th target to the j-th target, and the second relationship r_dk is a relationship from the d-th target to the k-th target, 1≤i,j,d,k≤N, N is the total number of targets, i≠j, d≠k, and the first relationship r_ij and the second relationship r_dk involve a same target; updating the relationship value of the first relationship r_ij by using the relationship value between different targets involved in the first relationship r_ij and the second relationship r_dk; and generating a scene graph using all the targets and the updated relationship value.

[0010] In some embodiments, when the i-th target and the d-th target are the same, and the j-th target and the k-th target are different, updating the relationship value of the first relationship r_ij includes: updating the relationship value of the first relationship r_ij according to the relationship value of the third relationship r_jk from the j-th target to the k-th target.

[0011] In some embodiments, updating the relationship value of the first relationship r_ij according to the relationship value of the third relationship r_jk includes: determining whether the overlap ratio between the j-th target and the k-th target is less than a preset threshold; if the overlap ratio between the j-th target and the k-th target is less than the preset threshold, multiplying the relationship value of the third relationship r_jk by a first weight to obtain a first feedback value, wherein the first weight is a positive number; and using the first feedback value to update the relationship value of the first relationship r_ij.

[0012] In some embodiments, if the overlap ratio between the j-th target and the k-th target is not less than a preset threshold, the relationship value of the third relationship r_jk is multiplied by a second weight to obtain a second feedback value, wherein the second weight is a negative number; and the relationship value of the first relationship r_ij is updated using the second feedback value.

[0013] In some embodiments, when the i-th target and the d-th target are different and the j-th target and the k-th target are the same, updating the relationship value of the first relationship r_ij includes: updating the relationship value of the first relationship r_ij according to the relationship value of the fourth relationship r_id from the i-th target to the d-th target.

[0014] In some embodiments, updating the relationship value of the first relationship r_ij according to the relationship value of the fourth relationship r_id includes: determining whether the overlap ratio between the i-th target and the d-th target is less than a preset threshold; if the overlap ratio between the i-th target and the d-th target is less than the preset threshold, multiplying the relationship value of the fourth relationship r_id by a third weight to obtain a third feedback value, wherein the third weight is a positive number; and using the third feedback value to update the relationship value of the first relationship r_ij.

[0015] In some embodiments, if the overlap ratio between the i-th target and the d-th target is not less than a preset threshold, the relationship value of the fourth relationship r_id is multiplied by a fourth weight to obtain a fourth feedback value, wherein the fourth weight is a negative number; and the relationship value of the first relationship r_ij is updated using the fourth feedback value.

[0016] According to a second aspect of an embodiment of the present disclosure, a scene graph generation device is provided, comprising: a first processing module configured to identify an input image to identify all targets in the input image; a second processing module configured to generate a relationship set, wherein the relationship set includes the relationship between any two targets among all the targets; a third processing module configured to select a second relationship r_dk associated with a first relationship r_ij from the relationship set, wherein the first relationship r_ij is a relationship from the i-th target to the j-th target, the second relationship r_dk is a relationship from the d-th target to the k-th target, 1≤i,j,d,k≤N, N is the total number of targets, i≠j, d≠k, and the first relationship r_ij and the second relationship r_dk involve a same target; the relationship value of the first relationship r_ij is updated by using the relationship value between the different targets involved in the first relationship r_ij and the second relationship r_dk; and a fourth processing module configured to generate a scene graph using all the targets and the updated relationship value.

[0017] According to a third aspect of an embodiment of the present disclosure, a scene graph generation device is provided, comprising: a memory configured to store instructions; a processor coupled to the memory, the processor being configured to execute a method as described in any of the above embodiments based on the instructions stored in the memory.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method involved in any of the above embodiments is implemented.

[0019] Other features and advantages of the present disclosure will become apparent from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0021] Figure 1 A schematic diagram of an image to be recognized according to an embodiment of the present disclosure;

[0022] Figure 2 Based on Figure 1 The scene generation graph;

[0023] Figure 3 A schematic flow chart of a scene graph generation method according to an embodiment of the present disclosure;

[0024] Figure 4 A schematic diagram of an image to be recognized according to another embodiment of the present disclosure;

[0025] Figure 5 A schematic diagram of an identified target according to an embodiment of the present disclosure;

[0026] Figure 6 A schematic diagram of relationship value feedback update according to an embodiment of the present disclosure;

[0027] Figure 7 A schematic diagram of relationship value feedback update according to another embodiment of the present disclosure;

[0028] Figure 8 for Figure 4 Schematic diagram of target recognition;

[0029] Figure 9 for Figure 8 A schematic diagram of the scene graph;

[0030] Figure 10 This is a schematic structural diagram of a scene graph generating device according to an embodiment of the present disclosure;

[0031] Figure 11 This is a schematic structural diagram of a scene graph generating device according to another embodiment of the present disclosure;

[0032] Figure 12 A schematic diagram of a scene graph generation framework according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all the embodiments. The following description of at least one exemplary embodiment is actually only illustrative and is in no way intended to limit the present disclosure and its application or use. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present disclosure.

[0034] Unless specifically stated otherwise, the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure.

[0035] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.

[0036] Technologies, methods and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, such technologies, methods and equipment should be considered part of the authorization specification.

[0037] In all examples shown and discussed herein, any specific values should be interpreted as merely exemplary and not limiting. Therefore, other examples of the exemplary embodiments may have different values.

[0038] It should be noted that like reference numerals and letters refer to like items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.

[0039] Figure 3 1 is a flow chart of a scene graph generation method according to an embodiment of the present disclosure. In some embodiments, the following scene graph generation method is performed by a scene graph generation device.

[0040] In step 301, the input image is recognized to identify all objects in the input image.

[0041] In some embodiments, as Figure 4 As shown, in the input image, a little boy is sitting at the table, eating food on a plate with a fork.

[0042] For example, by using Faster RCNN (Faster Region-based Convolutional Neural Network, fast regional convolutional neural network), Figure 4 The target in is identified. The recognition result is as follows Figure 5 shown.

[0043] like Figure 5 As shown in the figure, target 1 is a boy, target 3 is a hand, target 5 is a mouth, target 7 is food, target 2 includes a hand and a fork, target 4 includes a mouth and food, target 6 includes a plate and food, and target 8 includes a table, a plate, and food. That is, targets 2, 4, 6, and 8 are heterogeneous.

[0044] In step 302 , a relationship set is generated, where the relationship set includes the relationship between any two targets in all targets.

[0045] In some embodiments, such as ART (Adaptive Reweighting Transformer), the relationship between any two targets is determined.

[0046] For example, the context coefficient eij between target i and target j is

[0047] e ij =w T (W i x i ⊙W n x j ⊙(x ij +B ij )), (1)

[0048] Among them, x i is the visual feature of the i-th target in the image, x j is the visual feature of the jth target in the image, B ij is the spatial feature of the i-th target and the j-th target, x ij is the visual feature of the i-th target and the j-th target in the image, ω T 、W i and W n For the corresponding parameters.

[0049] Next, the relationship value between the i-th target and the j-th target is calculated using formula (2).

[0050] α_ij=softmax(e ij ) (2)

[0051] For example, the relationship from target 1 to target 3 is r_13, and the corresponding relationship value is α_13. The relationship from target 6 to target 8 is r_68, and the corresponding relationship value is α_68.

[0052] In step 303, a second relationship r_dk associated with the first relationship r_ij is selected from the relationship set, where the first relationship r_ij is a relationship from the i-th target to the j-th target, and the second relationship r_dk is a relationship from the d-th target to the k-th target, 1≤i, j, d, k≤N, N is the total number of targets, i≠j, d≠k, and the first relationship r_ij and the second relationship r_dk involve the same target.

[0053] In some embodiments, a RLP (Relationship Label Propagation) model is used to select a second relation r_dk associated with the first relation r_ij.

[0054] In some embodiments, the i-th target and the d-th target are the same, and the j-th target and the k-th target are different. In other words, if the first relation is r_ij, then the second relation is r_ik. That is, the first and second relations have the same subject.

[0055] For example, if the first relationship is r_12, the associated second relationships include r_13, r_14, r_17, and so on.

[0056] In some embodiments, the i-th target and the d-th target are different, and the j-th target and the k-th target are different. That is, if the first relation is r_ij, then the second relation is r_dj. That is, the first relation and the second relation have the same object.

[0057] For example, if the first relationship is r_18, the associated second relationships include r_48, r_68, and so on.

[0058] In step 304, the relationship value of the first relationship r_ij is updated by using the relationship value of the relationship between different targets involved in the first relationship r_ij and the second relationship r_dk.

[0059] In some embodiments, for the first relationship r_ij and the second relationship r_ik, the relationship value of the first relationship r_ij is updated according to the relationship value of the third relationship r_jk from the j-th target to the k-th target.

[0060] For example, if the first relationship is r_12 and the second relationship is r_14, the relationship value α_12 of the first relationship r_12 is updated using the relationship value α_24 of the relationship r_24.

[0061] In some embodiments, for the first relation r_ij and the second relation r_dj, the relation value of the first relation r_ij is updated according to the relation value of the fourth relation r_id from the i-th target to the d-th target.

[0062] For example, if the first relationship is r_68 and the second relationship is r_18, the relationship value α_68 of the first relationship r_68 is updated using the relationship value α_61 of the relationship r_61.

[0063] In some embodiments, as Figure 6 As shown, the relationships associated with relationship r_12 are r_14, r_17, and r_13. For relationships r_12 and r_14, the relationship value α_24 of relationship r_24 is fed back. For relationships r_12 and r_17, the relationship value α_27 of relationship r_27 is fed back. For relationships r_12 and r_13, the relationship value α_23 of relationship r_23 is fed back. Thus, the relationship value α_12 of relationship r_12 is updated using the relationship values α_24, α_27, and α_23.

[0064] like Figure 6 As shown, the relationships associated with relationship r_14 are r_18, r_24, and r_15. For relationships r_14 and r_18, the relationship value α_48 of relationship r_48 is fed back. For relationships r_14 and r_24, the relationship value α_12 of relationship r_12 is fed back. For relationships r_14 and r_15, the relationship value α_45 of relationship r_45 is fed back. Thus, the relationship value α_14 of relationship r_14 is updated using the relationship values α_48, α_12, and α_45.

[0065] like Figure 6 As shown, the relationships associated with relationship r_68 are r_18, r_63, and r_78. For relationships r_68 and r_18, the relationship value α_61 of relationship r_61 is fed back. For relationships r_68 and r_63, the relationship value α_83 of relationship r_83 is fed back. For relationships r_68 and r_78, the relationship value α_67 of relationship r_67 is fed back. Thus, the relationship value α_68 of relationship r_68 is updated using the relationship values α_61, α_83, and α_67.

[0066] In step 305 , a scene graph is generated using all the objects and the updated relationship values.

[0067] In the scene graph generation method provided in the above embodiment of the present disclosure, by utilizing the information transfer between relationships with the same subject or the same object, the heterogeneity and homogeneity between objects and relationships in complex scenes can be effectively identified, thereby generating a robust scene graph.

[0068] In some embodiments, an HMP (Heterophily-aware Message Passing) model is used to detect occlusion relationships between different objects.

[0069] In some embodiments, if relations r_ij and r_ik have the same subject, a determination is made as to whether the overlap ratio between the j-th target and the k-th target is less than a preset threshold. If the overlap ratio between the j-th target and the k-th target is less than the preset threshold, the relation value of relation r_jk is multiplied by a first weight to obtain a first feedback value, where the first weight is a positive number. For example, the first weight is 1. The first feedback value is then used to update the relation value of relation r_ij.

[0070] If the overlap ratio between the jth target and the kth target is not less than a preset threshold, the relationship value of the relationship r_jk is multiplied by a second weight to obtain a second feedback value, where the second weight is a negative number. For example, the second weight is -1. The relationship value of the relationship r_ij is then updated using the second feedback value.

[0071] It should be noted that the high overlap between the jth and kth targets interferes with scene graph generation. Research has shown that adding a negative weight to the relationship r_jk can mitigate the degradation in scene graph generation quality caused by heterogeneity.

[0072] In some embodiments, if relations r_ij and r_dj have the same object, a determination is made as to whether the overlap ratio between the i-th target and the d-th target is less than a preset threshold. If the overlap ratio between the i-th target and the d-th target is less than the preset threshold, the relation value of relation r_id is multiplied by a third weight to obtain a third feedback value, where the third weight is a positive number. For example, the third weight is 1. Next, the relation value of relation r_ij is updated using the third feedback value.

[0073] If the overlap ratio between the i-th target and the d-th target is not less than a preset threshold, the relationship value of the relationship r_id is multiplied by a fourth weight to obtain a fourth feedback value, where the fourth weight is a negative number. For example, the fourth weight is -1. Next, the relationship value of the relationship r_ij is updated using the fourth feedback value.

[0074] In some embodiments, as Figure 7 As shown, the relations associated with relation r_12 are r_14, r_17, and r_13. Figure 7 and Figure 6 The difference is that in Figure 7In the example, since the overlap ratio between target 2 and target 3 is large, the feedback value is -α_23. Therefore, the relationship value α_12 of the relationship r_12 is updated using the relationship values α_24, α_27, and -α_23.

[0075] like Figure 7 As shown, the relations associated with relation r_14 are r_18, r_24, and r_15. Figure 7 and Figure 6 The difference is that in Figure 7 In the example, since the overlap ratio between target 4 and target 5 is large, the feedback value is -α_45. Therefore, the relationship value α_14 of the relationship r_14 is updated using the relationship values α_48, α_12, and -α_45.

[0076] like Figure 7 As shown, the relations associated with relation r_68 are r_18, r_63, and r_78. Figure 7 and Figure 6 The difference is that in Figure 7 In the example, since the overlap ratio between target 6 and target 7 is large, the feedback value is -α_67. Therefore, the relationship value α_68 of the relationship r_68 is updated using the relationship values α_61, α_83, and -α_67.

[0077] For example, by Figure 4 The image shown in the figure is processed and a total of 8 visual targets are identified. The corresponding scene diagram is as follows Figure 9 shown.

[0078] Figure 10 FIG. 1 is a schematic diagram of the structure of a scene graph generating device according to an embodiment of the present disclosure. Figure 8 As shown, the scene graph generating apparatus includes a first processing module 101 , a second processing module 102 , a third processing module 103 and a fourth processing module 104 .

[0079] The first processing module 101 is configured to recognize the input image to identify all objects in the input image.

[0080] For example, by using Faster RCNN, Figure 4 The target in is identified. The recognition result is as follows Figure 5 shown.

[0081] The second processing module 102 is configured to generate a relationship set, where the relationship set includes the relationship between any two targets in all targets.

[0082] In some embodiments, such as the ART model, a relationship between any two targets is determined.

[0083] The third processing module 103 is configured to select a second relationship r_dk associated with the first relationship r_ij from the relationship set, wherein the first relationship r_ij is a relationship from the i-th target to the j-th target, the second relationship r_dk is a relationship from the d-th target to the k-th target, 1≤i,j,d,k≤N, N is the total number of targets, i≠j, d≠k, and the first relationship r_ij and the second relationship r_dk involve a same target; and update the relationship value of the first relationship r_ij by using the relationship value between different targets involved in the first relationship r_ij and the second relationship r_dk.

[0084] In some embodiments, the second relation r_dk associated with the first relation r_ij is selected using an RLP model.

[0085] In some embodiments, the i-th target and the d-th target are the same, and the j-th target and the k-th target are different. In other words, if the first relation is r_ij, then the second relation is r_ik. That is, the first and second relations have the same subject.

[0086] For example, if the first relationship is r_12, the associated second relationships include r_13, r_14, r_17, and so on.

[0087] In some embodiments, the i-th target and the d-th target are different, and the j-th target and the k-th target are different. That is, if the first relation is r_ij, then the second relation is r_dj. That is, the first relation and the second relation have the same object.

[0088] For example, if the first relationship is r_18, the associated second relationships include r_48, r_68, and so on.

[0089] In some embodiments, for the first relationship r_ij and the second relationship r_ik, the relationship value of the first relationship r_ij is updated according to the relationship value of the third relationship r_jk from the j-th target to the k-th target.

[0090] In some embodiments, for the first relation r_ij and the second relation r_dj, the relation value of the first relation r_ij is updated according to the relation value of the fourth relation r_id from the i-th target to the d-th target.

[0091] The fourth processing module 104 is configured to generate a scene graph using all the targets and the updated relationship values.

[0092] In some embodiments, the HMP model is used to detect occlusion relationships between different objects.

[0093] In some embodiments, if relations r_ij and r_ik have the same subject, a determination is made as to whether the overlap ratio between the j-th target and the k-th target is less than a preset threshold. If the overlap ratio between the j-th target and the k-th target is less than the preset threshold, the relation value of relation r_jk is multiplied by a first weight to obtain a first feedback value, where the first weight is a positive number. For example, the first weight is 1. The first feedback value is then used to update the relation value of relation r_ij.

[0094] If the overlap ratio between the jth target and the kth target is not less than a preset threshold, the relationship value of the relationship r_jk is multiplied by a second weight to obtain a second feedback value, where the second weight is a negative number. For example, the second weight is -1. The relationship value of the relationship r_ij is then updated using the second feedback value.

[0095] In some embodiments, if relations r_ij and r_dj have the same object, a determination is made as to whether the overlap ratio between the i-th target and the d-th target is less than a preset threshold. If the overlap ratio between the i-th target and the d-th target is less than the preset threshold, the relation value of relation r_id is multiplied by a third weight to obtain a third feedback value, where the third weight is a positive number. For example, the third weight is 1. Next, the relation value of relation r_ij is updated using the third feedback value.

[0096] If the overlap ratio between the i-th target and the d-th target is not less than a preset threshold, the relationship value of the relationship r_id is multiplied by a fourth weight to obtain a fourth feedback value, where the fourth weight is a negative number. For example, the fourth weight is -1. Next, the relationship value of the relationship r_ij is updated using the fourth feedback value.

[0097] Figure 11 FIG. 1 is a structural diagram of a scene graph generating device according to another embodiment of the present disclosure. Figure 11 As shown, the scene graph generating apparatus includes a memory 111 and a processor 112 .

[0098] The memory 111 is used to store instructions. The processor 112 is coupled to the memory 111. The processor 112 is configured to execute the instructions stored in the memory. Figure 3 The method according to any one of the embodiments.

[0099] like Figure 11 As shown, the scene graph generation apparatus further includes a communication interface 113 for exchanging information with other devices and a bus 114 through which the processor 112 , the communication interface 113 , and the memory 111 communicate with each other.

[0100] Memory 111 may include high-speed RAM memory or non-volatile memory, such as at least one disk storage device. Memory 111 may also be a memory array. Memory 111 may also be divided into blocks, and the blocks may be combined into virtual volumes according to certain rules.

[0101] Furthermore, the processor 112 may be a central processing unit (CPU), or may be an application-specific integrated circuit (ASIC), or may be configured to implement one or more integrated circuits of the embodiments of the present disclosure.

[0102] The present disclosure also relates to a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, which, when executed by a processor, implement the following Figure 3 The method according to any one of the embodiments.

[0103] In some embodiments, as Figure 12 As shown in the figure, the Faster RCNN model is used to identify all objects in the input image. The ART model is then used to determine the relationship between any two objects in the total number of objects. Next, the RLP model is used to select the associated relationships between the object relationships and update the object relationship values using the corresponding relationship values. Furthermore, the HMP model is used to detect whether the overlap ratio between any two objects is too large. If the overlap ratio between two objects is too large, RLP will first negate the relationship values of the two objects before performing the corresponding update.

[0104] In some embodiments, the functional unit module described above can be implemented as a general-purpose processor, a programmable logic controller (PLC), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any appropriate combination thereof for performing the functions described in the present disclosure.

[0105] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0106] The description of the present disclosure is provided for purposes of illustration and description and is not intended to be exhaustive or to limit the disclosure to the disclosed form. Many modifications and variations will be apparent to those skilled in the art. The embodiments are selected and described in order to better illustrate the principles and practical applications of the present disclosure and to enable those skilled in the art to understand the present disclosure and design various embodiments with various modifications suitable for specific applications.

Claims

1. A method for generating a scene graph, comprising: Recognizing an input image to identify all objects in the input image; generating a relationship set, wherein the relationship set includes a relationship between any two targets among all the targets; In the relationship set, a second relationship r_dk associated with a first relationship r_ij is selected, wherein the first relationship r_ij is a relationship from the i-th target to the j-th target, the second relationship r_dk is a relationship from the d-th target to the k-th target, 1≤i,j,d,k≤N, N is the total number of targets, i≠j,d≠k, and the first relationship r_ij and the second relationship r_dk involve the same target; Update the relationship value of the first relationship r_ij by using the relationship value between different targets involved in the first relationship r_ij and the second relationship r_dk; generating a scene graph using all the targets and the updated relationship values; Wherein, when the i-th target and the d-th target are the same, and the j-th target and the k-th target are different, updating the relationship value of the first relationship r_ij includes: Update the relationship value of the first relationship r_ij according to the relationship value of the third relationship r_jk from the j-th target to the k-th target; When the i-th target and the d-th target are different and the j-th target and the k-th target are the same, updating the relationship value of the first relationship r_ij includes: According to the relationship value of the fourth relationship r_id from the i-th target to the d-th target, the relationship value of the first relationship r_ij is updated.

2. The method according to claim 1, wherein Updating the relationship value of the first relationship r_ij according to the relationship value of the third relationship r_jk includes: Determining whether the overlap ratio between the j-th target and the k-th target is less than a preset threshold; If the overlap ratio between the j-th target and the k-th target is less than a preset threshold, multiplying the relationship value of the third relationship r_jk by a first weight to obtain a first feedback value, wherein the first weight is a positive number; The relationship value of the first relationship r_ij is updated using the first feedback value.

3. The method according to claim 2, further comprising: If the overlap ratio between the j-th target and the k-th target is not less than a preset threshold, multiplying the relationship value of the third relationship r_jk by a second weight to obtain a second feedback value, wherein the second weight is a negative number; The relation value of the first relation r_ij is updated using the second feedback value.

4. The method according to any one of claims 1 to 3, wherein Updating the relationship value of the first relationship r_ij according to the relationship value of the fourth relationship r_id includes: Determine whether the overlap ratio between the i-th target and the d-th target is less than a preset threshold; If the overlap ratio between the i-th target and the d-th target is less than a preset threshold, multiplying the relationship value of the fourth relationship r_id by a third weight to obtain a third feedback value, wherein the third weight is a positive number; The relation value of the first relation r_ij is updated using the third feedback value.

5. The method according to claim 4, further comprising: If the overlap ratio between the i-th target and the d-th target is not less than a preset threshold, multiplying the relationship value of the fourth relationship r_id by a fourth weight to obtain a fourth feedback value, wherein the fourth weight is a negative number; The relation value of the first relation r_ij is updated using the fourth feedback value.

6. A scene graph generating apparatus, comprising: A first processing module is configured to recognize an input image to identify all objects in the input image; A second processing module is configured to generate a relationship set, wherein the relationship set includes a relationship between any two targets among all the targets; The third processing module is configured to select a second relationship r_dk associated with the first relationship r_ij from the relationship set, wherein the first relationship r_ij is a relationship from the i-th target to the j-th target, the second relationship r_dk is a relationship from the d-th target to the k-th target, 1≤i,j,d,k≤N, N is the total number of targets, i≠j, d≠k, and the first relationship r_ij and the second relationship r_dk involve a same target; update the relationship value of the first relationship r_ij based on the relationship between the different targets involved in the first relationship r_ij and the second relationship r_dk New, wherein, when the i-th target and the d-th target are the same, and the j-th target and the k-th target are different, updating the relationship value of the first relationship r_ij includes: updating the relationship value of the first relationship r_ij according to the relationship value of the third relationship r_jk from the j-th target to the k-th target; when the i-th target and the d-th target are different, and the j-th target and the k-th target are the same, updating the relationship value of the first relationship r_ij includes: updating the relationship value of the first relationship r_ij according to the relationship value of the fourth relationship r_id from the i-th target to the d-th target; The fourth processing module is configured to generate a scene graph using all the targets and the updated relationship values.

7. A scene graph generating apparatus, comprising: a memory configured to store instructions; A processor is coupled to the memory, and the processor is configured to execute the method according to any one of claims 1 to 5 based on instructions stored in the memory.

8. A non-transitory computer-readable storage medium, wherein: The computer-readable storage medium stores computer instructions, and when the instructions are executed by a processor, the method according to any one of claims 1 to 5 is implemented.

Images