Scene graph skeleton construction method under reinforcement learning framework

Abstract

The present invention discloses a method for constructing a scene graph skeleton under a reinforcement learning framework, comprising: generating a graph structure based on a Markov decision process; constructing a graph structure generation environment; constructing a graph convolutional strategy network, and training the graph convolutional strategy network, iteratively adding nodes and edges to candidate subgraphs in a relational perception environment, and ultimately forming a scene graph skeleton of the image. The present invention utilizes the reinforcement learning framework to obtain a sparse scene graph skeleton, and uses the scene graph skeleton establishment process as a Markov decision process. A sparse scene graph skeleton is obtained by reinforcement learning, and a sparse scene graph skeleton with high information density and low complexity is constructed, providing a basis for accurate scene graph skeleton construction.

Description

Technical Field

[0001] The present invention belongs to the field of reinforcement learning technology, and in particular relates to a method for constructing a scene graph skeleton under a reinforcement learning framework. Background Art

[0002] An image scene graph is a directed graph consisting of nodes and edges. Nodes are categorized into three types: object categories, object attributes, and relationships between objects. Directed edges indicate the direction of interaction between nodes. By constructing an image scene graph, high-level semantic information in an image can be accurately represented in a structured manner, enabling direct computer storage and analysis. Compared to image description, the construction of a scene graph represents an upgrade in image description methods in two key aspects. First, a scene graph consists of many candidate regions, each of which contains high-level semantic information about object categories and attributes. Relationships between different objects are essential nodes in the scene graph. Unlike image descriptions, which often underfit the representation of relationships between objects, scene graphs enable a dense representation of relationships between objects in an image. Second, scene graphs are structured data, resolving the problem of image descriptions being limited to semi-structured text.

[0003] The current method for constructing scene graphs usually involves pre-building a fully connected scene graph skeleton, and then further classifying objects and relationships. The fully connected nature of the skeleton means that it contains a large amount of redundant information. Especially in real and complex scenes, this redundant information not only increases the complexity of the entire algorithm, but also affects the accuracy of object and relationship classification. How to establish a sparse scene graph skeleton with high information content and low complexity is the basis for establishing an efficient and accurate scene graph construction model in theory and engineering practice. Therefore, the construction of a sparse scene graph skeleton with high information content and low complexity is a key scientific problem to be solved in this field. Summary of the Invention

[0004] In view of this, the present invention proposes the problem of obtaining a sparse scene graph skeleton using a reinforcement learning framework, attempts to regard the process of establishing a scene graph skeleton as a Markov decision process, and uses reinforcement learning to obtain a sparse scene graph skeleton, providing a theoretical basis for the construction of an accurate scene graph skeleton.

[0005] The method for constructing a scene graph skeleton under a reinforcement learning framework disclosed in the present invention comprises the following steps:

[0006] Generate a graph structure based on a Markov decision process;

[0007] Build a graph structure generation environment;

[0008] Construct a graph convolutional policy network and train it. In a relation-aware environment, it continuously adds nodes and edges to the candidate subgraph in an iterative manner, eventually forming the scene graph skeleton of the image.

[0009] Furthermore, generating a graph structure based on a Markov decision process includes:

[0010] The generation process of the graph structure is expressed as a general decision process M = (S, A, P, R, γ), where S = {s i} is a state set consisting of all possible intermediate graphs and final graphs, A={a i} is a set of actions that describe the modification of the current graph structure at each moment, P = p(s t+1 |s t ,s t-1 ,…,s0,a t ) is the state transition distribution, which represents the probability of the result of executing an action, where s0,…s t ,s t+1 is the graph structure at time 0,…t,t+1, a t is the action at time t, R(s t ) is the designed reward function, indicating that the graph structure reaches state s t The reward after γ is the discount factor used to reduce the impact of future rewards on the current action;

[0011] The process of graph structure generation is represented as an iterative trajectory (s0, a0, r0, ..., s n ,a n ,r n ), where s n is the final generated graph structure, a n is the final generated action, r n The final generated reward;

[0012] The graph structure will increase the edges between nodes in each iteration. The state transition distribution of the graph structure after adding edges is expressed as:

[0013]

[0014] Among them, p(a t |s t ,…,s0) is the policy network π θ ,π θ It is a graph convolution strategy network that uses the graph structure generation process of the Markov decision process. In this process, the state transition distribution is required to satisfy the Markov property, that is, p(s t+1 |s t ,s t-1 ,…,s0,a t )=p(s t+1 |s t ,a t ); Under this property, the policy network πθ Taking the current graph structure s t The next step of execution is generated as input, that is, determining which two nodes should be connected or determining that the entire generation process stops.

[0015] Furthermore, the construction of the graph structure generation environment includes:

[0016] Under the graph structure generation strategy based on the Markov decision process, the environment gradually builds the scene graph skeleton in an iterative manner through the actions given by the policy network. In each iterative step, there are five components, namely state representation, policy network, action, state transition distribution and reward;

[0017] The state space is the state of the environment s t Defined as the graph structure G generated after the tth iteration t , each update of the graph structure is controlled by the reinforcement learning agent;

[0018] Action space: First, define a candidate child node set C = {C1, C2, ..., C s}, the nodes in the set are continuously added to the candidate subgraph during the graph generation process; then, in the tth iteration, the extended graph is defined as the union of the candidate subgraph and the candidate subnode set, denoted as G t ∪ C , where G t is a candidate subgraph, and C is a candidate child node. The actions are divided into three types: 1) connecting two nodes in the candidate subgraph that were not connected in the previous iteration. After this action, the candidate child node set does not change; 2) connecting a specific node in the candidate subgraph with a node in the candidate child node set. In this case, the connected nodes in the candidate child node set are removed; 3) connecting two specific nodes in the candidate child node set. In this case, the two connected nodes are removed from the candidate child node set.

[0019] State transition distribution: Incorporating domain-specific action rules into the state transition distribution. For scene graph skeleton generation tasks, the environment incorporates the connection rules between objects in the dataset. This connection rule is derived by counting all connections in the dataset. If there is no connection between objects in the dataset, this result is considered a connection rule.

[0020] Reward function: In the scene graph skeleton generation environment, the actions of the reinforcement learning agent are guided by two types of reward functions, namely intermediate rewards and final rewards; wherein, the intermediate rewards include domain-specific rule rewards and adversarial rewards. If the action does not violate the graph construction rules, a small amount of positive rewards are assigned based on the relationship statistics in the dataset, otherwise a small amount of negative rewards are assigned; the final reward is defined as the sum of the adversarial reward and the scene graph accuracy reward, wherein the scene graph accuracy reward is obtained from the recall rate of the evaluation index in the scene graph classification task, and the generative adversarial network is used to define the adversarial reward V(π θ ,D φ ).

[0021] Furthermore, the construction process of the generative adversarial network is expressed as:

[0022]

[0023] Among them, π θ is the strategic network; p data Defines the data distribution of the final graph or the intermediate graph, the final graph is used for the final reward, and the intermediate graph is used for the intermediate reward; D φ is the discriminator network, x represents the input graph structure, D φ (x)∈[0,1] is the output of the discriminant model, which is used to judge the rationality of the graph structure; E represents the mathematical expectation; since x is a graph data that is not differentiable with respect to the parameter φ, only the discriminator D φ Stochastic gradient descent is used for training, and for the policy network, the policy gradient method is introduced to train it.

[0024] Furthermore, in the scene graph skeleton generation environment, a graph convolutional strategy network is used to calculate the state transition probability caused by the action. The graph convolutional strategy network includes:

[0025] The input of the graph convolutional policy network is the candidate subgraph and candidate subnode set, and the output is the action to be taken in the current iteration step. The final structure and parameters of the network are the learning goals of the reinforcement learning agent;

[0026] Node embedding feature calculation: In order to be able to t ∪ C In the prediction of the action of the current iteration step, the graph convolutional network is used to calculate the embedding features of the node; through the L-layer graph convolutional network, a total of L layers of message passing are performed on each connection in the graph; at the lth layer of the network, the embedding process of the node is as follows:

[0027]

[0028] in, A is the adjacency matrix of the graph structure, I is the identity matrix with 1 on the diagonal and 0 on the rest of the diagonal. D is the degree matrix of the graph structure, W (l) is the parameter for training the lth layer of the graph convolutional network, H (0) is the deep feature of the object node, which is the output of the ROI pooling layer of the Faster R-CNN network, X = H (L) , H (l) ,H (l+1) are the deep features output by the neural network nodes in the lth and l+1th layers respectively;

[0029] Action prediction: At the tth iteration step, the action prediction is the concatenation of three components: the selection of two nodes and the prediction of termination, denoted as a t =CONCAT(a start ,a end ,a stop ), a start and a end are the two selected nodes, a stop is the terminal node, each component is sampled by the prediction distribution based on the neural network representation, when a stop =1, the iterative process of the graph structure in the reinforcement learning environment ends, and the current graph structure is the generated initial scene graph skeleton.

[0030] Furthermore, policy gradient training includes:

[0031] Proximal policy optimization (PPO) is used to train the graph convolutional policy network. The objective function of PPO is defined as follows:

[0032]

[0033] Among them, π θold (a t |s t ) is the policy network before update, π θ (a t |s t ) is the updated policy network, r t (θ) is the ratio of the policy network output before and after the network update, and is truncated to the interval [1-ε,1+ε]. Its function is to convert L CLIP (θ) becomes the lower limit of the conservative strategy iteration target, clip is an operator, which means that if the first item is less than the second item, the final value is the second item, if the first item is greater than the third item, the final value is the third item, and if it is between the two, it keeps its own value; ε is a hyperparameter that needs to be adjusted during training, A t is the advantage function, which describes the tThe advantage of taking a random action under the action is that the value function of the evaluation action is the discriminant function in the generative adversarial network. Its principle is to embed the graph first and then use the multi-layer perceptron to score.

[0034] The beneficial effects of the present invention are as follows:

[0035] The present invention uses the reinforcement learning framework to obtain the problem of sparse scene graph skeleton, regards the process of establishing the scene graph skeleton as a Markov decision process, uses reinforcement learning to obtain a sparse scene graph skeleton, and constructs a sparse scene graph skeleton with high information content and low complexity, providing a basis for the construction of an accurate scene graph skeleton. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 The graph structure generation flow chart of the present invention: (a) the environment state is defined as the union of the candidate graph and the candidate child nodes; (b) the graph convolutional strategy network obtains the embedded features of the node through message passing, and then generates the strategy π θ ; (c) By sampling the strategy, three components are obtained, including two NodeIDs and a stop sign Stop, which together constitute the actions of this iterative process; (d) the state transition distribution of the environment, including checking whether the action meets specific rules; (e) the state of the environment in the next iterative process; (f) the rewards in the environment, including intermediate rewards and final rewards. DETAILED DESCRIPTION

[0037] The present invention will be further described below with reference to the accompanying drawings, but the present invention is not limited in any way. Any changes or substitutions made based on the teachings of the present invention fall within the scope of protection of the present invention.

[0038] Scene graph skeleton construction method under reinforcement learning framework

[0039] This paper incorporates the establishment of an initial scene graph skeleton into the framework of reinforcement learning. It frames the problem of generating a random, complex, and non-differentiable scene graph skeleton as the problem of learning a reinforcement learning agent. This agent, as a type of graph convolutional network, can iteratively add nodes and edges to candidate subgraphs in a relation-aware environment, ultimately forming the image's scene graph skeleton. This paper involves generating graph structures based on a Markov decision process, designing and constructing a graph structure generation environment, and training a graph convolutional policy network.

[0040] (1) Graph structure generation based on Markov decision process

[0041] The key task of building a graph structure is how to specify an iterative generation process. This paper describes the generation process of the graph structure as a general decision process M = (S, A, P, R, γ), where S = {s i} is a state set consisting of all possible intermediate graphs and final graphs, A={a i} is a set of actions that describe the modifications made to the current graph structure at each moment, P = p(s t+1 |s t ,s t-1 ,…,s0,a t ) is the state transition distribution, which indicates the probability of a possible result of executing an action, R(s t ) is the designed reward function, indicating that the graph structure reaches state s t The reward after the action, γ is the discount factor, the purpose is to reduce the impact of future rewards on the current action. Then, the process of graph structure generation can be expressed as an iterative trajectory (s0, a0, r0, ..., s n ,a n ,r n ), where s n The graph structure is finally generated. In each iteration, the graph structure will increase the edges between nodes. The state transition distribution of the graph structure after adding edges can be expressed as:

[0042]

[0043] Among them, p(a t |s t ,…,s0) is usually represented as the policy network π θ , in the present invention π θ This invention designs a graph structure generation process that can be expressed as a Markov decision process. In this process, the state transition distribution is required to satisfy the Markov property, that is, p(s t+1 |s t ,s t-1 ,…,s0,a t )=p(s t+1 |s t ,a t ). Under this property, the policy network π θ Only the current graph structure s is needed t Generate the next execution action for the input, that is, determine which two nodes should be connected or determine that the entire generation process stops.

[0044] (2) Graph structure generation environment design

[0045] Under the graph structure generation strategy based on Markov decision process, the environment gradually builds the scene graph skeleton in an iterative manner through the actions given by the policy network. Figure 1The five main components that play a role in each iteration step are explained, namely the state representation, policy network, action, state transition distribution, and reward.

[0046] Figure 1 Generate a flowchart for the graph structure, where each row corresponds to an iterative step in the generation process. (a) The environment state is defined as the union of the candidate graph and the candidate child nodes; (b) The graph convolutional policy network obtains the embedded features of the nodes through message passing and then generates the policy π θ ; (c) By sampling the strategy, three components are obtained, including two NodeIDs and a stop sign Stop, which together constitute the actions of this iterative process; (d) the state transition distribution of the environment, including checking whether the action meets specific rules; (e) the state of the environment in the next iterative process; (f) the rewards in the environment, including intermediate rewards and final rewards.

[0047] State space: The state s of the environment t Defined as the graph structure G generated after the tth iteration t , each update of the graph structure is controlled by the reinforcement learning agent. Figure 1 (a)(e) describe the state of the graph structure before and after the action is updated in each iteration.

[0048] Action space: First, define a candidate child node set C = {C1, C2, ..., C s}, the nodes in the set are continuously added to the candidate subgraph during the graph generation process; then, in the tth iteration process, the extended graph is defined as the union of the candidate subgraph and the candidate child node set, which can be expressed as G t ∪C. Under this definition, actions can be divided into three types: 1) Connecting two nodes in the candidate subgraph that were not connected in the previous iteration, and the candidate subnode set does not change after this action; 2) Connecting a specific node in the candidate subgraph with a node in the candidate subnode set, and removing the connected nodes from the candidate subnode set; 3) Connecting two specific nodes in the candidate subnode set; in this case, removing the connected two nodes from the candidate subnode set. Figure 1 In (b), the green and yellow nodes are connected at time t+1, so the yellow node is removed from the candidate child node set and becomes part of the candidate subgraph.

[0049] State transition distribution: Incorporate domain-specific action rules into state transition distribution. For scene graph skeleton generation tasks, the environment combines the connection rules between objects in the dataset. This connection rule is obtained after counting all connections in the dataset. For example, if there is no connection between "cat" and "frisbee" in the VG dataset, then this result will be regarded as a connection rule. Figure 1 In (d), both actions pass the rule check and the environment updates its state based on these actions.

[0050] Reward function design: In the scene graph skeleton generation environment, the actions of the reinforcement learning agent are guided by two types of reward functions, namely intermediate rewards and final rewards. Among them, the intermediate rewards include domain-specific rule rewards and adversarial rewards. If the action does not violate the graph construction rules, a small amount of positive rewards are assigned based on the relationship statistics in the data set, otherwise a small amount of negative rewards are assigned; and the final reward is defined as the sum of the adversarial reward and the scene graph accuracy reward, wherein the scene graph accuracy reward can be obtained from the recall rate of the evaluation indicators in the scene graph classification task, such as Recall@50 and Recall@100. The present invention uses a generative adversarial network (GAN) to define the adversarial reward V(π θ ,D φ ). The construction process of generating adversarial can be expressed as:

[0051]

[0052] Among them, π θ is the strategic network; p data It defines the data distribution of the final graph (for final rewards) or the intermediate graph (for intermediate rewards); D φ is the discriminator network, whose structure is roughly the same as the policy network, x represents the input graph structure, D φ (x)∈[0,1] is the output of the discriminant model, which is used to judge the rationality of the graph structure; E represents the mathematical expectation. Since x is a graph data that is not differentiable with respect to the parameter φ, only the discriminator D φ Stochastic gradient descent can be used for training. For the policy network, the present invention introduces a policy gradient method to train it.

[0053] (3) Graph Convolutional Strategy Network

[0054] In the scene graph skeleton generation environment, a graph convolutional policy network is used to calculate the state transition probability caused by actions. The input of the graph convolutional policy network is a candidate subgraph and a set of candidate child nodes. The output is the action to be taken in the current iteration step. The final structure and parameters of the network are the learning goals of the reinforcement learning agent.

[0055] Node embedding feature calculation: In order to be able to t ∪ C To predict the action of the current iteration step, a graph convolutional network is used to calculate the node embedding features. Through the L-layer graph convolutional network, a total of L layers of message passing are performed on each connection in the graph. At the lth layer of the network, the node embedding process is as follows:

[0056]

[0057] in, A is the adjacency matrix of the graph structure, I is the identity matrix with 1 on the diagonal and 0 on the rest of the diagonal. D is the degree matrix of the graph structure, W (l) is the parameter for training the lth layer of the graph convolutional network, H (0) is the deep feature of the object node, which is the output of the ROI (Region Of Interests) pooling layer of the Faster R-CNN network, X = H (L) , H (l) ,H (l+1) are the deep features output by the neural network nodes in the lth and l+1th layers respectively.

[0058] Action prediction: At the tth iteration step, the action prediction is the concatenation of three components: the selection of two nodes and the prediction of termination, which can be expressed as a t =CONCAT(a start ,a end ,a stop Specifically, each component is sampled by the prediction distribution represented by the designed neural network. Figure 1 (c) shows that when a stop =1, the iterative process of the graph structure in the reinforcement learning environment ends, and the current graph structure is the generated initial scene graph skeleton.

[0059] (4) Policy Gradient Training

[0060] The network optimization method based on policy gradient is widely used by policy networks. This paper uses proximal policy optimization (PPO) to train the graph convolutional policy network. The objective function of PPO is defined as follows:

[0061]

[0062] Among them, r t (θ) is the ratio of the policy network output before and after the network update, and is truncated to the interval [1-ε,1+ε]. Its main function is to convert L CLIP (θ) becomes the lower limit of the conservative policy iteration target, A t is the advantage function (AdvantageFunction), which describes the state s t The action taken under the given condition has an advantage over taking a random action, and the value function for evaluating the action is the discriminant function in the generative adversarial network. Its principle is to embed the graph first and then use a multi-layer perceptron to score.

[0063] The beneficial effects of the present invention are as follows:

[0064] The present invention uses the reinforcement learning framework to obtain the problem of sparse scene graph skeleton, regards the process of establishing the scene graph skeleton as a Markov decision process, uses reinforcement learning to obtain a sparse scene graph skeleton, and constructs a sparse scene graph skeleton with high information content and low complexity, providing a basis for the construction of an accurate scene graph skeleton.

[0065] As used herein, the word "preferred" is intended to serve as an example, instance, or illustration. Any aspect or design described herein as "preferred" is not necessarily to be construed as advantageous over other aspects or designs. Rather, the use of the word "preferred" is intended to present concepts in a concrete manner. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or." That is, unless otherwise specified or clear from the context, "X employs A or B" is intended to mean any of the naturally inclusive permutations. That is, if X employs A; X employs B; or X employs both A and B, then "X employs A or B" is satisfied in any of the foregoing examples.

[0066] Moreover, although the present disclosure has been shown and described with respect to one or implementation, those skilled in the art will think of equivalent variations and modifications based on reading and understanding of this specification and the accompanying drawings. The present disclosure includes all such modifications and variations and is limited only by the scope of the appended claims. In particular, with respect to the various functions performed by the above-mentioned components (such as elements, etc.), the terms used to describe such components are intended to correspond to any component (unless otherwise indicated) that performs the specified function of the component (such as it is functionally equivalent), even if structurally different from the disclosed structure that performs the function in the exemplary implementation of the present disclosure shown herein. In addition, although the specific features of the present disclosure have been disclosed with respect to only one of several implementations, such features can be combined with one or other features of other implementations that can be desired and advantageous for a given or specific application. Moreover, insofar as the terms "including", "having", "containing" or their variations are used in specific embodiments or claims, such terms are intended to be included in a manner similar to the term "comprising".

[0067] The functional units in the embodiments of the present invention may be integrated into a single processing module, or each unit may exist physically separately, or multiple or more units may be integrated into a single module. The aforementioned integrated module may be implemented in the form of hardware or in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium. The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk, etc. The aforementioned devices or systems may execute the storage method in the corresponding method embodiment.

[0068] In summary, the above embodiment is one implementation method of the present invention, but the implementation method of the present invention is not limited to the described embodiment. Any other changes, modifications, substitutions, combinations, and simplifications that deviate from the spirit and principles of the present invention should be equivalent replacement methods and are included in the scope of protection of the present invention.

Claims

1. A scene graph skeleton construction method under a reinforcement learning framework, characterized in that: The method comprises the following steps: Generate a graph structure based on a Markov decision process; Build a graph structure generation environment; Construct and train a graph convolutional policy network, iteratively adding nodes and edges to candidate subgraphs in a relation-aware environment to ultimately form a scene graph for the image. The construction of the graph structure generation environment includes: Under the graph structure generation strategy based on the Markov decision process, the environment gradually builds the scene graph skeleton in an iterative manner through the actions given by the policy network. In each iterative step, there are five components, namely state representation, policy network, action, state transition distribution and reward; State space: The state s of the environment t Defined as the graph structure G generated after the tth iteration t , each update of the graph structure is controlled by the reinforcement learning agent; Action space: First, define a candidate child node set C = {C1, C2, ..., C s }, the nodes in the set are continuously added to the candidate subgraph during the graph generation process; then, in the tth iteration, the extended graph is defined as the union of the candidate subgraph and the candidate subnode set, denoted as G t ∪C, where G t is a candidate subgraph, and C is a candidate child node. The actions are divided into three types: 1) connecting two nodes in the candidate subgraph that were not connected in the previous iteration. After this action, the candidate child node set does not change; 2) connecting a specific node in the candidate subgraph with a node in the candidate child node set. In this case, the connected nodes in the candidate child node set are removed; 3) connecting two specific nodes in the candidate child node set. In this case, the two connected nodes are removed from the candidate child node set. State transition distribution: Incorporating domain-specific action rules into the state transition distribution. For scene graph skeleton generation tasks, the environment incorporates the connection rules between objects in the dataset. This connection rule is derived by counting all connections in the dataset. If there is no connection between objects in the dataset, this result is considered a connection rule. Reward function: In the scene graph skeleton generation environment, the actions of the reinforcement learning agent are guided by two types of reward functions, namely intermediate rewards and final rewards; wherein, the intermediate rewards include domain-specific rule rewards and adversarial rewards. If the action does not violate the graph construction rules, a small amount of positive rewards are assigned based on the relationship statistics in the dataset, otherwise a small amount of negative rewards are assigned; the final reward is defined as the sum of the adversarial reward and the scene graph accuracy reward, wherein the scene graph accuracy reward is obtained from the recall rate of the evaluation index in the scene graph classification task, and the generative adversarial network is used to define the adversarial reward V(π θ ,D φ ).

2. The method for constructing a scene graph skeleton under a reinforcement learning framework according to claim 1, wherein: The generating of the graph structure based on the Markov decision process includes: The generation process of the graph structure is expressed as a general decision process M = (S, A, P, R, γ), where S = {s i } is a state set consisting of all possible intermediate graphs and final graphs, A={a i } is a set of actions that describe the modification of the current graph structure at each moment, P = p(s t+1 |s t ,s t-1 ,...,s0,a t ) is the state transition distribution, which represents the probability of the result of executing an action, where s0,…s t ,s t+1 is the graph structure at time 0,…t,t+1, a t is the action at time t, R(s t ) is the designed reward function, indicating that the graph structure reaches state s t The reward after γ is the discount factor used to reduce the impact of future rewards on the current action; The process of graph structure generation is represented as an iterative trajectory (s0, a0, r0, ..., s n ,a n ,r n ), where s n is the final generated graph structure, a n is the final generated action, r n The final generated reward; The graph structure will increase the edges between nodes in each iteration. The state transition distribution of the graph structure after adding edges is expressed as: Among them, p(a t |s t ,...,s0) is the policy network π θ ,π θ It is a graph convolution strategy network that uses the graph structure generation process of the Markov decision process. In this process, the state transition distribution is required to satisfy the Markov property, that is, p(s t+1 |s t ,s t-1 ,...,s0,a t )=p(s t+1 |s t ,a t ); Under this property, the policy network π θ Taking the current graph structure s t The next step of execution is generated as input, that is, determining which two nodes should be connected or determining that the entire generation process stops.

3. The method for constructing a scene graph skeleton under a reinforcement learning framework according to claim 1, wherein: in, The construction process of the generative adversarial network is expressed as: Among them, π θ is the strategic network; p data Defines the data distribution of the final graph or the intermediate graph, the final graph is used for the final reward, and the intermediate graph is used for the intermediate reward; D φ is the discriminator network, x represents the input graph structure, D φ (x)∈[0,1] is the output of the discriminant model, which is used to judge the rationality of the graph structure; E represents the mathematical expectation; since x is a graph data that is not differentiable with respect to the parameter φ, only the discriminator D φ Stochastic gradient descent is used for training, and for the policy network, the policy gradient method is introduced to train it.

4. The method for constructing a scene graph skeleton under a reinforcement learning framework according to claim 3, wherein: In the scene graph skeleton generation environment, a graph convolutional policy network is used to calculate the state transition probability caused by the action. The graph convolutional policy network includes: The input of the graph convolutional policy network is the candidate subgraph and candidate subnode set, and the output is the action to be taken in the current iteration step. The final structure and parameters of the network are the learning goals of the reinforcement learning agent; Node embedding feature calculation: In order to be able to t In ∪C, the action of the current iteration step is predicted, and the graph convolutional network is used to calculate the embedding features of the nodes. Through the L-layer graph convolutional network, a total of L layers of message passing are performed on each connection in the graph. At the lth layer of the network, the node embedding process is as follows: in, A is the adjacency matrix of the graph structure, I is the identity matrix with 1 on the diagonal and 0 on the rest of the diagonal. D is the degree matrix of the graph structure, W (l) is the parameter for training the lth layer of the graph convolutional network, H (0) is the deep feature of the object node, which is the output of the ROI pooling layer of the FasterR-CNN network, X = H (L) , H (l) ,H (l+1) are the deep features output by the neural network nodes in the lth and l+1th layers respectively; Action prediction: At the tth iteration step, the action prediction is the concatenation of three components: the selection of two nodes and the prediction of termination, denoted as a t =CONCAT(a start ,a end ,a stop ), a start and a end are the two selected nodes, a stop is the terminal node, each component is sampled by the prediction distribution based on the neural network representation, when a stop =1, the iterative process of the graph structure in the reinforcement learning environment ends, and the current graph structure is the generated initial scene graph skeleton.

5. The method for constructing a scene graph skeleton under a reinforcement learning framework according to claim 4, wherein: Policy gradient training includes: Proximal policy optimization (PPO) is used to train the graph convolutional policy network. The objective function of PPO is defined as follows: Among them, π θold (a t |s t ) is the policy network before update, π θ (a t |s t ) is the updated policy network, r t (θ) is the ratio of the policy network output before and after the network update, and is truncated to the interval [1-ε,1+ε]. Its function is to convert L CLIP (θ) becomes the lower limit of the conservative strategy iteration target, clip is an operator, which means that if the first item is less than the second item, the final value is the second item, if the first item is greater than the third item, the final value is the third item, and if it is between the two, it keeps its own value; ε is a hyperparameter that needs to be adjusted during training, A t is the advantage function, which describes the t The advantage of taking a random action under the action is that the value function of the evaluation action is the discriminant function in the generative adversarial network. Its principle is to embed the graph first and then use the multi-layer perceptron to score.

Images