A traffic scene graph generation method integrating prior knowledge

Abstract

The present invention discloses a method for generating a traffic scene graph by integrating prior knowledge, comprising the following steps: inputting an image into a target detection network, obtaining the spatial features F of the target, S ; Store the offline triple database in the on-board computer memory, use the semantic encoder to obtain the high-dimensional triple G, and use the relationship probability space as prior knowledge; reason the relationship r between the target pairs p , and the relationship with the highest probability value in the relationship probability space #imgabs0# is used as the final relationship of the target pair. The present invention uses prior knowledge to filter out unrelated target pairs in traffic scene images, achieve dimensionality reduction in the reasoning process, reduce the computing resources occupied by reasoning about relationships between unrelated targets, improve the overall real-time performance of the algorithm, and improve the adaptability of the algorithm in vehicle-mounted embedded platforms. By integrating prior knowledge into the reasoning process, the present invention provides a priori bias for the reasoning process, alleviates the phenomenon that the reasoning process tends to be head-category-oriented, and improves the rationality of the network reasoning results.

Description

Technical Field

[0001] The present invention relates to the field of scene understanding, and in particular to a method for generating a traffic scene graph based on computer vision. Background Art

[0002] With the development of computer vision technology, visual scene understanding has received widespread attention, especially in the field of autonomous driving. Using computer vision technology to achieve environmental perception of traffic scenes has become a mainstream method. Among them, target detection is an important visual scene understanding task. The target detection algorithm can provide the targets contained in the image and the spatial location of the targets, providing important information for scene understanding. However, the target detection algorithm aims to infer the targets and positions in the image, but cannot obtain deep information such as the relationship between targets. For complex scenes such as traffic scenes, it is not only necessary to obtain the targets that appear in the scene, but also to understand the relationship between targets. This can provide important information basis for predicting the behavior of the targets and provide an important foundation for subsequent decision-making and path planning. In order to further improve the depth of traffic scene understanding, scene graph generation technology has emerged. Scene graph generation technology can not only provide the visual relationship between target pairs in the image, but also provide structured scene representation for advanced visual understanding tasks.

[0003] Currently, mainstream scene graph generation algorithms construct any two objects appearing in an image into object pairs and then infer the relationship between each object pair. However, when faced with complex scenes, especially complex traffic scenes, the number of objects appearing in the image is diverse and the number of object pairs grows exponentially. Therefore, more computing resources are required and cannot meet the real-time requirements of low-cost in-vehicle embedded devices. In addition, the relationship distribution in the scene graph has a significant long tail phenomenon, which will make the reasoning network's reasoning results more inclined to the head category, resulting in some unreasonable relationship reasoning results. Summary of the Invention

[0004] In view of the fact that the current traffic scene graph generation method has low real-time performance, cannot adapt to the long-tail problem in relationship distribution, and has the phenomenon of unreasonable relationship reasoning, the present invention proposes a traffic scene graph generation method that integrates prior knowledge, incorporates prior knowledge into the reasoning process, alleviates the phenomenon that the reasoning process tends to be biased towards the head category, and uses prior knowledge to provide constraints for the reasoning process, thereby improving the rationality of the reasoning results.

[0005] To achieve the above objectives, the technical solution of the present invention is as follows: A method for generating a traffic scene graph integrating prior knowledge comprises the following steps:

[0006] A: Preprocessing images

[0007] Use the vehicle-mounted camera to obtain traffic scene images, input the images into the target detection network, and obtain the category semantic information of the targets in the images. E , the visual information of the target I V and the spatial information of the target I S , use the semantic encoder to encode the category semantic information of the target and obtain the semantic feature F of the target E ; Input the image area within the target detection frame into the convolutional neural network to extract the visual features F of the target within the target detection frame V ; Use the normalization method to normalize the target space position coordinates and obtain the target's spatial features F S .

[0008] B: Acquiring prior knowledge

[0009] The offline triple database is stored in the on-board computer memory. The semantic encoder is used to encode the semantic information of the triples in the offline triple database. The encoded semantic information is input into the embedding layer network to obtain the high-dimensional triple G:

[0010] G={h,r,t}

[0011] Where h is the head entity word vector, r is the relation word vector, and t is the tail entity word vector. The head entity word vector and the tail entity word vector of each pair of triples in the offline triple database are input into the word embedding representation model to predict the relation word vector, and the following loss function is constructed:

[0012] loss = r-KGE(h,t)

[0013] Where KGE(·) is the word embedding representation model.

[0014] By training the word embedding representation model, the interaction features between triplets are captured, thereby realizing the modeling of triples; in addition, the constructed word embedding representation model infers the relationship probability space between the head entity and the tail entity. Among them, r i h,t It represents the probability of the i-th relationship between two entities when the head entity word vector is h and the tail entity word vector is t. K is the type of relationship word vector. The relationship probability space is used as prior knowledge to provide prior guidance for subsequent online relationship reasoning.

[0015] C: Reasoning target pair relationship

[0016] Any two targets appearing in the image are constructed into a target pair, and the semantic features F of the two targets in the target pair are E Input into the word embedding representation model to obtain the relationship probability space of the target pair And set the threshold TH , when the maximum value r in the relationship probability space max Greater than the threshold T H When , the relationship between the target pair is set to be related; when the maximum value r in the relationship probability space max Less than the threshold T H When , the relationship between the target pairs is set to be unrelated to achieve dimensionality reduction in the reasoning process; the concatenation algorithm concat(·) is used to concatenate the visual features, semantic features, spatial features and relationship probability space of the related target pairs to form a multimodal feature F M , the multimodal feature F M Input into the inference network composed of fully connected layers, inferring the relationship r between the target pairs p , the relationship probability space Sort by probability value from high to low, and only retain the top ten relationship probabilities to form a new relationship probability space To reduce the occurrence of unreasonable relationships, the output of the inference network is limited to In the space, if the output of the inference network exceeds space, the inference result is considered to be an unreasonable relationship, and the relationship probability space is used The relationship with the highest probability value is taken as the final relationship of the target pair.

[0017] Compared with the prior art, the present invention has the following beneficial effects:

[0018] 1. The present invention uses prior knowledge to filter out unrelated object pairs in traffic scene images, achieve dimensionality reduction in the reasoning process, reduce the computing resources occupied by reasoning about the relationships between unrelated objects, improve the overall real-time performance of the algorithm, and improve the adaptability of the algorithm in vehicle-mounted embedded platforms.

[0019] 2. The present invention integrates prior knowledge into the reasoning process to provide a priori bias for the reasoning process, thereby alleviating the phenomenon that the reasoning process tends to favor the head category; at the same time, prior knowledge also provides constraints for the relational reasoning network, thereby improving the rationality of the network reasoning results. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 Flowchart of the present invention. DETAILED DESCRIPTION

[0021] The present invention will be further described below with reference to the accompanying drawings. Figure 1 As shown, a method for generating a traffic scene graph integrating prior knowledge includes the following steps:

[0022] A: Image Preprocessing

[0023] Use the vehicle-mounted camera to obtain traffic scene images, input the images into the Yolov5 or SSD network, and obtain the category semantic information of the target in the image. E , the visual information of the target I V and the spatial information of the target I S , use one-hot semantic encoder or index encoder to encode the category semantic information of the target and obtain the semantic feature F of the target E ; Input the image area within the target detection frame into the visual feature extraction network VGG to extract the visual features F of the target within the target detection frame E ; Use the normalization method to normalize the target space position coordinates and obtain the target's spatial features F S , which can be summarized into the following formula:

[0024]

[0025] B: Prior knowledge acquisition

[0026] The offline triple database is stored in the on-board computer memory, and the semantic information of the triples in the offline database is encoded using a one-hot semantic encoder. The encoded semantic information is input into the embedding layer network to obtain the high-dimensional triple G.

[0027] G={h,r,t}

[0028] Where h is the head entity word vector, r is the relation word vector, and t is the tail entity word vector. The head entity word vector and the tail entity word vector of each triple in the offline database are input into the word embedding representation model TransH or TransR to predict the relation word vector and construct the loss function:

[0029] loss = r-KGE(h,t)

[0030] Where KGE(·) is the word embedding representation model.

[0031] By training the word embedding representation model, the interaction features between triplets can be captured, thereby realizing the modeling of triples. In addition, the constructed word embedding representation model can infer the probability space of the relationship between the head entity and the tail entity based on the head entity and the tail entity. in, It represents the probability of the i-th relationship between two entities when the head entity word vector is h and the tail entity word vector is t. The relationship probability space is used as prior knowledge to provide prior guidance for subsequent online relationship reasoning.

[0032] C: Target-to-Relation Reasoning

[0033] Any two targets appearing in the image are constructed into a target pair, and the semantic features F of the two targets in the target pair areE Input into TransH or TransR to obtain the relationship probability space of the target pair And set the threshold T H , when the maximum value r in the relationship probability space max Greater than the threshold T H When , the relationship between the target pair is set to be related; when the maximum value r in the relationship probability space max Less than the threshold T H When , the relationship between the target pairs is set to be unrelated to achieve dimensionality reduction in the reasoning process; the concatenation algorithm concat(·) is used to concatenate the visual features, semantic features, spatial features and relationship probability space of the related target pairs to form a multimodal feature F M :

[0034]

[0035] The multimodal feature F M Input into the inference network composed of fully connected layers, inferring the relationship r between the target pairs p :

[0036] r p =FC(F M )

[0037] Where FC is the inference network composed of fully connected layers.

[0038] The relation probability space Sort by probability value from high to low, and only retain the top ten relationship probabilities to form a new relationship probability space To reduce the occurrence of unreasonable relationships, the output of the inference network is limited to In the space, if the output of the inference network exceeds space, the inference result is considered to be an unreasonable relationship, and the relationship probability space is used The relationship with the highest probability value is taken as the final relationship of the target pair.

[0039] The present invention is not limited to this embodiment, and any equivalent concepts or modifications within the technical scope disclosed by the present invention are included in the protection scope of the present invention.

Claims

1. A method for generating a traffic scene graph by integrating prior knowledge, characterized by: The following steps are involved: A: Preprocessing images Use the vehicle-mounted camera to obtain traffic scene images, input the images into the target detection network, and obtain the category semantic information of the targets in the images. E , the visual information of the target I V and the spatial information of the target I S , use the semantic encoder to encode the category semantic information of the target and obtain the semantic feature F of the target E ; Input the image area within the target detection frame into the convolutional neural network to extract the visual features F of the target within the target detection frame V ; Use the normalization method to normalize the target space position coordinates and obtain the target's spatial features F S ; B: Acquiring prior knowledge The offline triple database is stored in the on-board computer memory. The semantic encoder is used to encode the semantic information of the triples in the offline triple database. The encoded semantic information is input into the embedding layer network to obtain the high-dimensional triple G: G={h,r,t} Where h is the head entity word vector, r is the relation word vector, and t is the tail entity word vector. The head entity word vector and the tail entity word vector of each pair of triples in the offline triple database are input into the word embedding representation model to predict the relation word vector, and the following loss function is constructed: loss = r-KGE(h,t) Where KGE(·) is the word embedding representation model; By training the word embedding representation model, the interaction features between triplets are captured, thereby realizing the modeling of triples; in addition, the constructed word embedding representation model infers the relationship probability space between the head entity and the tail entity. in, It represents the probability of the i-th relationship between two entities when the head entity word vector is h and the tail entity word vector is t, and K is the type of relationship word vector; The relational probability space is used as prior knowledge to provide prior guidance for subsequent online relational reasoning; C: Reasoning target pair relationship Any two targets appearing in the image are constructed into a target pair, and the semantic features F of the two targets in the target pair are E Input into the word embedding representation model to obtain the relationship probability space of the target pair And set the threshold T H , when the maximum value r in the relationship probability space max Greater than the threshold T H When , the relationship between the target pair is set to be related; when the maximum value r in the relationship probability space max Less than the threshold T H When , the relationship between the target pairs is set to be unrelated to achieve dimensionality reduction in the reasoning process; the concatenation algorithm concat(·) is used to concatenate the visual features, semantic features, spatial features and relationship probability space of the related target pairs to form a multimodal feature F M , the multimodal feature F M Input into the inference network composed of fully connected layers, inferring the relationship r between the target pairs p , the relationship probability space Sort by probability value from high to low, and only retain the top ten relationship probabilities to form a new relationship probability space To reduce the occurrence of unreasonable relationships, the output of the inference network is limited to In the space, if the output of the inference network exceeds space, the inference result is considered to be an unreasonable relationship, and the relationship probability space is used The relationship with the highest probability value is taken as the final relationship of the target pair.

Images