Scene graph generation method and device for visual traffic scene

Abstract

The present invention provides a method and device for generating a scene graph for a visual traffic scene. The method comprises: performing semantic segmentation on a target traffic scene image, obtaining semantic objects in the target traffic scene image, and obtaining a scene representation of the target traffic scene image based on the semantic objects; constructing a directed graph of the target traffic scene image using the target traffic scene image, foreground objects and background objects in the semantic objects, and the scene representation as nodes, and using the relationships between the nodes as edges; locating visual areas in the target traffic scene image based on the directed graph, and annotating the relationships between semantic objects in each visual area based on the directed graph to obtain a scene graph corresponding to the target traffic scene image. The scene graph generated by the present invention facilitates higher-level scene understanding and improves the accuracy and safety of autonomous driving.

Description

Technical Field

[0001] The present invention relates to the field of unmanned driving technology, and in particular to a method and device for generating a scene graph of a visual traffic scene. Background Art

[0002] Visual perception of traffic scenes is one of the core technologies for autonomous driving. The visual perception system primarily uses cameras as sensor input and, through a series of calculations and processing, accurately perceives the vehicle's surroundings.

[0003] Currently, road traffic perception primarily encompasses three key areas: dynamic object detection (vehicles, pedestrians, and non-motorized vehicles), static object recognition (traffic signs and traffic lights), and drivable area segmentation (road area and lane markings). A scene graph representation is generated based on perceived objects in road traffic. Existing scene graph representations are simply a collection of objects, object attributes, and image attributes, providing limited information. Summary of the Invention

[0004] The present invention provides a method and device for generating a scene graph of a visual traffic scene, which are used to solve the defect of the existing technology that the scene graph has insufficient information, and to enrich the information represented by the scene graph.

[0005] The present invention provides a method for generating a scene graph of a visual traffic scene, comprising:

[0006] Performing semantic segmentation on the target traffic scene image to obtain semantic objects in the target traffic scene image, and obtaining a scene representation of the target traffic scene image based on the semantic objects;

[0007] constructing a directed graph of the target traffic scene image by using the target traffic scene image, the foreground objects and background objects in the semantic objects, and the scene representation as nodes and the relationships between the nodes as edges;

[0008] The visual area in the target traffic scene image is located according to the directed graph, and the relationship between the semantic objects in each visual area is annotated according to the directed graph to obtain a scene graph corresponding to the target traffic scene image.

[0009] According to a method for generating a scene graph of a visual traffic scene provided by the present invention, obtaining a scene representation of the target traffic scene image according to the semantic object includes:

[0010] Obtaining categories of semantic objects in multiple sample traffic scene images;

[0011] Counting the occurrence frequency of each category in the categories of semantic objects of all sample traffic scene images, and taking categories with an occurrence frequency greater than a first preset threshold as candidate categories;

[0012] In a case where the category of the semantic object in the target traffic scene image exists in the candidate categories, the category of the semantic object in the target traffic scene image is used as the scene representation of the target traffic scene image.

[0013] According to a method for generating a scene graph of a visual traffic scene provided by the present invention, the method comprises: using the target traffic scene image, the foreground objects and background objects in the semantic objects, and the scene representation as nodes, and using the relationships between the nodes as edges to construct a directed graph of the target traffic scene image, including:

[0014] From bottom to top, the target traffic scene image is used as the first layer, the foreground object is used as the second layer, the background object is used as the third layer, and the scene representation is used as the fourth layer;

[0015] The target traffic scene image, foreground objects, background objects and scene representation are taken as nodes, and the nodes are connected to each other to form edges according to the relationship between nodes in each layer and the relationship between nodes between layers.

[0016] According to a scene graph generation method for a visual traffic scene provided by the present invention, the nodes of the first layer are connected to the nodes of the second layer, and to the nodes of the third layer;

[0017] The third layer nodes include ground nodes and structural nodes, and the structural nodes include building nodes, natural nodes and sky nodes;

[0018] The ground node and the structure node are connected to indicate an adjacency relationship;

[0019] The ground nodes and the nodes of the second layer are connected, indicating a contact relationship.

[0020] According to a method for generating a scene graph of a visual traffic scene provided by the present invention, the visual area includes a left visual area, a right visual area, a near visual area, and a far visual area;

[0021] The locating the visual area in the target traffic scene image according to the directed graph includes:

[0022] Determine, according to the positions of the nodes connected to each road surface node in the target traffic scene image, a bounding box containing nodes of the same category among the nodes connected to each road surface node;

[0023] The bottommost bounding box of the target traffic scene image among the bounding boxes corresponding to all road surface nodes is used as the near-viewing area;

[0024] The bounding box located above the near vision area and closest to the near vision area is used as the far vision area;

[0025] The bounding box located to the left of the near vision area and closest to the near vision area is used as the left vision area;

[0026] The bounding box located to the right of the near viewing area and closest to the near viewing area is used as the right viewing area.

[0027] According to a scene graph generation method for a visual traffic scene provided by the present invention, the visual area includes a left visual area, a right visual area, a near visual area, and a far visual area;

[0028] The locating the visual area in the target traffic scene image according to the directed graph includes:

[0029] According to the positions of the nodes connected to each road surface node in the target traffic scene image, the minimum distance between any two road surface nodes connected to any category of nodes is obtained;

[0030] If the minimum distance is less than a second preset distance, determining a bounding box based on any category of nodes connected between any two road surface nodes;

[0031] If the minimum distances corresponding to the road surface nodes are not less than the second preset distance, determining a bounding box containing nodes of any category connected to the road surface node;

[0032] The bounding box located at the bottom of the target traffic scene image among all the bounding boxes is used as the near-viewing area;

[0033] The bounding box located above the near vision area and closest to the near vision area is used as the far vision area;

[0034] The bounding box located to the left of the near vision area and closest to the near vision area is used as the left vision area;

[0035] The bounding box located to the right of the near viewing area and closest to the near viewing area is used as the right viewing area.

[0036] The present invention also provides a device for generating a scene graph of a visual traffic scene, comprising:

[0037] a segmentation module, configured to perform semantic segmentation on the target traffic scene image to obtain semantic objects in the target traffic scene image;

[0038] a construction module, configured to construct a directed graph of the target traffic scene image by using the target traffic scene image, the foreground objects and background objects in the semantic objects, and a predetermined scene representation of the target traffic scene image as nodes and the relationships between the nodes as edges;

[0039] A generation module is used to locate the visual area in the target traffic scene image according to the directed graph, and to mark the relationship between semantic objects in each visual area according to the directed graph to obtain a scene graph corresponding to the target traffic scene image.

[0040] The present invention also provides an electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the scene graph generation method for a visual traffic scene as described above are implemented.

[0041] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of any of the above-described methods for generating a scene graph of a visual traffic scene.

[0042] The present invention also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of any of the above-described methods for generating a scene graph of a visual traffic scene.

[0043] The scene graph generation method and device for visual traffic scenes provided by the present invention obtain node information of different semantic categories by semantically segmenting the traffic scene image, and organize it into a directed graph structure of the target traffic scene image, foreground objects, background objects and scene representation. Based on the directed graph structure of the traffic scene, the visual area of ​​the image is located and the relationship between objects in the visual area is annotated, thereby obtaining a higher level of scene understanding, facilitating the automatic driving system to correctly select the driving strategy according to the generated scene graph, and improving driving accuracy and safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the present invention or the prior art, a brief introduction is given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0045] Figure 1 It is a flow chart of a method for generating a scene graph of a visual traffic scene provided by the present invention;

[0046] Figure 2 It is a schematic diagram of a bottom-up scene hierarchical representation in the scene graph generation method of a visual traffic scene provided by the present invention;

[0047] Figure 3 It is a schematic diagram of the relationship between nodes in the scene graph generation method of the visual traffic scene provided by the present invention;

[0048] Figure 4This is one of the visual area division schematic diagrams in the scene graph generation method for a visual traffic scene provided by the present invention;

[0049] Figure 5 This is the second schematic diagram of visual area division in the scene graph generation method for visual traffic scenes provided by the present invention;

[0050] Figure 6 It is a structural schematic diagram of a scene graph generating device for a visual traffic scene provided by the present invention;

[0051] Figure 7 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0052] To make the objectives, technical solutions, and advantages of the present invention more clear, the technical solutions of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0053] The following combination Figure 1 A method for generating a scene graph of a visual traffic scene according to the present invention includes: step 101, performing semantic segmentation on a target traffic scene image, obtaining semantic objects in the target traffic scene image, and obtaining a scene representation of the target traffic scene image based on the semantic objects;

[0054] The target traffic scene image is the image for which scene graph generation is required. Semantic segmentation algorithms consist of an encoder network and a decoder network. The encoder is typically a pre-trained classification network, such as VGG (Visual Geometry Group) or ResNet, followed by a decoder network. The decoder's task is to semantically project the low-resolution recognition features learned by the encoder onto the high-resolution pixel space, resulting in a dense classification.

[0055] The target traffic scene image is semantically segmented using a semantic segmentation algorithm, such as R-CNN (Region Convolutional Neural Networks) and FCN (Fully Convolutional Network). This embodiment is not limited to a specific semantic segmentation algorithm.

[0056] Step 102: constructing a directed graph of the target traffic scene image by using the target traffic scene image, the foreground objects and background objects in the semantic objects, and the scene representation as nodes and the relationships between the nodes as edges;

[0057] Semantic segmentation is used to segment the target traffic scene image into semantic objects of different semantic categories, including foreground objects and background images. Foreground objects, such as people and vehicles, are the areas of interest in the autonomous driving traffic scene. Semantic objects other than foreground objects from the semantic segmentation are considered background objects.

[0058] Scene representation is used to describe traffic scenes and can be obtained based on the semantic information of semantic objects. To better present data of different modalities, the target traffic scene image, foreground image, background image, and scene representation are organized into a directed graph structure.

[0059] The edges formed by connecting nodes represent the visual relationships between nodes, such as spatial relationships, occlusion relationships, and subordination relationships.

[0060] Construct a relationship list suitable for traffic scene graphs. Relationships not relevant to traffic scenes are filtered out from the 50 most common relationships in the visual genome. Based on the traffic semantic graph, common relationships in traffic scenes and some 3D relationships are added. Ultimately, 63 relationship categories can be distinguished, divided into 8 root-level groups. Based on the categories of any two nodes, the relationship between the two nodes is found in the relationship list.

[0061] Step 103 : Locate the visual area in the target traffic scene image according to the directed graph, and annotate the relationship between semantic objects in each visual area according to the directed graph to obtain a scene graph corresponding to the target traffic scene image.

[0062] Optionally, in an autonomous driving scenario, the visual area in the target traffic scene image is generally divided into near-sightedness, far-sightedness, left-sightedness, and right-sightedness. The visual area in the target traffic scene is determined based on the position of each semantic object in the target traffic scene image and the relationship between each semantic object node and ground nodes. The relationships between semantic objects in the visual area are annotated using edges in a directed graph to generate a scene graph corresponding to the visual traffic scene image.

[0063] The autonomous driving system determines the driving plan based on the generated scene graph. If it determines through the scene graph that there is a vehicle in front, it determines whether to slow down or change lanes based on the distance between the vehicle and the vehicle in front in the image.

[0064] The relationships between semantic objects in each visual area are divided into the relationships between semantic objects within each layer, the relationships between semantic objects within each layer and the ground and structure layer nodes, and the relationships between semantic objects within the ground and structure layers.

[0065] This embodiment performs semantic segmentation on traffic scene images to obtain node information of different semantic categories, and organizes the information into a directed graph structure of the target traffic scene image, foreground objects, background objects, and scene representation. Based on the directed graph structure of the traffic scene, the visual area of ​​the image is located and the relationship between objects in the visual area is annotated to obtain a higher level of scene understanding, which facilitates the autonomous driving system to correctly select driving strategies based on the generated scene graph, thereby improving driving accuracy and safety.

[0066] Based on the above embodiment, the method of obtaining the scene representation of the target traffic scene image according to the semantic object in this embodiment includes: obtaining categories of semantic objects in a plurality of sample traffic scene images;

[0067] In this embodiment, some key categories are selected from the categories of semantic objects in the target traffic scene image to describe the traffic scene. The categories of semantic objects can be obtained from the semantic information of semantic objects in the sample traffic scene image set.

[0068] Counting the frequency of occurrence of each category in the categories of semantic objects of all sample traffic scene images, and taking categories with a frequency of occurrence greater than a first preset threshold as candidate categories;

[0069] The frequency of occurrence of each category among all semantic objects in all sample traffic scene images is counted, that is, the number of occurrences of each category is divided by the total number of occurrences of all categories. The category whose frequency of occurrence is greater than a first preset threshold is selected.

[0070] For example, based on the probability of each category in the Cityscapes traffic dataset, 30 semantic objects are divided into nine groups: pixels, ground, structure, nature, vehicle, sky, object, person, and scene. Categories are selected based on the number of semantic objects in each group.

[0071] In a case where the category of the semantic object in the target traffic scene image exists in the candidate categories, the category of the semantic object in the target traffic scene image is used as the scene representation of the target traffic scene image.

[0072] The categories of semantic objects in the target traffic scene image are matched with the selected categories, and the successfully matched categories are used as the scene representation of the target traffic scene image. This embodiment combines a scene directed graph and a hierarchical framework to represent the traffic scene, making the scene graph representation hierarchical and relational, facilitating higher-level scene understanding.

[0073] Based on the above embodiments, Figure 2 As shown, in this embodiment, the target traffic scene image, the foreground objects and background objects in the semantic objects, and the scene representation are used as nodes, and the relationships between the nodes are used as edges to construct a directed graph of the target traffic scene image, including: from bottom to top, the target traffic scene image is used as the first layer, the foreground objects are used as the second layer, the background objects are used as the third layer, and the scene representation is used as the fourth layer;

[0074] The foreground image includes people and vehicles, and the background image includes trees and roads. From bottom to top, the nodes of each layer represent the target traffic scene image, foreground image, background image, and scene representation respectively.

[0075] The bottom-layer semantic information is stored in the target traffic scene image. Nodes in this layer represent the entire source image. Nodes in the foreground layer represent the areas of interest in autonomous driving traffic scenes. The background layer represents the background of the traffic scene. The scene layer describes the overall properties of the scene in the image. When the image content belongs to only one scene, it can be considered as just one node.

[0076] The target traffic scene image, foreground objects, background objects and scene representation are taken as nodes, and the nodes are connected to form edges according to the relationship between nodes in each layer and the relationship between nodes between layers, such as Figure 3 shown.

[0077] Nodes within or between layers are connected to form edges, which represent the visual relationships between nodes, such as spatial relationships, occlusion relationships, and subordination relationships.

[0078] Based on the above embodiment, in this embodiment, the nodes of the first layer are connected to the nodes of the second layer and to the nodes of the third layer;

[0079] Connect the nodes in the bottom layer to the nodes in the foreground and background layers.

[0080] The third layer nodes include ground nodes and structural nodes, and the structural nodes include building nodes, natural nodes and sky nodes;

[0081] The background layer contains two types of nodes, namely ground and structure, where the structure node represents the background structure in the traffic scene.

[0082] The ground node and the structure node are connected to indicate an adjacency relationship;

[0083] The ground nodes and the nodes of the second layer are connected, indicating a contact relationship.

[0084] This embodiment realizes the interconnection of nodes between different layers or across different layers, and realizes the unification of multi-mode data.

[0085] Based on the above embodiment, the visual area in this embodiment includes a left visual area, a right visual area, a near visual area, and a far visual area; and locating the visual area in the target traffic scene image according to the directed graph includes: determining, based on the positions of the nodes connected to each road surface node in the target traffic scene image, a bounding box containing nodes of the same category among the nodes connected to each road surface node;

[0086] This embodiment uses a combination of ground type and visual regions to generate a scene graph based on actual needs. Visual regions within the target traffic scene image are annotated using bounding boxes. To prevent loss of regional relationships, the left, right, near, and far vision regions are added to localize the visual regions, taking into account the human visual attention mechanism in traffic scenes.

[0087] When performing visual area localization, the nodes connected to each road surface node are divided into multiple groups according to their categories. A bounding box is drawn for each group of nodes so that all nodes in each group are within the bounding box.

[0088] The bounding box located at the bottom of the target traffic scene image among the bounding boxes corresponding to all road surface nodes is used as the near-viewing area; the bounding box located above the near-viewing area and closest to the near-viewing area is used as the far-viewing area; the bounding box located to the left of the near-viewing area and closest to the near-viewing area is used as the left-viewing area; and the bounding box located to the right of the near-viewing area and closest to the near-viewing area is used as the right-viewing area.

[0089] like Figure 4 As shown, there are 5 ground objects in the target traffic scene image, and 4 visual areas are located according to the 5 ground objects.

[0090] Based on the above embodiment, the visual area in this embodiment includes a left visual area, a right visual area, a near visual area, and a far visual area; and locating the visual area in the target traffic scene image according to the directed graph includes: obtaining a minimum distance between nodes of any category connected by any two road surface nodes based on positions of nodes connected by each road surface node in the target traffic scene image;

[0091] For example, road node A connects to vehicle nodes C and D, and road node B connects to vehicle node E. Calculate the distance d1 between C and E, and the distance d2 between D and E. The minimum distance is the minimum of d1 and d2.

[0092] If the minimum distance is less than a second preset distance, determining a bounding box based on any category of nodes connected between any two road surface nodes;

[0093] If the minimum value of d1 and d2 is less than the second preset distance, the bounding box containing C, D, and E is determined, which may be a minimum rectangle containing C, D, and E.

[0094] If the minimum distances corresponding to the road surface nodes are not less than the second preset distance, determining a bounding box containing nodes of any category connected to the road surface node;

[0095] If there are three road nodes A, B, and C, and if the minimum distances corresponding to the vehicle nodes between A and B and between A and C are both greater than or equal to the second preset distance, then the bounding box is determined based on the vehicle nodes C and D connected to the road node A, that is, the minimum rectangle containing the vehicle nodes C and D is determined.

[0096] The bounding box located at the bottom of the target traffic scene image among all the bounding boxes is taken as the near vision area; the bounding box located above the near vision area and closest to the near vision area is taken as the far vision area; the bounding box located to the left of the near vision area and closest to the near vision area is taken as the left vision area; the bounding box located to the right of the near vision area and closest to the near vision area is taken as the right vision area, as shown in FIG. Figure 5 shown.

[0097] The scene graph generating device for a visual traffic scene provided by the present invention is described below. The scene graph generating device for a visual traffic scene described below and the scene graph generating method for a visual traffic scene described above can refer to each other.

[0098] like Figure 6 As shown, the apparatus includes a segmentation module 601, a construction module 602, and a generation module 603, wherein:

[0099] The segmentation module 601 is used to perform semantic segmentation on the target traffic scene image to obtain semantic objects in the target traffic scene image;

[0100] The construction module 602 is configured to construct a directed graph of the target traffic scene image by using the target traffic scene image, the foreground objects and background objects in the semantic objects, and a predetermined scene representation of the target traffic scene image as nodes and the relationships between the nodes as edges;

[0101] The generation module 603 is used to locate the visual area in the target traffic scene image according to the directed graph, and mark the relationship between semantic objects in each visual area according to the directed graph to obtain a scene graph corresponding to the target traffic scene image.

[0102] This embodiment performs semantic segmentation on traffic scene images to obtain node information of different semantic categories, and organizes the information into a directed graph structure of the target traffic scene image, foreground objects, background objects, and scene representation. Based on the directed graph structure of the traffic scene, the visual area of ​​the image is located and the relationship between objects in the visual area is annotated to obtain a higher level of scene understanding, which facilitates the autonomous driving system to correctly select driving strategies based on the generated scene graph, thereby improving driving accuracy and safety.

[0103] Figure 7 An example of a physical structure diagram of an electronic device is shown below. Figure 7 As shown, the electronic device may include: a processor 710, a communication interface 720, a memory 730, and a communication bus 740, wherein the processor 710, the communication interface 720, and the memory 730 communicate with each other via the communication bus 740. The processor 710 may call the logic instructions in the memory 730 to execute a method for generating a scene graph of a visual traffic scene, the method comprising: performing semantic segmentation on a target traffic scene image, obtaining semantic objects in the target traffic scene image, and obtaining a scene representation of the target traffic scene image based on the semantic objects; constructing a directed graph of the target traffic scene image using the target traffic scene image, the foreground objects and background objects in the semantic objects, and the scene representation as nodes and the relationships between the nodes as edges; locating visual areas in the target traffic scene image according to the directed graph, and annotating the relationships between the semantic objects in each visual area according to the directed graph to obtain a scene graph corresponding to the target traffic scene image.

[0104] In addition, the logic instructions in the above-mentioned memory 730 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0105] On the other hand, the present invention also provides a computer program product, which includes a computer program, which can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the scene graph generation method for the visual traffic scene provided by the above-mentioned methods, which includes: performing semantic segmentation on the target traffic scene image, obtaining semantic objects in the target traffic scene image, and obtaining a scene representation of the target traffic scene image based on the semantic objects; taking the target traffic scene image, the foreground objects and background objects in the semantic objects, and the scene representation as nodes, and the relationships between the nodes as edges, to construct a directed graph of the target traffic scene image; locating the visual area in the target traffic scene image according to the directed graph, and marking the relationships between the semantic objects in each visual area according to the directed graph to obtain a scene graph corresponding to the target traffic scene image.

[0106] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a scene graph generation method for a visual traffic scene provided by the above-mentioned methods, the method comprising: performing semantic segmentation on a target traffic scene image, obtaining semantic objects in the target traffic scene image, and obtaining a scene representation of the target traffic scene image based on the semantic objects; constructing a directed graph of the target traffic scene image by taking the target traffic scene image, foreground objects and background objects in the semantic objects, and the scene representation as nodes and the relationships between the nodes as edges; locating the visual areas in the target traffic scene image according to the directed graph, and labeling the relationships between the semantic objects in each visual area according to the directed graph to obtain a scene graph corresponding to the target traffic scene image.

[0107] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0108] Through the description of the above embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus a necessary general hardware platform, or of course, by hardware. Based on this understanding, the essence of the above technical solution or the part that contributes to the existing technology can be embodied in the form of a software product. The computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or certain parts of the embodiments.

[0109] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for generating a scene graph of a visual traffic scene, characterized in that: include: Performing semantic segmentation on the target traffic scene image to obtain semantic objects in the target traffic scene image, and obtaining a scene representation of the target traffic scene image based on the semantic objects; constructing a directed graph of the target traffic scene image by using the target traffic scene image, the foreground objects and background objects in the semantic objects, and the scene representation as nodes and the relationships between the nodes as edges; Locating a visual area in the target traffic scene image according to the directed graph, and annotating the relationship between semantic objects in each visual area according to the directed graph to obtain a scene graph corresponding to the target traffic scene image; Locating visual regions in the target traffic scene image according to the directed graph, and labeling relationships between semantic objects in each visual region according to the directed graph, including: The nodes connected to each road surface node are divided into multiple groups according to their categories, and a bounding box is drawn for each group of nodes so that all nodes in each group are within the bounding box; The visual area in the target traffic scene image is annotated with a bounding box; the visual area includes a left visual area, a right visual area, a near visual area, and a far visual area; The bottommost bounding box of the target traffic scene image among the bounding boxes corresponding to all road surface nodes is used as the near-viewing area; The bounding box located above the near vision area and closest to the near vision area is used as the far vision area; The bounding box located to the left of the near vision area and closest to the near vision area is used as the left vision area; The bounding box located to the right of the near viewing area and closest to the near viewing area is used as the right viewing area.

2. The method for generating a scene graph of a visual traffic scene according to claim 1, characterized in that: The acquiring a scene representation of the target traffic scene image according to the semantic object includes: Obtaining categories of semantic objects in multiple sample traffic scene images; Counting the frequency of occurrence of each category in the categories of semantic objects of all sample traffic scene images, and taking categories with a frequency of occurrence greater than a first preset threshold as candidate categories; In a case where the category of the semantic object in the target traffic scene image exists in the candidate categories, the category of the semantic object in the target traffic scene image is used as the scene representation of the target traffic scene image.

3. The method for generating a scene graph of a visual traffic scene according to claim 1, characterized in that: The method comprises: using the target traffic scene image, the foreground object and the background object in the semantic object, and the scene representation as nodes, and using the relationships between the nodes as edges to construct a directed graph of the target traffic scene image, including: From bottom to top, the target traffic scene image is used as the first layer, the foreground object is used as the second layer, the background object is used as the third layer, and the scene representation is used as the fourth layer; The target traffic scene image, foreground objects, background objects and scene representation are taken as nodes, and the nodes are connected to each other to form edges according to the relationship between nodes in each layer and the relationship between nodes between layers.

4. The method for generating a scene graph of a visual traffic scene according to claim 3, characterized in that: The nodes of the first layer are connected to the nodes of the second layer and to the nodes of the third layer; The third layer nodes include ground nodes and structural nodes, and the structural nodes include building nodes, natural nodes and sky nodes; The ground node and the structure node are connected to indicate an adjacency relationship; The ground nodes and the nodes of the second layer are connected, indicating a contact relationship.

5. The method for generating a scene graph of a visual traffic scene according to claim 4, characterized in that: The locating the visual area in the target traffic scene image according to the directed graph includes: According to the positions of the nodes connected to each road surface node in the target traffic scene image, the minimum distance between any two road surface nodes connected to any category of nodes is obtained; If the minimum distance is less than a second preset distance, determining a bounding box based on any category of nodes connected between any two road surface nodes; If there are minimum distances corresponding to the road surface nodes that are not less than the second preset distance, a bounding box containing nodes of any category connected to the road surface node is determined.

6. A device for generating a scene graph of a visual traffic scene, characterized in that: include: a segmentation module, configured to perform semantic segmentation on the target traffic scene image to obtain semantic objects in the target traffic scene image; a construction module, configured to construct a directed graph of the target traffic scene image by using the target traffic scene image, the foreground objects and background objects in the semantic objects, and a predetermined scene representation of the target traffic scene image as nodes and the relationships between the nodes as edges; a generating module, configured to locate visual areas in the target traffic scene image according to the directed graph, and annotate relationships between semantic objects in each visual area according to the directed graph, to obtain a scene graph corresponding to the target traffic scene image; Generate modules, including: The nodes connected to each road surface node are divided into multiple groups according to their categories, and a bounding box is drawn for each group of nodes so that all nodes in each group are within the bounding box; The visual area in the target traffic scene image is annotated with a bounding box; the visual area includes a left visual area, a right visual area, a near visual area, and a far visual area; The bottommost bounding box of the target traffic scene image among the bounding boxes corresponding to all road surface nodes is used as the near-viewing area; The bounding box located above the near vision area and closest to the near vision area is used as the far vision area; The bounding box located to the left of the near vision area and closest to the near vision area is used as the left vision area; The bounding box located to the right of the near viewing area and closest to the near viewing area is used as the right viewing area.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the method for generating a scene graph of a visual traffic scene as claimed in any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method for generating a scene graph of a visual traffic scene according to any one of claims 1 to 5 are implemented.

9. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method for generating a scene graph of a visual traffic scene according to any one of claims 1 to 5 are implemented.

Images