Road scene graph generation method and device, equipment and medium

Abstract

The invention provides a road scene graph generation method and device, equipment and a medium, and relates to the technical field of automatic driving, and the method comprises the steps: taking a multi-view image obtained by a vehicle as input data, inputting a multi-task traffic element detection model, and obtaining traffic elements in a BEV space, the traffic elements comprising traffic entities and vector map elements; according to the position information of the traffic elements in the BEV space, calculating to obtain a spatial relationship among different traffic elements, and according to the spatial relationship among different traffic elements and attributes of traffic entities, calculating to obtain an interaction relationship among different traffic elements; and constructing a road scene graph according to the traffic elements and the spatial relationship and the interaction relationship among the different traffic elements. According to the method, the road scene graph which contains traffic entities and map elements, integrates dynamic and static information and considers geometric and semantic information is constructed, and the overall perception and deep understanding of an automatic driving vehicle on a complex and dynamic road environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a method, device, equipment and medium for generating a road scene graph based on multi-task traffic element detection. Background Art

[0002] Building a dedicated map for autonomous vehicles is a current research hotspot. From the perspective of cartography, the maps used by autonomous vehicles need to record the location, geometry, and attributes of objects and phenomena in the environment, and also reflect their spatial distribution, combination, association, and changes and development over time. At present, the extraction of geometric and semantic information of environmental objects in road scenes has become relatively mature, forming typical map forms such as high-precision maps, autonomous driving maps, occupancy maps, and machine maps, and establishing standards such as OpenDRIVE. Autonomous driving also requires association information for map information, because focusing on the geometric semantics of objects in isolation and ignoring the association between objects will make it difficult for autonomous vehicles to cope with the complex dynamic interaction between elements in road scenes. In particular, as the semantic information involved in autonomous road scenes becomes more and more complex, such as special road scenes such as intersections, accurate and comprehensive relationship information is even more important than high-precision geometric data.

[0003] Road Scene Graph is considered to be one of the effective ways to provide map information for autonomous vehicles. The main purpose of the road scene graph is to automatically identify traffic elements and their relationships in the road scene and to represent them in a structured form in the form of a graph. The road scene graph can explicitly model the relationships between traffic elements in a complex and dynamic road environment, which can improve the understanding and reasoning ability of autonomous vehicles about the road environment. The scene graph can also be used as an intermediate representation for semantic reasoning, so that autonomous vehicles can evaluate and predict potential risks that cannot be perceived by geometric information. However, the road scene graph is essentially a graph representation of road scene elements and their relationships, and there is still a certain gap between it and the map that uniformly represents position, geometry, attributes, spatial relationships and semantic relationships.

[0004] From the perspective of providing a unified representation of position, geometry, attributes, spatial relationships, and semantic relationships for autonomous vehicles, existing research on road element detection and scene graph generation has the following problems: (1) Existing road scene graphs have unclear relationship patterns. Most scene graph relationship patterns only consider the spatial relationships between traffic entities, and do not express the rich semantic relationships between traffic elements and the interaction and influence relationships; most studies define interaction relationships as undirected and do not distinguish between subjects and objects, resulting in overlaps between many relationship concepts. (2) The problem of joint and unified representation of road scene element content. From the current research status, the current research on 3D object detection, road element detection, and road scene relationship extraction is in a relatively separate state, lacking a joint and unified representation of various road scene elements and their relationships. For example, high-precision maps have achieved the evolution from static data to dynamic and static data coupling, but the relationship expression between scene elements is weak; road scene graphs focus on the traffic entities and their relationship representation in the scene, but ignore the detection of map elements. Summary of the invention

[0005] In response to the problems that the relationship model of the road scene graph is unclear and target detection, road element detection and road scene relationship extraction are not unified, the embodiment of the present application provides a road scene graph generation method based on multi-task traffic element detection to construct a road scene graph that includes traffic entities and map elements, integrates dynamic and static information, and takes into account geometric and semantic information.

[0006] The embodiment of the present application provides the following technical solution: a method for generating a road scene graph, comprising:

[0007] Taking the multi-view images acquired by the vehicle as input data, the multi-view images are input into a multi-task traffic element detection model, and traffic elements in the BEV space are obtained through the multi-task traffic element detection model, wherein the traffic elements include traffic entities and vector map elements; wherein the multi-task traffic element detection model is a detection model constructed based on an encoding-decoding structure;

[0008] According to the position information of the traffic elements in the BEV space, the spatial relationship between different traffic elements is calculated, and according to the spatial relationship between different traffic elements and the attributes of traffic entities, the interactive relationship between different traffic elements is calculated;

[0009] A road scene graph is constructed based on the traffic elements and the spatial relationships and interactive relationships between different traffic elements.

[0010] According to an embodiment of the present application, obtaining traffic elements in the BEV space through the multi-task traffic element detection model includes:

[0011] Taking the multi-view images of the vehicle as input data, vectorizing the map elements contained in the multi-view images through the map element vectorization module in the multi-task traffic element detection model;

[0012] Extracting image features of the multi-view image through the encoding unit in the multi-task traffic element detection model, converting the image features into BEV features in the BEV space, inputting the BEV features into the traffic element prediction unit in the multi-task traffic element detection model, and obtaining the prediction result of the traffic element;

[0013] Taking the center of the vehicle as the origin, the coordinates of the vector map elements and the traffic entities are aligned and unified into the same coordinate system in the BEV space to obtain the traffic entities and vector map elements in the BEV space.

[0014] According to an embodiment of the present application, vectorizing the map elements contained in the multi-view image by a map element vectorization module in the multi-task traffic element detection model includes:

[0015] The shape boundaries of the map elements in the multi-view image are sampled sequentially, and each map element is converted into a polyline consisting of a fixed number of ordered two-dimensional point sequences, so as to complete the structured modeling of the map elements and determine the vector representation of the map elements in each multi-view image.

[0016] According to an embodiment of the present application, the coordinates of the vector map element and the traffic entity are aligned to be unified into the same coordinate system in the BEV space, including:

[0017] Taking the center of the vehicle as the origin, the right side of the vehicle as the X direction, the vehicle direction as the Y direction, and the vertical upward as the Z direction; the three-dimensional bounding box of the traffic element in the global coordinate system is converted to the lidar coordinate system after translation and rotation, and then the converted traffic entity and vector map element are fused and visualized within the predefined BEV space range to achieve coordinate alignment of the traffic elements contained at the current moment.

[0018] According to an embodiment of the present application, the spatial relationship between different traffic elements is calculated based on the position information of the traffic elements in the BEV space, wherein the spatial relationship includes a distance relationship, an orientation relationship and a topological relationship, including:

[0019] Define the subject and object in each pair of relations, use the coordinates between the subject and the object to perform quantitative calculation of the distance relation, obtain the quantitative distance, and then divide the quantitative distance into a semantically qualitative distance relation according to an application-oriented distance threshold;

[0020] The relative orientation relationship is used to express the directional relationship between the subject and the object, and the subject is used as the reference of the object to determine the orientation relationship between the subject and the object;

[0021] The topological relationship between the subject and the object is determined using the surface topology calculation method.

[0022] According to an embodiment of the present application, the attributes of the traffic entity include the category, center point coordinates, bounding box parameters and speed of the traffic element in the prediction result of the traffic element.

[0023] According to an embodiment of the present application, constructing a road scene graph according to the traffic elements and the spatial relationship and interaction relationship between different traffic elements includes:

[0024] The traffic elements are taken as nodes, and the spatial relationships and interactive relationships between different traffic elements are taken as directed edges between the nodes in the scene graph to generate a road scene graph.

[0025] The present application also provides a road scene graph generating device, comprising:

[0026] A multi-task traffic element detection module is used to obtain multi-view images of the vehicle, input the multi-view images into a multi-task traffic element detection model, and obtain traffic elements in the BEV space through the multi-task traffic element detection model, wherein the traffic elements include traffic entities and vector map elements; wherein the multi-task traffic element detection model is a detection model constructed based on an encoding-decoding structure;

[0027] A road scene relationship extraction module, used to calculate the spatial relationship between different traffic entities and vector map elements according to the position information of the traffic elements in the BEV space, and to calculate the interactive relationship between different traffic entities and vector map elements according to the spatial relationship and the attributes of the traffic entities;

[0028] The road scene graph generation module is used to construct a road scene graph according to the traffic elements and the spatial relationships and interactive relationships between the traffic elements.

[0029] The present application also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above-mentioned road scene graph generation method when executing the computer program.

[0030] The present application also provides a computer-readable storage medium, which stores a computer program for executing the above-mentioned road scene graph generation method.

[0031] Compared with the prior art, the beneficial effects that can be achieved by at least one of the above-mentioned technical solutions adopted in the embodiments of this specification include at least the following: The embodiments of the present invention expand the road scene graph model based on the principles of cartography, and propose a directed relational scene graph model that includes multiple types of traffic elements (traffic entities and map elements) and covers the geometric and semantic relationships between elements, thereby enriching the content and functions of the road scene graph. The embodiments of the present invention directly perform traffic entity and map element detection under BEV (bird's eye view), effectively improving the detection accuracy of both, and providing more accurate environmental object location, geometry and attribute information for road scene graph construction. The embodiments of the present invention can construct a road scene graph that includes multiple types of traffic elements, integrates dynamic and static information, and takes into account geometric and semantic information. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 is a schematic diagram of a road scene graph generation method according to an embodiment of the present invention;

[0034] Figure 2 is a schematic diagram of a road scene graph generating device according to an embodiment of the present invention;

[0035] Figure 3 It is a structural schematic diagram of the computer device of the present invention. DETAILED DESCRIPTION

[0036] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0037] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0038] like Figure 1As shown, an embodiment of the present invention provides a method for generating a road scene graph, comprising:

[0039] S101. Taking the multi-view images acquired by the vehicle as input data, inputting the multi-view images into a multi-task traffic element detection model, and obtaining traffic elements in the BEV space through the multi-task traffic element detection model, wherein the traffic elements include traffic entities and vector map elements; wherein the multi-task traffic element detection model is a detection model constructed based on an encoding-decoding structure;

[0040] S102. Calculate the spatial relationship between different traffic elements according to the location information of the traffic elements in the BEV space, and calculate the interactive relationship between different traffic elements according to the spatial relationship between different traffic elements and the attributes of traffic entities;

[0041] S103. Construct a road scene graph according to the traffic elements and the spatial relationships and interactive relationships between different traffic elements.

[0042] The road scene graph is a structured data representation of traffic elements in the scene and the relationships between them, and is composed of a node set and an edge set. The node set includes traffic elements and map elements, and the edge set is a directed edge connecting these nodes. In order to more clearly define the concept of each relationship, the embodiment of the present invention expresses the directionality of the relationship by distinguishing the subject and object of the relationship.

[0043] The road scene graph generation method of the embodiment of the present invention mainly includes two main parts based on traffic element detection and relationship extraction. The traffic element detection part aims to generate road scene graph nodes, and simultaneously obtain traffic entities and vector map elements in the BEV space from multi-perspective images of autonomous driving vehicles. In this part, the embodiment of the present invention designs a multi-task traffic element detection model, which takes multi-perspective images as input and directly obtains traffic entities and vectorized map elements in the BEV space. Directly performing traffic element detection in the BEV feature space not only avoids the problem of ranging required by two-dimensional target detection methods, but also improves the detection accuracy of each other by intersecting the tasks of traffic entity and map element detection. At the same time, using BEV to represent traffic elements helps to determine the relative position relationship between different elements, and can effectively avoid the problems of scale inconsistency and object occlusion under horizontal perspective.

[0044] The road scene relationship extraction part aims to capture the spatial and semantic relationships between different traffic and map elements, and construct them as directed edges between nodes in the scene graph. The embodiment of the present invention constructs a relationship extraction method that integrates geometric and semantic information. By establishing rules that integrate entity attributes and relationship characteristics, a corresponding discriminant function is established for each relationship, and then the detected traffic elements and their attributes are used to perform relationship calculations to obtain the spatial and semantic relationships between different traffic elements, and finally express them in the road scene graph in the form of directed edges.

[0045] According to a specific implementation of this embodiment, obtaining traffic elements in the BEV space through the multi-task traffic element detection model includes:

[0046] Taking the multi-view images of the vehicle as input data, vectorizing the map elements contained in the multi-view images through the map element vectorization module in the multi-task traffic element detection model, and predicting and obtaining the vector map elements;

[0047] Extracting image features of the multi-view image through the encoding unit in the multi-task traffic element detection model, converting the image features into BEV features in the BEV space, inputting the BEV features into the traffic element prediction unit in the multi-task traffic element detection model, and obtaining the prediction result of the traffic element;

[0048] Taking the center of the vehicle as the origin, the coordinates of the vector map elements and the traffic entities are aligned and unified into the same coordinate system in the BEV space to obtain the traffic entities and vector map elements in the BEV space.

[0049] According to a specific implementation of this embodiment, vectorizing the map elements contained in the multi-view image by a map element vectorization module in the multi-task traffic element detection model includes:

[0050] The shape boundaries of the map elements in the multi-view image are sampled sequentially, and each map element is converted into a polyline consisting of a fixed number of ordered two-dimensional point sequences, so as to complete the structured modeling of the map elements and determine the vector representation of the map elements in each multi-view image.

[0051] According to a specific implementation of this embodiment, the coordinates of the vector map element and the traffic entity are aligned to be unified into the same coordinate system in the BEV space, including:

[0052] Taking the center of the vehicle as the origin, the right side of the vehicle as the X direction, the vehicle direction as the Y direction, and the vertical upward as the Z direction; the three-dimensional bounding box of the traffic element in the global coordinate system is converted to the lidar coordinate system after translation and rotation, and then the converted traffic entity and vector map element are fused and visualized within the predefined BEV space range to achieve coordinate alignment of the traffic elements contained at the current moment.

[0053] In the specific implementation, this embodiment designs a multi-task traffic element detection model based on the encoding-decoding structure. The specific idea is: take the 6 multi-view images obtained by the vehicle as input, first vectorize the map elements contained in the image, then use the encoding unit to extract the multi-view image features and convert them into BEV features, then input the features into the traffic element prediction unit, and finally output them to the coordinate alignment module and unify the two types of traffic elements into the same coordinate system. In this way, the traffic entity data centered on the vehicle at this moment is generated. The specific operations are as follows:

[0054] (1) Vectorization of map elements

[0055] Most existing map element recognition methods obtain rasterized maps through semantic segmentation. However, the map elements represented at the pixel level lack instance-level information and cannot be accurately calculated in the scene graph. In order to highlight the instance-level features of map elements, this embodiment performs structured modeling on map elements before model training to form a vector representation of map elements.

[0056] The main idea is to convert each map element into a polyline consisting of a fixed number of ordered 2D points by sampling along the shape boundary. Specifically, N polylines are used. As a primitive to represent each map element, each polyline are sampled into 20 ordered points v i，n The collection of The vectorization of map elements can not only predict the normalized coordinates of each map element under BEV directly from the input two-dimensional image, avoiding complicated post-processing, but also help to model the spatial relationship between map elements in the subsequent process.

[0057] (2) Coding unit

[0058] The unit first uses the backbone network to extract features from each image to obtain a multi-view feature map. In order to balance model accuracy and training time, in traffic entity detection, this embodiment uses the ResNet-101 residual network and FPN pre-trained on ImageNet as the backbone network, and the backbone network in map element detection is ResNet-50 and FPN. Then, it is input into the view conversion module to convert the image features into BEV features, providing a unified surrounding environment representation for perception tasks.

[0059] (3) Traffic element prediction unit

[0060] Since map elements and traffic entities are expressed differently under BEV, this embodiment designs different prediction units (traffic entity prediction unit and map element prediction unit) according to the data characteristics of the two traffic elements, supporting vector map elements and dynamic traffic entity detection tasks respectively. The two prediction units have similar structures, both consisting of a decoding module and a prediction head, wherein the internal structure of the decoding module is the same, comprising 6 layers, each layer following the Transformer architecture, and is used to interact the defined query parameters with the BEV features obtained from the encoder, the difference being the input and output of the decoding module.

[0061] (4) Coordinate alignment module

[0062] The traffic entity prediction unit outputs a three-dimensional bounding box in the global coordinate system, and the map element prediction unit outputs a two-dimensional sampling point in the laser radar coordinate system. The center of the vehicle can be used as the origin for the coordinate system (the center of the vehicle, the right side of the vehicle as the X direction, the vehicle direction as the Y direction, and the vertical upward as the Z direction). Specifically, the three-dimensional bounding box of the traffic entity is first converted to the laser radar coordinate system after translation and rotation, and then the converted traffic entity and map element data are fused so that the traffic elements contained at that moment are aligned in coordinates and visualized within the predefined BEV range.

[0063] Based on the detection of traffic elements, the embodiments of the present invention can further carry out road scene relationship extraction and scene graph construction. Based on the patterns of spatial relationships and interactive relationships, and with multi-perspective two-dimensional images, traffic entities and map elements as input, this embodiment proposes an algorithm for automatically generating frame-by-frame scene graphs. First, the spatial relationship is calculated using the location information of traffic elements under BEV; second, the interactive relationship between different traffic elements is calculated in combination with the attributes of traffic entities and the contextual knowledge of the scene; third, based on the detected map elements and road elements, the category relationship between elements is inferred; finally, the element set and the relationship set are combined to generate a scene graph.

[0064] According to a specific implementation of this embodiment, the spatial relationship between the different traffic elements is calculated based on the position information of the traffic elements in the BEV space, wherein the spatial relationship includes a distance relationship, an orientation relationship and a topological relationship, including:

[0065] Define the subject and object in each pair of relations, use the coordinates between the subject and the object to perform quantitative calculation of the distance relation, obtain the quantitative distance, and then divide the quantitative distance into a semantically qualitative distance relation according to an application-oriented distance threshold;

[0066] The relative orientation relationship is used to express the directional relationship between the subject and the object, and the subject is used as the reference of the object to determine the orientation relationship between the subject and the object;

[0067] The topological relationship between the subject and the object is determined using the surface topology calculation method.

[0068] The specific implementation includes:

[0069] (1) Spatial relationship extraction

[0070] For the distance relationship, the coordinates between the subject and the object are first used to perform quantitative calculation of the distance relationship, and then the quantitative distance is divided into semantic qualitative distance relationships according to the application-oriented distance threshold, such as near collision (0.5 meters), near (4 meters), visible (7 meters), etc. The calculation method of the distance relationship DisRel is expressed as DisRel = {(x sub ,y sub ), (x ob ,y ob ), m Dis , r Dis}, where (x sub ,y sub ) and (x ob ,y ob ) represent the two-dimensional coordinates of the subject and the object, r Dis is the category of distance relationship.

[0071] For the orientation relationship, this embodiment uses the relative orientation relationship to express the directional relationship between the subject and the object, and uses the subject as the reference of the object to describe the orientation relationship. First, the vehicle is taken as the center and the direction of the vehicle is taken as the starting point, and the angle intervals are divided into eight angular intervals in a clockwise direction at equal intervals, which correspond to the qualitative expressions of the eight directional relationships. The calculation method of the directional relationship DirRel is expressed as DirRel = {(x sub ,y sub ), (x ob ,y ob ), m Dir , rDir},in

[0072] For topological relationships, there are mainly two types of relationships: is in (inside) and nearby (adjacent). In order to facilitate the calculation of topological relationships related to map elements, this paper converts map elements represented by polylines into surface representations, and the algorithm is based on the surface topology calculation method. isIn The calculation method is expressed as TopRel isIn ={(p tl , p tr , p bl , p br , p c ,),(M1,M2)}, where p tl 、p tr 、p bl 、p br 、p c are the four corner points and the center point of the two-dimensional bounding box of the traffic entity. M1 and M2 represent different map elements. The map element where the traffic entity is located is determined based on the number of reference points contained in M1 and M2. Nearby is used to describe the adjacency relationship between different map elements. TopRel nearby The calculation method is expressed as RopRel nearby ={(M1, M2), m Top}, where m Top A topological computational method for representing surface-to-surface relationships in the region connectivity (RCC) theory of reference space.

[0073] (2) Extraction of interactive relationships Interactive relationships are a further exploration of spatial relationships. When traffic elements are within a certain distance range, their local proximity will have mutual interactions, and the distance threshold will participate in this interaction. Therefore, the calculation of interactive relationships not only needs to consider the attributes or semantic information between multiple objects, but also needs to take into account the influence of different levels and types of spatial relationships between multiple objects. Traffic entity attributes include the category, center point coordinates, bounding box parameters and speed of traffic entities. Since map elements themselves carry certain semantic information, some interactive relationships can be inferred based on the spatial relationship between traffic entities and map elements.

[0074] The embodiment of the present invention uses the average accuracy under different distance thresholds as an evaluation index to quantitatively evaluate the detection effect of traffic elements. In addition, in order to evaluate the effectiveness of the multi-task traffic element detection model, the embodiment of the present invention carried out a combination of qualitative and quantitative evaluation experiments on the nuScenes validation set. The quantitative evaluation results show that compared with the existing methods, the detection model of the embodiment of the present invention can achieve the highest average precision (mAP), which is 1.5% higher than the existing most advanced methods. By comparison, in complex situations where there is severe occlusion between entities, the model of the embodiment of the present invention can achieve more accurate detection results than other methods.

[0075] In order to verify the stability and generalization of the method of the embodiment of the present invention in actual scenarios, the embodiment of the present invention uses the nuScenes validation set to construct the scene graph, and selects traffic entities with prediction values ​​greater than 0.3 and map elements greater than 0.4 as nodes in the graph. Considering that most of the current scene graph construction methods use the model in the existing method (RoadScene2vec) for relationship extraction, the embodiment of the present invention uses it as a comparison method, and the specific comparison results are as follows:

[0076] (1) In a simple single-lane scene, compared with the existing method (RoadScene2vec) that can only infer the positional relationship between some traffic entities in front of the ego vehicle and the ego vehicle, the method of the embodiment of the present invention can not only perceive all traffic elements around the ego vehicle within a certain distance threshold, solving the problem of limited perception range caused by using a single camera, but also obtain richer and fine-grained semantic relationships.

[0077] (2) In a scene with dense entities, most pedestrians or vehicles are very close in spatial position and move in the same pattern, thus forming a group. The embodiment of the present invention hierarchically clusters the identified pedestrians and vehicles, and the clustering criterion is the distance between objects (the threshold between pedestrians is 1 meter, and the threshold between vehicles is 5 meters), and defines the generated new nodes as "groups". Compared with the existing method (RoadScene2vec), this strategy can reduce the amount of model calculation in the relationship extraction process and simplify the final scene graph.

[0078] (3) In complex road intersection scenes such as intersections, there are complex semantic relationships between traffic elements and map elements, as well as between map elements. The existing method (RoadScene2vec) roughly divides the lanes where other vehicles are located into three types: left lane, middle lane, and right lane based on the relative distance between the other vehicles and the vehicle in the horizontal direction. This virtual lane relationship cannot provide accurate environmental information for autonomous driving vehicles. In contrast, the method of the embodiment of the present invention can not only obtain a more fine-grained spatial relationship between traffic entities and real map elements by integrating map element detection into scene graph construction, but also further add interactive relationships between the two according to the type of traffic entity, the semantic information of the map elements, and the driving direction of the vehicle and the lane, such as <pedestrian, crossing on, pedestrian crossing>. In addition, the scene graph constructed by the method of the embodiment of the present invention can also express the interactive relationship between map elements, for example, judging the connectivity relationship between lanes based on the topological relationship between the lane centerline and different lanes.

[0079] In order to construct a road scene graph with more accurate information and richer content, an embodiment of the present invention proposes a multi-task traffic element detection and road scene graph generation method. This method unifies traffic entities and map elements into the same coordinate system by constructing an end-to-end multi-task traffic element detection model, and improves the average accuracy of traffic entity and map element detection by 1.5% and 1.9% respectively. A relationship model that includes multiple types of traffic elements and takes into account geometric and semantic information is established, which not only expands and forms a wider perception range, but also can identify more comprehensive and detailed relationships between traffic elements. The number of nodes and edges obtained is 4.4 times and 6.7 times that of representative methods respectively. The method of the embodiment of the present invention can construct a road scene graph with wider coverage, richer and more detailed content, and can be applied to different types of road scenes.

[0080] like Figure 2 As shown, the present application also provides a road scene graph generating device 200, comprising:

[0081] The multi-task traffic element detection module 201 is used to use the multi-view images acquired by the vehicle as input data, input the multi-view images into a multi-task traffic element detection model, and obtain traffic elements in the BEV space through the multi-task traffic element detection model, wherein the traffic elements include traffic entities and vector map elements; wherein the multi-task traffic element detection model is a detection model constructed based on an encoding-decoding structure;

[0082] A road scene relationship extraction module 202 is used to calculate the spatial relationship between different traffic entities and vector map elements according to the position information of the traffic elements in the BEV space, and to calculate the interactive relationship between different traffic entities and vector map elements according to the spatial relationship and the attributes of the traffic entities;

[0083] The road scene graph generation module 203 is used to construct a road scene graph according to the traffic elements and the spatial relationships and interactive relationships between the traffic elements.

[0084] During specific implementation, the multi-task traffic element detection module 201 uses the multi-view image of the vehicle as input data, and vectorizes the map elements contained in the multi-view image through the map element vectorization module in the multi-task traffic element detection model, and obtains the vector representation of the map elements; extracts the image features of the multi-view image through the encoding unit in the multi-task traffic element detection model, and converts the image features into BEV features under the BEV space, and inputs the BEV features into the traffic element prediction unit in the multi-task traffic element detection model to obtain the prediction results of the traffic elements; takes the center of the vehicle as the origin, aligns the coordinates of the vector map elements and the traffic entities, unifies them into the same coordinate system under the BEV space, and obtains the traffic entities and vector map elements under the BEV space.

[0085] Among them, the multi-task traffic element detection module 201 samples the shape boundaries of the map elements in the multi-view image in sequence, converts each map element into a broken line composed of a fixed number of ordered two-dimensional point columns, completes the structured modeling of the map elements, and determines the vector representation of the map elements in each multi-view image.

[0086] The multi-task traffic element detection module 201 takes the center of the vehicle as the origin, the right side of the vehicle as the X direction, the vehicle direction as the Y direction, and the vertical upward as the Z direction; the three-dimensional bounding box of the traffic element in the global coordinate system is converted to the lidar coordinate system after translation and rotation, and then the converted traffic entity and vector map element are data fused and visualized within the predefined BEV space range, so that the traffic elements contained at the current moment can achieve coordinate alignment.

[0087] In specific implementation, the road scene relationship extraction module 202 defines the subject and object in each pair of relationships, uses the coordinates between the subject and the object to quantitatively calculate the distance relationship to obtain the quantitative distance, and then divides the quantitative distance into semantic qualitative distance relationships according to the application-oriented distance threshold; uses the relative orientation relationship to express the directional relationship between the subject and the object, and uses the subject as a reference for the object to determine the orientation relationship between the subject and the object; and uses the surface topology calculation method to determine the topological relationship between the subject and the object.

[0088] In a specific implementation, the road scene graph generation module 203 uses the traffic elements as nodes, and the spatial relationships and interactive relationships between different traffic elements as directed edges between nodes in the scene graph to generate a road scene graph.

[0089] In one embodiment, a computer device is provided, such as Figure 3 As shown, it includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the above-mentioned road scene graph generation method is implemented.

[0090] Specifically, the computer device may be a computer terminal, a server or a similar computing device.

[0091] In this embodiment, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program for executing the above-mentioned road scene graph generating method.

[0092] Specifically, computer-readable storage media include permanent and non-permanent, removable and non-removable media that can be used to store information by any method or technology. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer-readable storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, tape disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable storage media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0093] Obviously, those skilled in the art should understand that the modules or steps of the above-mentioned embodiments of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order from that here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the embodiments of the present invention are not limited to any specific combination of hardware and software.

[0094] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for generating a road scene graph, characterized in that: include: Taking the multi-view images acquired by the vehicle as input data, the multi-view images are input into a multi-task traffic element detection model, and traffic elements in the BEV space are obtained through the multi-task traffic element detection model, wherein the traffic elements include traffic entities and vector map elements; wherein the multi-task traffic element detection model is a detection model constructed based on an encoding-decoding structure; According to the position information of the traffic elements in the BEV space, the spatial relationship between different traffic elements is calculated, and according to the spatial relationship between different traffic elements and the attributes of traffic entities, the interactive relationship between different traffic elements is calculated; A road scene graph is constructed based on the traffic elements and the spatial relationships and interactive relationships between different traffic elements.

2. The road scene graph generation method according to claim 1, characterized in that: The traffic elements in the BEV space are obtained by the multi-task traffic element detection model, including: Taking the multi-view images of the vehicle as input data, vectorizing the map elements contained in the multi-view images through the map element vectorization module in the multi-task traffic element detection model; Extracting image features of the multi-view image through the encoding unit in the multi-task traffic element detection model, converting the image features into BEV features in the BEV space, inputting the BEV features into the traffic element prediction unit in the multi-task traffic element detection model, and obtaining the prediction result of the traffic element; Taking the center of the vehicle as the origin, the coordinates of the vector map elements and the traffic entities are aligned and unified into the same coordinate system in the BEV space to obtain the traffic entities and vector map elements in the BEV space.

3. The road scene graph generation method according to claim 2, characterized in that: Vectorizing the map elements contained in the multi-view image by a map element vectorization module in the multi-task traffic element detection model, including: The shape boundaries of the map elements in the multi-view image are sampled sequentially, and each map element is converted into a polyline consisting of a fixed number of ordered two-dimensional point sequences, so as to complete the structured modeling of the map elements and determine the vector representation of the map elements in each multi-view image.

4. The road scene graph generation method according to claim 2, characterized in that: Aligning the coordinates of the vector map elements and the traffic entities to the same coordinate system in the BEV space includes: Taking the center of the vehicle as the origin, the right side of the vehicle as the X direction, the vehicle direction as the Y direction, and the vertical upward as the Z direction; the three-dimensional bounding box of the traffic element in the global coordinate system is converted to the lidar coordinate system after translation and rotation, and then the converted traffic entity and vector map element are fused and visualized within the predefined BEV space range to achieve coordinate alignment of the traffic elements contained at the current moment.

5. The road scene graph generation method according to claim 1, characterized in that: According to the position information of the traffic elements in the BEV space, the spatial relationship between different traffic elements is calculated, wherein the spatial relationship includes a distance relationship, an orientation relationship and a topological relationship, including: Define the subject and object in each pair of relations, use the coordinates between the subject and the object to perform quantitative calculation of the distance relation, obtain the quantitative distance, and then divide the quantitative distance into a semantically qualitative distance relation according to an application-oriented distance threshold; The relative orientation relationship is used to express the directional relationship between the subject and the object, and the subject is used as the reference of the object to determine the orientation relationship between the subject and the object; The topological relationship between the subject and the object is determined using the surface topology calculation method.

6. The road scene graph generation method according to claim 2, characterized in that: The attributes of the traffic entity include the category, center point coordinates, bounding box parameters and speed of the traffic element in the prediction result of the traffic element.

7. The road scene graph generation method according to claim 1, characterized in that: According to the traffic elements and the spatial relationship and interaction relationship between different traffic elements, a road scene graph is constructed, including: The traffic elements are taken as nodes, and the spatial relationships and interactive relationships between different traffic elements are taken as directed edges between the nodes in the scene graph to generate a road scene graph.

8. A road scene graph generating device, characterized in that: include: A multi-task traffic element detection module is used to use the multi-view images acquired by the vehicle as input data, input the multi-view images into a multi-task traffic element detection model, and obtain traffic elements in the BEV space through the multi-task traffic element detection model, wherein the traffic elements include traffic entities and vector map elements; wherein the multi-task traffic element detection model is a detection model constructed based on an encoding-decoding structure; A road scene relationship extraction module, used to calculate the spatial relationship between different traffic entities and vector map elements according to the position information of the traffic elements in the BEV space, and to calculate the interactive relationship between different traffic entities and vector map elements according to the spatial relationship and the attributes of the traffic entities; The road scene graph generation module is used to construct a road scene graph according to the traffic elements and the spatial relationships and interactive relationships between the traffic elements.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the road scene graph generating method according to any one of claims 1 to 7 is implemented.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program for executing the road scene graph generation method according to any one of claims 1 to 7.

Images