A multi-modal fusion knowledge graph traffic dynamic occlusion tracking method and system

Abstract

The application discloses a kind of multimodal fusion knowledge graph's traffic dynamic occlusion tracking method and system, the method includes respectively to the feature extraction and target tracking of two-dimensional image and three-dimensional point cloud data of driving vehicle, obtain two-dimensional trajectory and three-dimensional trajectory;Based on the target tracking algorithm of adaptive weight, two-dimensional trajectory and three-dimensional trajectory are fused;Based on urban road traffic scene, ontology model is constructed;Vehicle trajectory is mapped into ontology model by semantic description, and urban scene knowledge graph is obtained;The semantic conversion of urban scene knowledge graph is carried out, and the occlusion target tracking is carried out by inference machine to establish rule, and target tracking result is obtained.The system includes first trajectory extraction module, second trajectory extraction module, trajectory fusion module, model construction module, atlas generation module and tracking inference module.Using the application can realize high-precision trajectory tracking under complex conditions.The application can be widely applied in target detection technical field.

Description

Technical Field

[0001] This invention relates to the field of target detection technology, and in particular to a traffic dynamic occlusion tracking method and system based on multimodal fusion knowledge graph. Background Technology

[0002] In recent years, the number of cars on the road has increased dramatically with the rapid pace of modernization in my country. This has led to increasingly prominent traffic safety hazards and frequent traffic accidents. To ensure the safety of life and property, artificial intelligence-based autonomous vehicles will be a disruptive and far-reaching product in the traditional automotive market. In autonomous driving systems, the environmental perception module is the most important component, and target detection is its most basic function. Therefore, how to effectively detect targets has always been a hot and challenging issue in the field of autonomous driving.

[0003] Most current target detection methods are based on image processing algorithms such as YOLOv5. However, image-based target detection methods use a single sensor to sample the real world, which inevitably leads to a problem: the camera's adaptability to changes in shadows, weather, and lighting is insufficient, resulting in low anti-interference capabilities in various environments. Furthermore, single-sensor methods struggle to adapt to complex real-world situations. In addition to the aforementioned single-sensor issues, occlusion areas exist during target detection, making it difficult to determine the target's tracking position. Existing target detection methods cannot provide more accurate and sufficient information under complex conditions, failing to achieve optimal choices for traffic safety. Summary of the Invention

[0004] To address the aforementioned technical problems, the present invention aims to provide a traffic dynamic occlusion tracking method and system based on multimodal fusion knowledge graph, which can achieve high-precision trajectory tracking under complex conditions.

[0005] The first technical solution adopted in this invention is: a traffic dynamic occlusion tracking method based on multimodal fusion knowledge graph, comprising the following steps:

[0006] Feature extraction and target tracking are performed on two-dimensional images of moving vehicles to obtain two-dimensional trajectories;

[0007] Feature extraction and target tracking are performed on the 3D point cloud data of the moving vehicle to obtain the 3D trajectory;

[0008] The target tracking algorithm based on adaptive weights fuses two-dimensional and three-dimensional trajectories to obtain the vehicle trajectory;

[0009] An ontology model is constructed based on urban road traffic scenarios;

[0010] By mapping vehicle trajectories to an ontology model through semantic description, a knowledge graph of urban scenes is obtained.

[0011] Semantic transformation is performed on the knowledge graph of urban scenes, and rules are established through inference engine to track occluded targets and obtain target tracking results.

[0012] Furthermore, the step of extracting features and tracking targets from the two-dimensional image of the moving vehicle to obtain a two-dimensional trajectory specifically includes:

[0013] Based on Yolov5s and the attention mechanism, feature extraction is performed on the two-dimensional image of the moving vehicle to obtain the target two-dimensional features;

[0014] Target tracking is performed based on the DeepSORT algorithm and two-dimensional features of the target to obtain a two-dimensional trajectory.

[0015] This optimization step allows for further understanding of the relationship between global and local information, which helps the network better understand input features.

[0016] Furthermore, the step of extracting features and tracking targets from the 3D point cloud data of the moving vehicle to obtain a 3D trajectory specifically includes:

[0017] Based on PointRCNN and attention module, feature extraction is performed on the 3D point cloud data of the driving vehicle to obtain the target 3D features;

[0018] Target tracking is performed based on the AB3DMOT algorithm and the target's three-dimensional features to obtain a three-dimensional trajectory.

[0019] This preferred step involves adding an attention module to the RoI head to improve the performance of PointRCNN. The attention module enhances the ability to capture high-dimensional global features after point cloud region pooling.

[0020] Furthermore, the step of fusing the two-dimensional and three-dimensional trajectories to obtain the vehicle trajectory in the target tracking algorithm based on adaptive weights specifically includes:

[0021] The fusion weight value of the two-dimensional trajectory is determined based on the two-dimensional trajectory information;

[0022] The number of 3D point clouds is calculated based on the horizontal and vertical angles of the lidar, and the relationship between the number of point clouds in the horizontal and vertical directions and the distance is obtained.

[0023] The relationship between the number of points in the horizontal and vertical directions and the distance is normalized and averaged to obtain the relationship between the number of points in the horizontal and vertical directions.

[0024] The fusion weight value of the 3D trajectory is determined based on the relationship between the number of point clouds and the distance.

[0025] The vehicle trajectory is obtained by weighted fusion of the two-dimensional trajectory and the three-dimensional trajectory based on the fusion weight values ​​of the two-dimensional trajectory and the fusion weight values ​​of the three-dimensional trajectory.

[0026] Through this optimization step, the fusion weight values ​​of two-dimensional and three-dimensional trajectories are set based on the relationship between trajectory information and distance, which greatly improves the reliability of the fused trajectory.

[0027] Furthermore, the vehicle trajectory is expressed as follows:

[0028]

[0029]

[0030] Among them, track final Represents vehicle trajectory, track 2D Represents a two-dimensional trajectory, track 3D Let β represent the 3D trajectory, d represent the fusion weight value of the 3D trajectory, and D represent the distance between the traffic participant and the LiDAR. Lidar f represents the effective range of the lidar. sidefinal (d) represents the relationship between the number of point clouds on the normalized sides and the distance, f rearfinal (d) represents the relationship between the number of point clouds on the back side after normalization and the distance.

[0031] Furthermore, the ontology model includes traffic participants, environmental information, behavioral information, and road information, wherein:

[0032] The traffic participants include all vehicle information in a given driving scenario;

[0033] The behavioral information includes the driving behavior of different vehicles in the driving scenario;

[0034] The road information includes the basic road network and physical structure in the driving scenario, and the physical structure includes basic traffic facilities.

[0035] Furthermore, the urban scene knowledge graph is logically divided into a pattern layer and a data layer, wherein:

[0036] The pattern layer stores the task information defined by the ontology model;

[0037] The data layer stores data information from specific scenarios.

[0038] Furthermore, the step of semantically transforming the urban scene knowledge graph and establishing rules through an inference engine to track occluded targets and obtain target tracking results specifically includes:

[0039] Semantic transformation is performed on the knowledge graph of urban scenes to obtain a semantic description with reasoning and computational utility.

[0040] Based on empirical knowledge and semantic descriptions with reasoning and computational utility, rules are written for traffic scenarios with potential occlusion targets to obtain target occlusion evaluation rules.

[0041] The occluded target is inversely inferred based on the inference engine and the target occlusion evaluation rules to obtain the target tracking result.

[0042] The second technical solution adopted in this invention is: a traffic dynamic occlusion tracking system based on multimodal fusion knowledge graph, comprising:

[0043] The first trajectory extraction module is used to extract features and track targets from the two-dimensional image of the moving vehicle to obtain a two-dimensional trajectory;

[0044] The second trajectory extraction module is used to extract features and track targets from the three-dimensional point cloud data of the moving vehicle to obtain the three-dimensional trajectory.

[0045] The trajectory fusion module fuses two-dimensional and three-dimensional trajectories based on an adaptive weighted target tracking algorithm to obtain the vehicle trajectory;

[0046] The model building module constructs an ontology model based on urban road traffic scenarios;

[0047] The knowledge graph generation module maps vehicle trajectories to the ontology model through semantic description, thereby obtaining a knowledge graph of the urban scene.

[0048] The tracking inference module is used to perform semantic transformation on the urban scene knowledge graph and to establish rules through the inference engine to track occluded targets and obtain target tracking results.

[0049] The beneficial effects of the method and system of this invention are as follows: This invention improves the feature extraction performance of two-dimensional targets by integrating an attention mechanism with Yolov5s; it enhances the extraction performance of point clouds regarding global features by adding an attention mechanism module to PointRCNN; it adopts a target tracking algorithm based on adaptive weights to better associate the trajectory features detected by radar with the target trajectory features corresponding to its image, which can effectively improve the target trajectory tracking accuracy and robustness; it constructs a knowledge graph of urban scenes to describe the interaction relationships between entities in the scene, performs semantic transformation on the scene description information, and establishes rules through an inference engine to track occluded targets, thereby improving the target tracking confidence; ultimately, it achieves high-precision trajectory tracking under complex conditions, further improving the safety of vehicles driving on urban roads. Attached Figure Description

[0050] Figure 1This is a flowchart of the steps of a traffic dynamic occlusion tracking method based on a multimodal fusion knowledge graph according to the present invention;

[0051] Figure 2 This is a structural block diagram of a traffic dynamic occlusion tracking system based on a multimodal fusion knowledge graph according to the present invention;

[0052] Figure 3 This is a general framework diagram of a traffic dynamic occlusion tracking method based on multimodal fusion knowledge graph of the present invention;

[0053] Figure 4 (a) is a Yolov5s structure diagram of a traffic dynamic occlusion tracking method based on a multimodal fusion knowledge graph according to the present invention;

[0054] Figure 4 (b) is the original Yolov5s structure diagram;

[0055] Figure 5 (a) is a PointRCNN structure diagram of a traffic dynamic occlusion tracking method based on a multimodal fusion knowledge graph according to the present invention;

[0056] Figure 5 (b) is the original PointRCNN architecture diagram;

[0057] Figure 6 This is a schematic diagram of the effective detection area of ​​lidar and camera in a traffic dynamic occlusion tracking method based on multimodal fusion knowledge graph according to the present invention.

[0058] Figure 7 This is a schematic diagram of the lateral section calculation point cloud of a traffic dynamic occlusion tracking method based on a multimodal fusion knowledge graph according to the present invention;

[0059] Figure 8 This is a schematic diagram of the longitudinal section calculation point cloud of a traffic dynamic occlusion tracking method based on a multimodal fusion knowledge graph according to the present invention;

[0060] Figure 9 This is a schematic diagram of an ontology model based on a traffic scene for a traffic dynamic occlusion tracking method based on a multimodal fusion knowledge graph according to the present invention. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0062] Reference Figure 1 and Figure 3This invention provides a traffic dynamic occlusion tracking method based on multimodal fusion knowledge graph, the method comprising the following steps:

[0063] S1. Perform feature extraction and target tracking on the two-dimensional image of the moving vehicle to obtain a two-dimensional trajectory;

[0064] Specifically, to accurately and continuously extract the trajectories of traffic participants, obtaining high-quality object detection results is crucial. Attention mechanisms can effectively process data from different dimensions and maximize data utilization with limited resources. Generally, various attention mechanisms enhance salient features and suppress insignificant features. Therefore, referring to... Figure 4 (a) and Figure 4 (b) This invention uses YOLOv5s as the baseline. To improve the network's performance in extracting object features, an attention mechanism is added to the model. Attention modules are added to four positions in the YOLOv5s backbone, and the number of layers in the attention modules should be the same as the number of layers in the upper-level modules. After adding the attention mechanism, the connection between global and local information is further obtained, which helps the network better understand the input features. This results in a better two-dimensional image feature extraction effect than the original YOLOv5s. After feature extraction using the YOLOv5s model with the added attention mechanism, the target's two-dimensional features are obtained. Then, target tracking is performed according to the DeepSORT algorithm and the target's two-dimensional features to obtain the two-dimensional trajectory.

[0065] S2. Perform feature extraction and target tracking on the 3D point cloud data of the moving vehicle to obtain the 3D trajectory;

[0066] Specifically, because a single sensor is insufficient to provide more accurate and sufficient information under complex conditions, this invention adds a LiDAR sensor to achieve 3D feature detection. This invention uses PointRCNN as the baseline. OpenPCDET is used to acquire the detection results from the point cloud. The PointRCNN structure in OpenPCDET consists of four parts: Backbone 3-D, Backbone 2D, Dense Head, and RoI Head. Local and global features are key factors in object detection using point cloud data. However, the characteristics of point clouds easily change with distance, object shape, etc., and local features of point clouds are difficult to capture. Global features have greater stability than local features. Therefore, object detection performance can be enhanced by improving the ability to extract global features from point cloud data. (Refer to...) Figure 5 (a) and Figure 5(b) An attention module is added to the RoI head to improve the performance of PointRCNN. The attention module enhances the ability to capture high-dimensional global features after point cloud region pooling. After feature extraction using PointRCNN with the added attention module, the target's 3D features are obtained; then, target tracking is performed based on the AB3DMOT algorithm and the target's 3D features to obtain the 3D trajectory.

[0067] S3. The target tracking algorithm based on adaptive weights fuses the two-dimensional trajectory and the three-dimensional trajectory to obtain the vehicle trajectory;

[0068] Specifically, the trajectory extracted by this invention includes two-dimensional and three-dimensional trajectories, and the fusion of the two-dimensional and three-dimensional trajectories can be expressed as follows:

[0069] track final =f(track) 2D ,track 3D ,α,β) (1)

[0070] Among them, track final Represents vehicle trajectory, track 2D Represents two-dimensional trajectory information, track 3D The three-dimensional trajectory information is represented by α and β, which are weighting factors for the two-dimensional and three-dimensional trajectories, respectively, and are used to represent the proportion of two-dimensional and three-dimensional trajectory information in the final trajectory.

[0071] The amount of data in the point cloud changes as the distance between the object and the lidar sensor changes. The quality of the point cloud degrades significantly as the object moves further away from the lidar sensor. In contrast, detection and tracking methods related to 2D images are less sensitive to distance; even if an object is far from the camera, meaning it can only be identified within a few pixels of the image, it can still be detected and tracked. Therefore, compared to 2D image detection methods, the performance of 3D point cloud detection is more easily affected by the distance between the object and the data collection device.

[0072] Reference Figure 6 Taking a T-junction as an example, the difference between the effective target detection and tracking range of lidar and that of a camera is as follows: Figure 8 As shown, there are three scenarios depending on the distance between the object and the data collection device. The first scenario is similar to car A, where it is not within the detection area of ​​the lidar but within the detection area of ​​the camera. In this case, only two-dimensional trajectory information can be acquired. Therefore, the trajectory information of car A can be represented as track. final =track 2D The second scenario is similar to car B. The object is located in the blind spot of the camera and can only be detected by LiDAR; in this case, the trajectory information of car B can be obtained from the track...final =track 3D This means that if the object is vehicle C, then both the lidar and the vehicle's camera can detect it simultaneously. Its trajectory can be represented by a weighted average of the two-dimensional and three-dimensional trajectory information, as shown in the following expression:

[0073] track final =(α*track) 2D +β*track 3D ) / (α+β) (2).

[0074] S3.1 Determine the fusion weight value of the two-dimensional trajectory based on the two-dimensional trajectory information;

[0075] Specifically, two-dimensional trajectory information is not sensitive to distance; it can be accurately detected and tracked regardless of whether the object is far or near. Therefore, in this specific embodiment of the invention, the fusion weight value α of the two-dimensional trajectory is determined to be a fixed number 1. The result of three-dimensional trajectory information is sensitive to distance; the quality of the point cloud decreases significantly with increasing distance. Therefore, the fusion weight value of the three-dimensional trajectory is determined by the relationship between the number of points in the point cloud and the distance. Thus, Formula 2 can be transformed into the following form:

[0076]

[0077] Among them, track final Represents vehicle trajectory, track 2D Represents a two-dimensional trajectory, track 3D Let β represent the 3D trajectory, d represent the distance between the traffic participant and the lidar, and D represent the distance between the traffic participant and the lidar. Lidar This indicates the effective range of the lidar.

[0078] S3.2 Calculate the number of three-dimensional point clouds based on the horizontal and vertical angles of the lidar, and obtain the relationship between the number of point clouds in the horizontal and vertical directions and the distance;

[0079] Specifically, since the quality of 3D detection and tracking results is strongly related to distance, and distance has a significant impact on the number of point clouds, the relationship between the weighting coefficients of 3D trajectory information, distance, and the number of point clouds is expressed as follows:

[0080] β=f(d),N points =f(d) (4)

[0081] Where d represents the distance between the traffic participant and the lidar, and N points This indicates the number of point clouds.

[0082] The connection between β and d can be achieved through N pointsThe relationship between d and 'l' is used to establish that the detected object can be considered rectangular, with its length, width, and height defined as l, w, and h. The installation height of the lidar is h0. Because the lidar has different angular resolutions in the horizontal and vertical directions, the number of point clouds should be considered from both directions respectively.

[0083] The geometric relationship between the vertical lidar and traffic participants is as follows: Figure 7 As shown, the horizontal angle can be calculated using the following expression.

[0084]

[0085]

[0086] Where h represents the height of the traffic participant, h0 represents the installation height of the lidar, λ represents the angle of the first laser beam, and θ represents the angle of the second laser beam.

[0087] Therefore, the angle between the first laser beam and the second laser beam is calculated using the following expression:

[0088]

[0089] Where, β v This indicates the angle between the first and second laser beams in the vertical direction.

[0090] Finally, the number of point clouds in the traffic participant cross-section is calculated using the following expression:

[0091]

[0092] Where, N v μ represents the number of point clouds in the cross-section of traffic participants. v This indicates the angular resolution in the vertical direction.

[0093] The geometric relationship between the horizontal lidar and traffic participants is as follows: Figure 8 As shown, the vertical angle can be calculated using the following expression:

[0094]

[0095] Where, β h The angle between the first and second laser beams in the horizontal direction is represented by l, and the length of the traffic participant is represented by l.

[0096] Based on the angles calculated from the two directions above, the total number of point clouds of traffic participants can be calculated using the following expression:

[0097]

[0098] Where, N v N represents the number of point clouds in the cross-section of traffic participants. h μ represents the number of point clouds in the longitudinal section of traffic participants. h This indicates the angular resolution in the horizontal direction.

[0099] According to equation (10), the number of point clouds of an object is determined only by the installation height of the lidar, the distance between the lidar and the object, and the angular resolution of the lidar:

[0100] N points =f(w,l,h,d,h0,μ) h ,μ v (11)

[0101] Then, through experimental methods, N was obtained. points The relationship between the distance d and the vehicle and the LiDAR, as mentioned above, is also related. Depending on whether it is horizontal or vertical, the horizontal and vertical directions N can be calculated separately. points The function of d is expressed as follows:

[0102]

[0103]

[0104] Among them, f sidefinal (d) represents the relationship between the number of normalized side point clouds and distance, f rearfinal (d) represents the relationship between the normalized back point cloud count and distance, f side (d) represents the relationship between the number of unnormalized side point clouds and distance, f rear (d) represents the relationship between the number of unnormalized back point clouds and distance.

[0105] S3.3 Normalize and average the relationship between the number of point clouds and the distance in the horizontal and vertical directions to obtain the relationship between the number of point clouds and the distance.

[0106] Specifically, formulas 12 and 13 are normalized to obtain a function with a range of [0,1]. Finally, based on the horizontal and vertical functional relationships, an average is performed to obtain the final point cloud data and the relationship between distance.

[0107] S3.4 Determine the fusion weight value of the 3D trajectory based on the relationship between the number of point clouds and the distance;

[0108] Specifically, the fusion weight value of the three-dimensional trajectory is expressed as follows:

[0109]

[0110] S3.5. The two-dimensional trajectory and the three-dimensional trajectory are weighted and fused based on the fusion weight value of the two-dimensional trajectory and the fusion weight value of the three-dimensional trajectory to obtain the vehicle trajectory.

[0111] Specifically, the fusion weight value of the three-dimensional trajectory obtained in step S3.4 is substituted into formula 3 to obtain the functional relationship between the fusion of the two-dimensional trajectory and the three-dimensional trajectory. Through this relationship, the vehicle trajectory is finally calculated.

[0112] S4. Construct an ontology model based on urban road traffic scenarios;

[0113] Specifically, refer to Figure 9 This invention, in its specific embodiments, is based on the OWL language and uses the Stanford University open-source software Protege as a tool. It constructs the ontology model of this invention based on the logical scenarios within the three-layer abstract hierarchy of scenarios proposed by the German Pegasus project. This model mainly includes traffic participants, environmental information, behavioral information, and road information, wherein:

[0114] The traffic participants include all vehicle information in a given driving scenario.

[0115] The environmental information includes macro-environmental information of the driving scenario, such as time information, light intensity, weather conditions, etc.

[0116] The behavioral information includes the driving behavior of different vehicles in a driving scenario, including longitudinal and lateral behaviors.

[0117] The road information includes the basic road network and physical structure in the driving scenario. The basic road network mainly consists of T-junctions and crossroads, while the physical structure consists of some basic traffic facilities existing in the road, such as traffic lights and warning signs.

[0118] S5. By semantically describing the vehicle trajectory, we can map it onto the ontology model to obtain a knowledge graph of the urban scene.

[0119] Specifically, in this embodiment of the invention, vehicle information and vehicle trajectory information obtained from roadside sensors are fused through semantic description and mapped onto the constructed ontology model to form a knowledge graph for each specific scenario. The constructed knowledge graph is then stored in the graph database Neo4j. The constructed knowledge graph can be logically divided into two main layers: a schema layer and a data layer. The schema layer primarily handles the task information defined by the ontology model, while the data layer stores the data information within the specific scenario. After semantic description of the vehicle information and vehicle trajectory information obtained from roadside sensors is completed, the input data is converted into triples, and the output data is stored in the knowledge graph for a more intuitive organization and aggregation.

[0120] S6. Perform semantic transformation on the urban scene knowledge graph, and establish rules through inference engine to track occluded targets and obtain target tracking results.

[0121] S6.1 Perform semantic transformation on the urban scene knowledge graph to obtain a semantic description with reasoning and computational utility;

[0122] Specifically, since the knowledge graph constructed in step S5 is based on the OWL ontology language to model traffic scenes, when it is necessary to make inference judgments about the position of occluded targets in the scene, judgment rules are also required. However, since the OWL language is only a descriptive language and cannot perform any reasoning or computation, a semantic conversion step is required for the OWL language. This step mainly converts the semantic description based on the OWL language into a semantic description based on the Prolog language, and then uses an inference engine to achieve real-time target tracking.

[0123] S6.2. Based on empirical knowledge and semantic descriptions with reasoning and computational utility, rules are written for traffic scenarios with potential occlusion targets to obtain target occlusion evaluation rules.

[0124] Specifically, before inferring the location of occluded targets in a traffic scene, it is necessary to first determine whether there are occluded objects in the scene. This requires combining expert experience and knowledge. For common traffic scenes with potential occlusion targets, the aforementioned Prolog language is used to write potential target occlusion judgment rules. The words in the rule statements need to use the vocabulary from the previously constructed ontology model.

[0125] S6.3. Based on the inference engine and target occlusion evaluation rules, reverse inference is performed on the occluded target to obtain the target tracking result.

[0126] Specifically, after the target occlusion assessment rules are written, the system can perform online inference and judgment on potentially occluded targets in traffic scenes based on these rules. In this specific embodiment, SWI-Prolog is selected as the inference engine for target occlusion tracking. The inference process of SWI-Prolog uses inverse reasoning methods such as pattern matching and backtracking to determine whether there are occluded targets and to judge the possible locations of the targets, thereby achieving target tracking. The inference steps are as follows: First, the risk(Ego,X) rule is searched in the risk database. This rule describes the risk relationship between the main vehicle and other elements in the scene. These elements include dynamic elements such as other traffic participants, vehicles, and pedestrians, as well as static elements such as traffic lights and temporary obstacles. After the risk(Ego,X) rule is found, corresponding rule matching is performed. This step mainly determines whether there is a call to another rule in the matched scene. If a call exists, the system will continue to search for the corresponding rule. If no rule is called, the next rule clause will be executed. When a rule clause doesn't match, the system returns to the risl(Ego,X) rule for further searching. If a rule clause successfully matches, the inference engine checks if it's the last rule clause. If not, it continues matching the next rule clause, processing clauses as they are matched. This continues until the rule clause becomes the last. At this point, a backtracking check is performed. If a backtracking point exists, the system returns to the clause containing that backtracking point and performs the previous matching steps. Finally, when the last rule clause is reached and no backtracking point exists, the final matching result is output.

[0127] Reference Figure 2 A traffic dynamic occlusion tracking system based on multimodal fusion knowledge graph, comprising:

[0128] The first trajectory extraction module is used to extract features and track targets from the two-dimensional image of the moving vehicle to obtain a two-dimensional trajectory;

[0129] The second trajectory extraction module is used to extract features and track targets from the three-dimensional point cloud data of the moving vehicle to obtain the three-dimensional trajectory.

[0130] The trajectory fusion module fuses two-dimensional and three-dimensional trajectories based on an adaptive weighted target tracking algorithm to obtain the vehicle trajectory;

[0131] The model building module constructs an ontology model based on urban road traffic scenarios;

[0132] The knowledge graph generation module maps vehicle trajectories to the ontology model through semantic description, thereby obtaining a knowledge graph of the urban scene.

[0133] The tracking inference module is used to perform semantic transformation on the urban scene knowledge graph and to establish rules through the inference engine to track occluded targets and obtain target tracking results.

[0134] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0135] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A traffic dynamic occlusion tracking method of a multi-modal fusion knowledge graph, characterized in that, The method comprises the following steps: feature extraction and target tracking are performed on the two-dimensional image of the running vehicle to obtain a two-dimensional trajectory; feature extraction and target tracking are performed on the three-dimensional point cloud data of the running vehicle to obtain a three-dimensional trajectory; a target tracking algorithm based on adaptive weights is used to fuse the two-dimensional trajectory and the three-dimensional trajectory to obtain a vehicle trajectory; an ontology model is constructed based on an urban road traffic scene; the vehicle trajectory is mapped into the ontology model through semantic description to obtain an urban scene knowledge graph; semantic conversion is performed on the urban scene knowledge graph, and a reasoning machine is used to establish rules for occluded target tracking to obtain a target tracking result; the ontology model comprises traffic participants, environmental information, behavior information and road information, wherein: the traffic participants comprise all vehicle information in a certain driving scene; the behavior information comprises driving behaviors of different vehicles in the driving scene; the road information comprises a basic road network and a physical structure in the driving scene, and the physical structure comprises basic traffic facilities; the urban scene knowledge graph is logically divided into a mode layer and a data layer, wherein: the mode layer stores task information defined by the ontology model; the data layer stores data information in a specific scene; the step of performing semantic conversion on the urban scene knowledge graph and using a reasoning machine to establish rules for occluded target tracking to obtain a target tracking result specifically comprises: performing semantic conversion on the urban scene knowledge graph to obtain semantic description with reasoning calculation utility; based on experience knowledge and semantic description with reasoning calculation utility, rules are written for a potential occluded target traffic scene to obtain target occlusion evaluation rules; based on a reasoning machine and the target occlusion evaluation rules, reverse reasoning is performed on the occluded target to obtain a target tracking result.

2. The traffic dynamic occlusion tracking method of claim 1, wherein, The step of performing feature extraction and target tracking on the two-dimensional image of the running vehicle to obtain a two-dimensional trajectory specifically comprises: based on Yolov5s and an attention mechanism, feature extraction is performed on the two-dimensional image of the running vehicle to obtain target two-dimensional features; based on a DeepSORT algorithm and the target two-dimensional features, target tracking is performed to obtain a two-dimensional trajectory. 3.The traffic dynamic occlusion tracking method of multi-modal fusion knowledge graph according to claim 1, characterized in that, The step of performing feature extraction and target tracking on the three-dimensional point cloud data of the running vehicle to obtain a three-dimensional trajectory specifically comprises: based on PointRCNN and an attention module, feature extraction is performed on the three-dimensional point cloud data of the running vehicle to obtain target three-dimensional features; based on an AB3DMOT algorithm and the target three-dimensional features, target tracking is performed to obtain a three-dimensional trajectory.

4. The traffic dynamic occlusion tracking method of claim 1, wherein, The step of using a target tracking algorithm based on adaptive weights to fuse the two-dimensional trajectory and the three-dimensional trajectory to obtain a vehicle trajectory specifically comprises: a fusion weight value of the two-dimensional trajectory is determined based on two-dimensional trajectory information; a relationship between the number of point clouds and the distance in the horizontal and vertical directions is obtained by calculating the number of three-dimensional point clouds based on the angles of the horizontal and vertical directions of the laser radar; the relationship between the number of point clouds and the distance in the horizontal and vertical directions is normalized and averaged to obtain a relationship between the number of point clouds and the distance; a fusion weight value of the three-dimensional trajectory is determined based on the relationship between the number of point clouds and the distance. The two-dimensional trajectory and the three-dimensional trajectory are weightedly fused based on a fusion weight value of the two-dimensional trajectory and a fusion weight value of the three-dimensional trajectory, to obtain the vehicle trajectory. 5.The traffic dynamic occlusion tracking method of the multi-modal fusion knowledge graph according to claim 1, characterized in that, The vehicle trajectory has an expression as follows: wherein, represents a vehicle trajectory, represents a two-dimensional trajectory, represents a three-dimensional trajectory, represents a weight factor for a three-dimensional trajectory, represents a distance between a traffic participant and a lidar, represents an effective distance of a lidar, represents a relationship between the number of point clouds and distance of a normalized side, represents a relationship between the number of point clouds and distance of a normalized back. 6.A traffic dynamic occlusion tracking system of multi-modal fusion knowledge graph, characterized in that, The method comprises the following steps: The first trajectory extraction module is configured to perform feature extraction and target tracking on a two-dimensional image of a vehicle to obtain a two-dimensional trajectory. The second trajectory extraction module is configured to perform feature extraction and target tracking on three-dimensional point cloud data of the vehicle to obtain a three-dimensional trajectory. The trajectory fusion module is configured to fuse the two-dimensional trajectory and the three-dimensional trajectory based on an adaptive weight target tracking algorithm to obtain the vehicle trajectory. The model construction module is configured to construct an ontology model based on an urban road traffic scene. The graph generation module is configured to map the vehicle trajectory into the ontology model through semantic description to obtain an urban scene knowledge graph. The tracking reasoning module is configured to perform semantic conversion on the urban scene knowledge graph, and perform occlusion target tracking through a reasoning machine to obtain a target tracking result. The ontology model comprises traffic participants, environmental information, behavior information and road information, wherein the traffic participants comprise all vehicle information in a certain driving scene, the behavior information comprises driving behaviors of different vehicles in the driving scene, and the road information comprises a basic road network and a physical structure in the driving scene, and the physical structure comprises basic traffic facilities. The urban scene knowledge graph is logically divided into a schema layer and a data layer, wherein the schema layer stores task information defined by the ontology model, and the data layer stores data information in a specific scene. The semantic conversion on the urban scene knowledge graph and the occlusion target tracking through the reasoning machine to obtain the target tracking result specifically comprise the following steps: performing semantic conversion on the urban scene knowledge graph to obtain semantic description with reasoning calculation utility; performing rule writing on a traffic scene of a potential occlusion target based on experience knowledge and the semantic description with reasoning calculation utility to obtain a target occlusion evaluation rule; and performing reverse reasoning on the occlusion target based on the reasoning machine and the target occlusion evaluation rule to obtain the target tracking result.

Images