A scene flow digital twin method and system based on dynamic trajectory flow

Abstract

The present invention discloses a scene flow digital twin method and system based on dynamic trajectory flow. First, a detection and tracking integrated multimodal fusion perception enhancement network is proposed to achieve accurate extraction and identification of target semantic trajectory while greatly improving network processing speed; and a target trajectory prediction method combining road layout and traffic force coupling relationship is proposed to model the temporal evolution law of semantics in the scene and achieve target trajectory prediction in the traffic operation environment; based on trajectory extraction and semantic identification and predicted motion trajectory, the temporal evolution law of traffic situation at the meso level is modeled to obtain a scene flow digital twin based on the real target dynamic trajectory flow. The present invention can effectively achieve accurate extraction and identification of target semantic trajectory, while visualizing scene flow digital twin, and provide decision support for precise traffic control services.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic control and relates to a scene flow digital twin method and system based on dynamic trajectory flow. Background Art

[0002] Deep learning is an important branch of artificial intelligence. Artificial intelligence based on deep learning architecture has been widely used in various fields such as computer vision, natural language processing, sensor fusion, biometrics, and autonomous driving. Relevant departments have established global industry reference standards for defining automated or autonomous vehicles to assess six levels (L0 to L5) of autonomous driving technology. At present, autonomous driving is restricted by factors such as laws and management policies. It will take some time for L4 and L5 autonomous driving vehicles to be on the road, but L3 autonomous driving technology with restrictions (that is, the driver does not need to monitor the road conditions, and the system can achieve full control of the vehicle under special conditions) is expected to be realized in the next five years. As an essential component of L3 to L5 autonomous driving technology, the advanced driver assistance system (ADAS) needs to complete multiple functions such as perception, fusion, planning, decision-making and warning. Due to the complex and changeable traffic operating conditions in real road scenes, this brings many severe challenges to autonomous driving technology based on computer vision. Such as road structure, road width, road quality, light and dark when driving, cold and warm climate, traffic safety facilities, traffic signals, traffic markings and road traffic signs.

[0003] Faced with a highly complex traffic operating environment, relying solely on widely deployed visual sensing equipment, the uncertainty of objective natural conditions poses challenges to the accuracy of visual perception and the robustness of algorithms.

[0004] Most existing multimodal information fusion methods only focus on the detection of multiple targets in traffic operation environments and multi-target tracking based on detectors. Obtaining high-level semantic traffic information (target motion trajectory information and coupling relationships, abnormal driving behaviors, etc.) brings challenges to multimodal information fusion perception.

[0005] The intelligent roadside system not only needs to perceive the traffic participants in real time, but more importantly, it needs to understand the traffic behavior and scene flow within the perception boundary of the traffic operation environment with the help of edge computing resources and computing power. The macro traffic situation is not enough to describe the influence and state changes between vehicles within the traffic flow. The scene flow digital twin and simulation deduction that reflect the traffic situation at the meso-micro level are challenging. Summary of the invention

[0006] The purpose of the present invention is to solve the problems in the prior art and provide a scene flow digital twin method and system based on dynamic trajectory flow, which can effectively realize the accurate extraction and identification of target semantic trajectory, and visualize the scene flow digital twin at the same time to provide decision support for precise traffic control services.

[0007] In order to achieve the above object, the present invention adopts the following technical solutions:

[0008] A scene flow digital twin method based on dynamic trajectory flow, comprising:

[0009] Construct a detection and tracking integrated multi-modal fusion perception enhancement network to extract and identify the target semantic trajectory;

[0010] The road traffic semantics are extracted to obtain a highly parameterized top view of the virtual road layout that has a mapping relationship with the real traffic scene;

[0011] Based on the virtual road layout top view, a road layout traffic semantic grid coding vector is obtained;

[0012] Based on the impact of other targets in the traffic scene on a certain target, a target coupling relationship model is constructed;

[0013] Construct a traffic force constraint model based on the target coupling relationship model and the actual road layout;

[0014] Based on the traffic force constraint model and the traffic semantic grid coding vector of the road layout, a long and short-term memory trajectory prediction network based on the traffic force coupling relationship constraint is established;

[0015] Based on the long short-term memory trajectory prediction network constrained by traffic force coupling relations, the target's motion trajectory is predicted to obtain the predicted motion trajectory;

[0016] Based on trajectory extraction, semantic recognition and predicted motion trajectory, the temporal evolution law of the traffic situation at the meso-level is modeled to obtain the scene flow digital twin based on the real target dynamic trajectory flow.

[0017] A further improvement of the present invention is:

[0018] Construct a detection and tracking integrated multimodal fusion perception enhancement network to extract and identify the target semantic trajectory, specifically:

[0019] The detection and tracking integrated multimodal fusion perception enhancement network includes a multimodal fusion perception enhancement module and a detection and tracking integrated network; the multimodal fusion perception enhancement module includes a resolution attention enhancement module and a feature fusion enhancement model;

[0020] The resolution attention enhancement module is used to learn the invariant feature representation of different modal information;

[0021] The feature fusion enhancement model defines a feature-related tensor pool, aggregates the output tensors of each modal convolution into the tensor pool for feature fusion, and outputs the fused features as the input of the main network;

[0022] The detection and tracking integrated network includes a main network and three sub-networks; the main network is a 3D parameter sharing convolution main network, which acts as a feature extractor to extract different features and send them to the three sub-networks respectively;

[0023] The three subnetworks are motion reasoning subnetwork, driving behavior recognition subnetwork and occlusion recognition subnetwork. The motion reasoning subnetwork is used to track the trajectory of the object; the driving behavior recognition subnetwork is used to recognize the driving behavior; and the occlusion recognition subnetwork is used to recognize the occluded part of the target.

[0024] The road traffic semantics are extracted to obtain a highly parameterized virtual road layout top view that has a mapping relationship with the real traffic scene, specifically:

[0025] Based on the coupling relationship between the road topology structure and the running trajectory of traffic participants in the traffic scene, the road layout traffic semantics is highly parameterized;

[0026] Construct a virtual road layout top view extraction cascade network parameterized by real scenes. Based on the highly parameterized road layout traffic semantics, a virtual road layout top view extraction cascade network parameterized by virtual-real hybrid training is constructed.

[0027] Based on the pixel space mapping relationship between the real traffic scene image and the trained virtual road layout top view, the virtual-reality mapping of the road layout is constructed to obtain a highly parameterized virtual road layout top view with a mapping relationship with the real traffic scene.

[0028] Based on the coupling relationship between the road topology structure and the running trajectory of traffic participants in the traffic scene, the road layout traffic semantics is highly parameterized, specifically:

[0029] Based on the topological attributes of the main road starting and ending points and the auxiliary road distance, line shape and position road intersection relationship in the traffic scene, the road layout attributes of the number, width and whether the lanes are one-way, the traffic sign attributes of the lane speed limit and lane line shape, and the pedestrian area attributes of the width of crosswalks and sidewalks that constrain pedestrian behavior, all attributes are assigned a unique ID from a data perspective to realize the road layout traffic semantic parameterization.

[0030] The construction of a virtual road layout top view extraction cascade network parameterized by real scenes is based on the road layout traffic semantics being highly parameterized, and a virtual road layout top view extraction cascade network parameterized by hybrid training combining virtuality and reality is specifically as follows:

[0031] Collect RGB images of road traffic, and extract them through semantic segmentation network to obtain the semantic top view of the real road;

[0032] Based on the simulator, a fully annotated simulated road image is sampled to obtain a top view of the simulated road;

[0033] Feature extraction is performed on the real road semantic top view and the simulated road top view respectively, and a virtual-real adversarial loss function is established based on virtual-real hybrid training.

[0034] Iterate the loss function to bridge the gap between the simulated road top view and the real road semantic top view;

[0035] The virtual-real confrontation loss function is:

[0036]

[0037] in, represents the loss function under real data supervision, represents the supervised loss function under simulated data, and the scalar λ r and λ s It represents the importance weights of real data and simulated data.

[0038] Based on the pixel space mapping relationship between the real traffic scene image and the trained virtual road layout top view, the cascade network is extracted to construct the virtual-real mapping of the road layout, and a highly parameterized virtual road layout top view with a mapping relationship with the real traffic scene is obtained, specifically:

[0039] Based on the grid coding algorithm of multi-scale adaptive search, the historical trajectory of the target in the real traffic scene provided by the integrated detection and tracking network is encoded into the top view of the virtual road layout to obtain the virtual coordinate trajectory and the corresponding road layout parameters. At the same time, the obtained virtual coordinate trajectory and the corresponding road layout parameters are integrated to obtain the road layout traffic semantic grid coding vector.

[0040] The target historical trajectory in the real traffic scene provided by the detection and tracking integrated network is the acquired trajectory extraction and semantic recognition.

[0041] Based on the impact of other targets in the traffic scene on a certain target, a target coupling relationship model is constructed, specifically:

[0042] The inter-target forces are established based on the radial kernel function, the inter-target influence weights are established by target type and inter-target distance, and the inter-target relationships are coupled based on the weighted summation of the forces acting on each target, thus constructing a target coupling relationship model.

[0043] The target coupling relationship model is At time t, the impact of other targets in the traffic scene on a target i is:

[0044]

[0045] in, represents the interaction between targets i and j at time t; is a weight vector used to express the differences in interactions between different moving targets; n(t) represents the number of targets in the traffic scene at time t.

[0046] The traffic force constraint model is constructed based on the target coupling relationship model and the actual road layout, specifically:

[0047] Traffic force is the combined force between the coupling relationship between targets and the actual road layout on the target. The traffic force on target i at time t is Defined as:

[0048]

[0049] in, It is the coupling relationship between targets; is the semantic encoding information of the road layout where target i is located at time t; c i is the type of moving target given by the behavior recognition subnetwork, which is used to express the difference in the impact of the same road layout on different types of targets; the mapping E gives the force of the road layout on target i based on the target type and road layout semantic information.

[0050] Based on the traffic force constraint model and the traffic semantic grid coding vector of the road layout, a long-short term memory trajectory prediction network based on traffic force coupling relationship constraints is established, specifically:

[0051] According to the distance and type of targets, the interaction force and influence weight between targets are established through radial kernel function;

[0052] Based on the interaction force and influence weight between targets, the influence of other targets in the traffic scene on the predicted target is obtained;

[0053] According to the type of moving target and the semantic coding information of the road layout given by the top view of the virtual road layout, the force of the road layout on the predicted target is mapped;

[0054] The influence of other traffic targets on the predicted target is combined with the amount of force exerted by the road layout on the predicted target to obtain the traffic force on the predicted target.

[0055] The historical motion state of the predicted target itself is concatenated with all traffic forces and enters the LSTM network for time series modeling.

[0056] Based on trajectory extraction, semantic recognition and predicted motion trajectory, the temporal evolution law of traffic situation at the meso-level is modeled to obtain the scene flow digital twin based on the real target dynamic trajectory flow, specifically:

[0057] The historical trajectory and predicted motion trajectory of the target in the real traffic scene are restored to the virtual entity of the actual traffic operation environment, the time-series evolution law of the traffic situation at the mesoscopic level is modeled, and the evolution process of the three-dimensional traffic situation map is visualized;

[0058] The time series evolution law modeling is specifically as follows: a time series evolution law model is constructed according to the trajectory, speed and traffic force constraint model:

[0059]

[0060] in, and They represent the position and velocity of target i at time t, It can be calculated from the position of the target in two frames and the interval between frames. It is the traffic force constraint model;

[0061] The virtual entity is a three-dimensional model of a road scene generated by importing a highly parameterized simulated road layout top view into a three-dimensional simulation tool.

[0062] A scene flow digital twin system based on dynamic trajectory flow, comprising:

[0063] A first building module, which is used to build a detection and tracking integrated multi-modal fusion perception enhancement network to extract and identify the target semantic trajectory;

[0064] An extraction module, the extraction module is used to extract the road traffic semantics to obtain a highly parameterized virtual road layout top view having a mapping relationship with the real traffic scene;

[0065] An acquisition module, wherein the acquisition module acquires a road layout traffic semantic grid coding vector based on a virtual road layout top view;

[0066] A second construction module, wherein the second construction module constructs a target coupling relationship model based on the influence of other targets in the traffic scene on a certain target;

[0067] A third construction module, wherein the third construction module constructs a traffic force constraint model based on the target coupling relationship model and the actual road layout;

[0068] A trajectory prediction network construction module, wherein the trajectory prediction network construction module establishes a long short-term memory trajectory prediction network based on a traffic force coupling relationship constraint based on a traffic force constraint model and a road layout traffic semantic grid coding vector;

[0069] A prediction module, wherein the prediction module predicts the motion trajectory of the target based on a long short-term memory trajectory prediction network constrained by a traffic force coupling relationship to obtain a predicted motion trajectory;

[0070] The digital twin module models the temporal evolution law of the traffic situation at the meso-level based on trajectory extraction, semantic recognition and predicted motion trajectory, and obtains the scene flow digital twin based on the real target dynamic trajectory flow.

[0071] Compared with the prior art, the present invention has the following beneficial effects:

[0072] The present invention proposes a detection and tracking integrated multimodal fusion perception enhancement network to obtain the historical trajectory of the target in the real traffic scene. It can effectively fuse the convolution output tensors of each modality and extract the characteristics of each dimension of the target in the real traffic scene respectively; realize the accurate extraction and identification of the semantic trajectory of the target; at the same time, construct a long and short-term memory trajectory prediction network based on the traffic force coupling relationship constraint to predict the motion trajectory of the target, and according to the trajectory extraction and semantic identification as well as the predicted motion trajectory, model the temporal evolution law of the traffic situation at the meso-level, obtain the scene flow digital twin based on the real target dynamic trajectory flow, and provide decision support for precise traffic control services. BRIEF DESCRIPTION OF THE DRAWINGS

[0073] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0074] Figure 1 It is a flow chart of the scene flow digital twin method based on dynamic trajectory flow of the present invention;

[0075] Figure 2 It is a structural diagram of the scene flow digital twin method based on dynamic trajectory flow of the present invention;

[0076] Figure 3 This is a structural diagram of the detection and tracking integrated multi-modal fusion perception enhancement network of the present invention;

[0077] Figure 4 This is a network structure diagram of the long and short-term memory trajectory prediction based on the traffic force coupling relationship constraint of the present invention;

[0078] Figure 5 Extracting a network structure diagram for the parametric road layout top view of the present invention;

[0079] Figure 6 This is a structural diagram of the scene flow digital twin system based on dynamic trajectory flow of the present invention. DETAILED DESCRIPTION

[0080] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings here can be arranged and designed in various different configurations.

[0081] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention claimed for protection, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0082] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0083] In the description of the embodiments of the present invention, it should be noted that if the terms "upper", "lower", "horizontal", "inner", etc. indicate an orientation or positional relationship based on the orientation or positional relationship shown in the drawings, or the orientation or positional relationship in which the product of the invention is usually placed when in use, it is only for the convenience of describing the present invention and simplifying the description, and does not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention. In addition, the terms "first", "second", etc. are only used to distinguish the description, and cannot be understood as indicating or implying relative importance.

[0084] In addition, if the term "horizontal" appears, it does not mean that the component must be absolutely horizontal, but can be slightly tilted. For example, "horizontal" only means that its direction is more horizontal than "vertical", which does not mean that the structure must be completely horizontal, but can be slightly tilted.

[0085] In the description of the embodiments of the present invention, it is also necessary to explain that, unless otherwise clearly specified and limited, the terms "set", "install", "connect", and "connect" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal connection of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0086] The present invention is further described in detail below in conjunction with the accompanying drawings:

[0087] See also Figure 1 The present invention discloses a scene flow digital twin method based on dynamic trajectory flow, comprising:

[0088] S101: Build a detection and tracking integrated multi-modal fusion perception enhancement network to extract and identify the target semantic trajectory.

[0089] See also Figure 2 , Figure 3 and Figure 4 , the detection and tracking integrated multimodal fusion perception enhancement network includes a multimodal fusion perception enhancement module and a detection and tracking integrated network; the multimodal fusion perception enhancement module includes a resolution attention enhancement module and a feature fusion enhancement model;

[0090] The resolution attention enhancement module is used to learn the invariant feature representation of different modal information;

[0091] The feature fusion enhancement model defines a feature-related tensor pool, aggregates the output tensors of each modal convolution into the tensor pool for feature fusion, and outputs the fused features as the input of the main network;

[0092] The detection and tracking integrated network includes a main network and three sub-networks; the main network is a 3D parameter sharing convolution main network, which acts as a feature extractor to extract different features and send them to the three sub-networks respectively;

[0093] The three subnetworks are motion reasoning subnetwork, driving behavior recognition subnetwork and occlusion recognition subnetwork. The motion reasoning subnetwork is used to track the trajectory of the object; the driving behavior recognition subnetwork is used to recognize the driving behavior; and the occlusion recognition subnetwork is used to recognize the occluded part of the target.

[0094] A resolution attention enhancement module is constructed in the middle of the convolutional block of the integrated detection and tracking network to extract the spatial attribute features of different modalities and learn the invariant feature expression of different modal information through adaptive weight allocation. In addition, multi-layer attention feature cascading is realized through residual connection to achieve adaptive selection of features at different layers, which can ultimately obtain more accurate contextual information and improve the overall performance of the network.

[0095] A feature fusion enhancement model is constructed based on different modal convolution feature map groups of spatial attention. By defining the feature association tensor pool, the multimodal convolution outputs are gathered in the tensor pool for fusion. The output is used as the input of the corresponding convolution layers of the three subnets to obtain accurate trajectory extraction and identification.

[0096] Since the evaluation results of mainstream tracking models are greatly affected by the detection results, the present invention proposes a multi-modal fusion detection and tracking integrated end-to-end network, which can implicitly detect the target object in the tracker, and can also eliminate the impact of the previous detector bias and error on the tracking network. This network consists of a 3D parameter sharing convolution main network and three sub-networks with different task functions, and the object trajectory is tracked, the driving behavior is identified, and the target occlusion is recognized under the three sub-networks. First, the 3D parameter sharing convolution main network is used as a feature extractor to process the 2D images mapped from the NF frame video and the NF frame radar point cloud respectively; secondly, the features of the six intermediate layers in the network are fused and sent to the three sub-networks respectively.

[0097] Motion inference sub-network: Construct a 3D convolutional neural network with multimodal fusion features as input, and synchronously extract the target features of NF frames and the target motion association between frames layer by layer.

[0098] Driving behavior identification subnetwork: Construct a 3D convolutional neural network with multimodal fusion features as input, explore its mapping relationship with driving behavior layer by layer, and define the multimodal spatiotemporal feature mathematical expression of "normal driving behavior" and "abnormal driving behavior" (swaying, tilting, sideslipping, fast U-turns, large radius turns and sudden braking, etc.); utilize the rich layer-by-layer multimodal convolutional fusion features, combined with the motion trajectory characteristics of the motion subnetwork, and jointly optimize the mapping function, in order to learn a more accurate abnormal driving behavior classification model.

[0099] Occlusion recognition subnetwork: Calculate whether each anchor tube is occluded at any time t. If it is occluded, it means that the target cannot be detected and tracked, that is, it is filtered out in the non-maximum suppression stage. If it is not occluded, it is selected and compared with the true value, and then assigned a true value label to participate in training, thereby improving the tracking accuracy and robustness of the entire network.

[0100] S102, extracting the road traffic semantics to obtain a highly parameterized virtual road layout top view having a mapping relationship with a real traffic scene.

[0101] Based on the coupling relationship between the road topology structure and the running trajectory of traffic participants in the traffic scene, the road layout traffic semantics is highly parameterized;

[0102] Construct a virtual road layout top view extraction cascade network parameterized by real scenes. Based on the highly parameterized road layout traffic semantics, a virtual road layout top view extraction cascade network parameterized by virtual-real hybrid training is constructed.

[0103] Based on the pixel space mapping relationship between the real traffic scene image and the trained virtual road layout top view, the virtual-reality mapping of the road layout is constructed to obtain a highly parameterized virtual road layout top view with a mapping relationship with the real traffic scene.

[0104] Based on the coupling relationship between the road topology structure and the running trajectory of traffic participants in the traffic scene, the road layout traffic semantics is highly parameterized, specifically:

[0105] Based on the topological attributes of the main road starting and ending points and the auxiliary road distance, line shape and position road intersection relationship in the traffic scene, the road layout attributes of the number, width and whether the lanes are one-way, the traffic sign attributes of the lane speed limit and lane line shape, and the pedestrian area attributes of the width of crosswalks and sidewalks that constrain pedestrian behavior, all attributes are assigned a unique ID from a data perspective to realize the road layout traffic semantic parameterization.

[0106] The highly parameterized semantics of road layout traffic is proposed. The coupling relationship between the road topology and the running trajectory of traffic participants in the traffic scene is studied, and the road intersection relationship such as the starting and ending points of the main road and the distance, line shape, and position of the auxiliary road in the traffic scene is defined, which is helpful to improve the flexibility of modeling three-way or four-way intersections; the role and semantic expression of refined road parameters and universal traffic rules in the reasoning of road layout in traffic scenes are studied, and the layout attributes of a single road such as the number of lanes, width, and whether it is one-way traffic, the traffic sign attributes such as lane speed limit value and lane line shape, and the scene elements of pedestrian behavior constraints such as crosswalks, sidewalks and width are defined. A parameter list is established to help clarify the constraints on vehicle driving behavior and trajectory reasoning prediction. By studying the structural characteristics of complex traffic scenes, the role of refined road layout and universal traffic rules in the reasoning of macro traffic scene layout, several traffic attributes are defined. They are divided into four categories: topological attributes of road macro structure, lane-level attributes of refined road layout, pedestrian area attributes and traffic sign attributes that constrain the behavior of traffic participants, and the definition of key attributes in each category is explained by taking real traffic scenes as examples.

[0107] Construct a virtual road layout top view extraction cascade network parameterized by real scenes. Based on the highly parameterized road layout traffic semantics, a virtual road layout top view extraction cascade network parameterized by virtual-real hybrid training is constructed. Specifically:

[0108] See also Figure 5 , collect RGB images of road traffic, and extract the RGB images of road traffic through the semantic segmentation network to obtain the semantic top view of the real road;

[0109] Based on the simulator, a fully annotated simulated road image is sampled to obtain a top view of the simulated road;

[0110] Feature extraction is performed on the real road semantic top view and the simulated road top view respectively, and a virtual-real adversarial loss function is established based on virtual-real hybrid training.

[0111] Iterate the loss function to bridge the gap between the simulated road top view and the real road semantic top view;

[0112] The virtual-real confrontation loss function is:

[0113]

[0114] in, represents the loss function under real data supervision, represents the supervised loss function under simulated data, and the scalar λ r and λ s It represents the importance weights of real data and simulated data.

[0115] A highly parameterized road layout top view extraction network with mixed training combining virtual and real is proposed. The traffic scene is understood through real RGB images, and the road layout scene parameters and its simulated top view are predicted. First, the network is based on a large number of fully annotated simulated road top views and a small number of manually annotated, incompletely annotated and noisy real traffic scene images collected, two sources as input. The existing semantic segmentation network is used to obtain the semantic top view of the real image, and the dataset of corresponding scene attributes is obtained based on the definition of road layout traffic semantic parameters. Secondly, the mapping relationship between the top view and the scene parameters is constructed. Finally, the video data is used as input, and CRF is used to improve the temporal smoothness based on the scene prediction parameter vector.

[0116] Based on the pixel space mapping relationship between the real traffic scene image and the trained virtual road layout top view, the cascade network is extracted to construct the virtual-real mapping of the road layout, and a highly parameterized virtual road layout top view with a mapping relationship with the real traffic scene is obtained, specifically:

[0117] Based on the grid coding algorithm of multi-scale adaptive search, the historical trajectory of the target in the real traffic scene provided by the integrated detection and tracking network is encoded into the top view of the virtual road layout to obtain the virtual coordinate trajectory and the corresponding road layout parameters. At the same time, the obtained virtual coordinate trajectory and the corresponding road layout parameters are integrated to obtain the road layout traffic semantic grid coding vector.

[0118] The target historical trajectory in the real traffic scene provided by the detection and tracking integrated network is the acquired trajectory extraction and semantic recognition.

[0119] S103, based on the virtual road layout top view, obtaining a road layout traffic semantic grid coding vector.

[0120] S104, constructing a target coupling relationship model based on the impact of other targets in the traffic scene on a certain target.

[0121] The influence mechanism of the interaction between positional relationships of different targets is studied, the interaction between different targets is expressed by radial kernel function, and the strength of the interaction between multiple targets is described qualitatively and quantitatively.

[0122] The inter-target forces are established based on the radial kernel function, the inter-target influence weights are established by target type and inter-target distance, and the inter-target relationships are coupled based on the weighted summation of the forces acting on each target, thus constructing a target coupling relationship model.

[0123] The target coupling relationship model is At time t, the impact of other targets in the traffic scene on a target i is:

[0124]

[0125] in, represents the interaction between targets i and j at time t; is a weight vector used to express the differences in interactions between different moving targets; n(t) represents the number of targets in the traffic scene at time t.

[0126] S105, constructing a traffic force constraint model based on the target coupling relationship model and the actual road layout.

[0127] Traffic force is the combined force between the coupling relationship between targets and the actual road layout on the target. The traffic force on target i at time t is Defined as:

[0128]

[0129] in, It is the coupling relationship between targets; is the semantic encoding information of the road layout where target i is located at time t; c i is the type of moving target given by the behavior recognition subnetwork, which is used to express the difference in the impact of the same road layout on different types of targets; the mapping E gives the force of the road layout on target i based on the target type and road layout semantic information.

[0130] S106, based on the traffic force constraint model and the road layout traffic semantic grid coding vector, a long short-term memory trajectory prediction network based on traffic force coupling relationship constraints is established.

[0131] According to the distance and type of targets, the interaction force and influence weight between targets are established through radial kernel function;

[0132] Based on the interaction force and influence weight between targets, the influence of other targets in the traffic scene on the predicted target is obtained;

[0133] According to the type of moving target and the semantic coding information of the road layout given by the top view of the virtual road layout, the force of the road layout on the predicted target is mapped;

[0134] The influence of other traffic targets on the predicted target is combined with the amount of force exerted by the road layout on the predicted target to obtain the traffic force on the predicted target.

[0135] The historical motion state of the predicted target itself is concatenated with all traffic forces and enters the LSTM network for time series modeling.

[0136] S107, predicting the motion trajectory of the target based on the long short-term memory trajectory prediction network constrained by the traffic force coupling relationship to obtain a predicted motion trajectory;

[0137] S108, based on trajectory extraction, semantic recognition and predicted motion trajectory, models the temporal evolution law of traffic situation at the meso-level, and obtains the scene flow digital twin based on the real target dynamic trajectory flow.

[0138] The historical trajectory and predicted motion trajectory of the target in the real traffic scene are restored to the virtual entity of the actual traffic operation environment, the time-series evolution law of the traffic situation at the mesoscopic level is modeled, and the evolution process of the three-dimensional traffic situation map is visualized;

[0139] The time series evolution law modeling is specifically as follows: a time series evolution law model is constructed according to the trajectory, speed and traffic force constraint model:

[0140]

[0141] in, and They represent the position and velocity of target i at time t, It can be calculated from the position of the target in two frames and the interval between frames. It is the traffic force constraint model;

[0142] The virtual entity is a three-dimensional model of a road scene generated by importing a highly parameterized simulated road layout top view into a three-dimensional simulation tool.

[0143] See also Figure 6 The present invention discloses a scene flow digital twin system based on dynamic trajectory flow, comprising:

[0144] A first building module, which is used to build a detection and tracking integrated multi-modal fusion perception enhancement network to extract and identify the target semantic trajectory;

[0145] An extraction module, the extraction module is used to extract the road traffic semantics to obtain a highly parameterized virtual road layout top view having a mapping relationship with the real traffic scene;

[0146] An acquisition module, wherein the acquisition module acquires a road layout traffic semantic grid coding vector based on a virtual road layout top view;

[0147] A second construction module, wherein the second construction module constructs a target coupling relationship model based on the influence of other targets in the traffic scene on a certain target;

[0148] A third construction module, wherein the third construction module constructs a traffic force constraint model based on the target coupling relationship model and the actual road layout;

[0149] A trajectory prediction network construction module, wherein the trajectory prediction network construction module establishes a long short-term memory trajectory prediction network based on a traffic force coupling relationship constraint based on a traffic force constraint model and a road layout traffic semantic grid coding vector;

[0150] A prediction module, wherein the prediction module predicts the motion trajectory of the target based on a long short-term memory trajectory prediction network constrained by a traffic force coupling relationship to obtain a predicted motion trajectory;

[0151] The digital twin module models the temporal evolution law of the traffic situation at the meso-level based on trajectory extraction, semantic recognition and predicted motion trajectory, and obtains the scene flow digital twin based on the real target dynamic trajectory flow.

[0152] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A scene flow digital twin method based on dynamic trajectory flow, characterized in that: include: Construct a detection and tracking integrated multi-modal fusion perception enhancement network to extract and identify the target semantic trajectory; The road traffic semantics are extracted to obtain a highly parameterized top view of the virtual road layout that has a mapping relationship with the real traffic scene; Based on the virtual road layout top view, a road layout traffic semantic grid coding vector is obtained; Based on the impact of other targets in the traffic scene on a certain target, a target coupling relationship model is constructed; Construct a traffic force constraint model based on the target coupling relationship model and the actual road layout; Based on the traffic force constraint model and the traffic semantic grid coding vector of the road layout, a long and short-term memory trajectory prediction network based on the traffic force coupling relationship constraint is established; Based on the long short-term memory trajectory prediction network constrained by traffic force coupling relations, the target's motion trajectory is predicted to obtain the predicted motion trajectory; Based on trajectory extraction, semantic recognition and predicted motion trajectory, the temporal evolution law of traffic situation at the mesoscopic level is modeled to obtain the scene flow digital twin based on the real target dynamic trajectory flow; The target coupling relationship model is constructed based on the influence of other targets in the traffic scene on a certain target, specifically: The inter-target forces are established based on the radial kernel function, the inter-target influence weights are established by target type and inter-target distance, and the inter-target relationships are coupled based on the weighted summation of the forces acting on each target, thus constructing a target coupling relationship model. The target coupling relationship model is At time t, the impact of other targets in the traffic scene on a target i is: in, represents the interaction between targets i and j at time t; is a weight vector used to express the differences in the interactions between different moving targets; n(t) represents the number of targets in the traffic scene at time t; The traffic force constraint model is constructed based on the target coupling relationship model and the real road layout, specifically: Traffic force is the combined force between the coupling relationship between targets and the actual road layout on the target. The traffic force on target i at time t is Defined as: in, It is the coupling relationship between targets; is the semantic encoding information of the road layout where target i is located at time t; c i is the type of moving target given by the behavior recognition subnetwork, which is used to express the difference in the impact of the same road layout on different types of targets; the mapping E gives the force of the road layout on target i based on the target type and road layout semantic information.

2. The scene flow digital twin method based on dynamic trajectory flow according to claim 1 is characterized in that: The construction of the detection and tracking integrated multi-modal fusion perception enhancement network extracts and identifies the target semantic trajectory, specifically: The detection and tracking integrated multimodal fusion perception enhancement network includes a multimodal fusion perception enhancement module and a detection and tracking integrated network; The multimodal fusion perception enhancement module includes a resolution attention enhancement module and a feature fusion enhancement model; The resolution attention enhancement module is used to learn the invariant feature expression of different modal information; The feature fusion enhancement model defines a feature association tensor pool, aggregates the output tensors of each modal convolution into the tensor pool for feature fusion, and outputs the fused features as the input of the main network; The detection and tracking integrated network includes a main network and three sub-networks; the main network is a 3D parameter sharing convolution main network, and the 3D parameter sharing convolution main network acts as a feature extractor to extract different features and send them to the three sub-networks respectively; The three subnetworks are respectively a motion reasoning subnetwork, a driving behavior recognition subnetwork and an occlusion recognition subnetwork. The motion reasoning subnetwork is used to track the trajectory of the object; the driving behavior recognition subnetwork is used to recognize the driving behavior; and the occlusion recognition subnetwork is used to recognize the occluded part of the target.

3. The scene flow digital twin method based on dynamic trajectory flow according to claim 1 is characterized in that: The extraction of road traffic semantics to obtain a highly parameterized virtual road layout top view having a mapping relationship with the real traffic scene is specifically: Based on the coupling relationship between the road topology structure and the running trajectory of traffic participants in the traffic scene, the road layout traffic semantics is highly parameterized; Construct a virtual road layout top view extraction cascade network parameterized by real scenes. Based on the highly parameterized road layout traffic semantics, a virtual road layout top view extraction cascade network parameterized by virtual-real hybrid training is constructed. Based on the pixel space mapping relationship between the real traffic scene image and the trained virtual road layout top view, the virtual-reality mapping of the road layout is constructed to obtain a highly parameterized virtual road layout top view with a mapping relationship with the real traffic scene.

4. The scene flow digital twin method based on dynamic trajectory flow according to claim 3 is characterized in that: Based on the coupling relationship between the road topology structure and the running trajectory of the traffic participant target in the traffic scene, the road layout traffic semantics is highly parameterized, specifically: Based on the topological attributes of the main road starting and ending points and the auxiliary road distance, line shape and position road intersection relationship in the traffic scene, the road layout attributes of the number, width and whether the lanes are one-way, the traffic sign attributes of the lane speed limit and lane line shape, and the pedestrian area attributes of the width of the crosswalk and sidewalk that constrains pedestrian behavior, all attributes are assigned a unique ID from a data perspective to achieve road layout traffic semantic parameterization; The construction of a virtual road layout top view extraction cascade network parameterized by real scenes is based on the road layout traffic semantics being highly parameterized, and a virtual road layout top view extraction cascade network parameterized by hybrid training combining virtuality and reality is specifically as follows: Collect RGB images of road traffic, and extract them through semantic segmentation network to obtain the semantic top view of the real road; Based on the simulator, a fully annotated simulated road image is sampled to obtain a top view of the simulated road; Feature extraction is performed on the real road semantic top view and the simulated road top view respectively, and a virtual-real adversarial loss function is established based on virtual-real hybrid training. Iterate the loss function to bridge the gap between the simulated road top view and the real road semantic top view; The virtual-real confrontation loss function is: in, represents the loss function under real data supervision, represents the supervised loss function under simulated data, and the scalar λ r and λ s It represents the importance weights of real data and simulated data.

5. The scene flow digital twin method based on dynamic trajectory flow according to claim 4 is characterized in that: The pixel space mapping relationship of the cascade network is extracted based on the real traffic scene image and the trained virtual road layout top view, a virtual-real mapping of the road layout is constructed, and a highly parameterized virtual road layout top view having a mapping relationship with the real traffic scene is obtained. Specifically, based on a grid coding algorithm of multi-scale adaptive search, the historical trajectory of the target in the real traffic scene provided by the integrated detection and tracking network is encoded into the virtual road layout top view to obtain a virtual coordinate trajectory and corresponding road layout parameters, and the obtained virtual coordinate trajectory and the corresponding road layout parameters are integrated to obtain a road layout traffic semantic grid coding vector; the historical trajectory of the target in the real traffic scene provided by the integrated detection and tracking network is the obtained trajectory extraction and semantic recognition.

6. The scene flow digital twin method based on dynamic trajectory flow according to claim 1 is characterized in that: The long-short term memory trajectory prediction network based on the traffic force constraint model and the road layout traffic semantic grid coding vector is established based on the traffic force coupling relationship constraint, specifically: According to the distance and type of targets, the interaction force and influence weight between targets are established through radial kernel function; Based on the interaction force and influence weight between targets, the influence of other targets in the traffic scene on the predicted target is obtained; According to the type of moving target and the semantic coding information of the road layout given by the top view of the virtual road layout, the force of the road layout on the predicted target is mapped; The influence of other traffic targets on the predicted target is combined with the amount of force exerted by the road layout on the predicted target to obtain the traffic force on the predicted target. The historical motion state of the predicted target itself is concatenated with all traffic forces and enters the LSTM network for time series modeling.

7. The scene flow digital twin method based on dynamic trajectory flow according to claim 6 is characterized in that: Based on trajectory extraction, semantic recognition and predicted motion trajectory, the temporal evolution law of traffic situation at the meso-level is modeled to obtain the scene flow digital twin based on the real target dynamic trajectory flow, specifically: The historical trajectory and predicted motion trajectory of the target in the real traffic scene are restored to the virtual entity of the actual traffic operation environment, the time-series evolution law of the traffic situation at the mesoscopic level is modeled, and the evolution process of the three-dimensional traffic situation map is visualized; The time series evolution law modeling is specifically as follows: a time series evolution law model is constructed according to the trajectory, speed and traffic force constraint model: in, and They represent the position and velocity of target i at time t, It can be calculated from the position of the target in two frames and the interval between frames. It is the traffic force constraint model; The virtual entity is a three-dimensional model of a road scene generated by importing a highly parameterized simulated road layout top view into a three-dimensional simulation tool.

8. A scene flow digital twin system based on dynamic trajectory flow, characterized in that: include: A first building module, which is used to build a detection and tracking integrated multi-modal fusion perception enhancement network to extract and identify the target semantic trajectory; An extraction module, the extraction module is used to extract the road traffic semantics to obtain a highly parameterized virtual road layout top view having a mapping relationship with the real traffic scene; An acquisition module, wherein the acquisition module acquires a road layout traffic semantic grid coding vector based on a virtual road layout top view; A second construction module, wherein the second construction module constructs a target coupling relationship model based on the influence of other targets in the traffic scene on a certain target; A third construction module, wherein the third construction module constructs a traffic force constraint model based on the target coupling relationship model and the actual road layout; A trajectory prediction network construction module, wherein the trajectory prediction network construction module establishes a long short-term memory trajectory prediction network based on a traffic force coupling relationship constraint based on a traffic force constraint model and a road layout traffic semantic grid coding vector; A prediction module, wherein the prediction module predicts the motion trajectory of the target based on a long short-term memory trajectory prediction network constrained by a traffic force coupling relationship to obtain a predicted motion trajectory; A digital twin module, which models the temporal evolution of traffic situations at the mesoscopic level based on trajectory extraction, semantic recognition, and predicted motion trajectories, and obtains a scene flow digital twin based on the real target dynamic trajectory flow; The target coupling relationship model is constructed based on the influence of other targets in the traffic scene on a certain target, specifically: The inter-target forces are established based on the radial kernel function, the inter-target influence weights are established by target type and inter-target distance, and the inter-target relationships are coupled based on the weighted summation of the forces acting on each target, thus constructing a target coupling relationship model. The target coupling relationship model is At time t, the impact of other targets in the traffic scene on a target i is: in, represents the interaction between targets i and j at time t; is a weight vector used to express the differences in the interactions between different moving targets; n(t) represents the number of targets in the traffic scene at time t; The traffic force constraint model is constructed based on the target coupling relationship model and the real road layout, specifically: Traffic force is the combined force between the coupling relationship between targets and the actual road layout on the target. The traffic force on target i at time t is Defined as: in, It is the coupling relationship between targets; is the semantic encoding information of the road layout where target i is located at time t; c i is the type of moving target given by the behavior recognition subnetwork, which is used to express the difference in the impact of the same road layout on different types of targets; the mapping E gives the force of the road layout on target i based on the target type and road layout semantic information.

Images