Driving data processing method, prediction method, system, device and medium
Through feature point matching, geo-registration and object detection algorithms, natural driving data is processed and converted into discrete time dynamic graphs, GCN-LSTM model is constructed, and jitter and stability problems in drone data acquisition and processing are solved. Improve the prediction accuracy of the motion state of the vehicle and surrounding vehicles.
Patent Information
- Application Number
- CN202410944901.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-15
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2044-07-15
AI Technical Summary
When collecting and processing natural driving data, there is a large jitter and poor stability, making it difficult to convert continuous time video data into discrete time forms to apply to dynamic graph convolution-long short-term memory network model, resulting in unclear local features and learning The training complexity is high, the prediction results are poorly generalized, and it is difficult to accurately predict the movement status of the vehicles and surrounding vehicles.
Feature point matching, geo-registration and object detection algorithms are used to process video data, convert it into world coordinates, and build a dynamic graph convolution-long short-term memory network model (GCN-LSTM), which is converted into discrete time dynamic graphs through sampling and training data. Reduce learning parameters, reduce model complexity, and improve prediction accuracy.
It improves the stability and utilization of natural driving data, enhances the local structural information of dynamic graph nodes, improves the prediction accuracy of the motion state of vehicles and surrounding vehicles, reduces the complexity of the model, and improves generalization capabilities.
Smart Images

Figure CN118486100B_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of natural vehicle driving, and in particular relates to a method for processing driving data, a method for acquiring training data, a method for predicting driving status, a testing method, and a system, device, and medium thereof. Background Art
[0002] At present, when vehicles are driving naturally on traffic roads, when drones are used to collect natural driving data of vehicles, the speed of adjusting the spatial state is slow and the jitter is obvious, and the stability of the data after processing is not high. The continuous-time video data collected by natural driving is difficult to be applied to the prediction of vehicle motion state through discrete-time dynamic graphs. The neural network model constructed by dynamic graph convolution often has weak local structural information. The above convolution operation requires a large number of parameters to be learned, the amount of data calculation is large, and the generalization of the results is poor. The accuracy of the predicted vehicle state is low, which is difficult to meet the control and driving requirements of the vehicle on the traffic road. At present, it is generally difficult to introduce the driver's control amount to update the predicted motion state of the vehicle, and the accuracy of the prediction of the motion state of the vehicles around the vehicle is low, which is difficult to meet the requirements of actual driving on the road. Summary of the invention
[0003] The present invention provides a driving data processing method, a training data acquisition method, a driving state prediction method, a testing method and a system, device and medium thereof, so as to solve the problems of large jitter and poor stability in the natural driving data collection and processing of a vehicle by a drone; and difficulty in applying the collected continuous-time video data in the form of discrete time to an improved dynamic graph convolution-long short-term memory network model GCN-LSTM model for time series regression of dynamic graph nodes, resulting in unclear local features, high complexity of learning and training calculations resulting in large amount of calculations, poor regression during training, poor generalization of prediction results, low accuracy, difficulty in updating the state of the vehicle according to driver confrontation control, and low accuracy in predicting the motion state of vehicles around the vehicle.
[0004] The present invention provides a method for processing driving data, which comprises: processing collected natural driving video data based on a video stabilization method; georeferencing the video data; obtaining a rotation detection frame of a vehicle's natural driving based on a target detection algorithm; tracking the heading angle of the vehicle's natural driving and the pixel coordinates of the video data; and trajectory smoothing processing of world coordinates converted from pixel coordinates to obtain natural driving data.
[0005] The present invention also provides a method for acquiring training data, the method comprising: acquiring a discrete time dynamic graph arranged in the order of image frames based on the continuous time dynamic graph; ; Dynamic graph based on each discrete time Create a node set based on each vehicle detected in the process, the adjacency relationship between each vehicle and its surrounding vehicles, and the node feature elements of each vehicle. , adjacency matrix and the feature matrix ; Based on dynamic graph The image frames are sampled to obtain the adjacency matrix of vehicle training data in natural driving video data and the feature matrix ; Wherein, t represents the duration of the tth preset cycle, τ represents an integer and satisfies 1≤t≤τ.
[0006] The present invention also provides a driving state prediction method, which comprises: constructing a dynamic graph convolution-long short-term memory network model (GCN-LSTM) based on the node feature element data of each vehicle obtained; training the dynamic graph convolution-long short-term memory network model (GCN-LSTM) based on the node feature element data of each vehicle obtained by applying the driving data processing method and the training data acquisition method until each node feature in the training data is regressed in time series; converting the dynamic graph data continuously collected and processed within a first preset time length into a dynamic graph of discrete time. , the discrete-time dynamic graph The input is given to the trained dynamic graph convolution-long short-term memory network model (GCN-LSTM); based on the trained dynamic graph convolution-long short-term memory network model (GCN-LSTM), the status of the node feature elements of all current vehicles is predicted.
[0007] The present invention also provides a driving state simulation test method, which applies the above-mentioned driving state prediction method. The driving state simulation test method includes: simulating the movement of all current vehicles based on the simulator according to the predicted state results of the node characteristic elements of all current vehicles in the simulation test; and simulating the movement state of vehicles around the own vehicle based on the movement state of the own vehicle.
[0008] The present invention also provides a driving state testing method, which applies the above-mentioned driving state simulation testing method and introduces the driver's confrontation control. The driving state testing method includes: in the simulation test, obtaining the lateral and / or longitudinal control data of the driver controlling the own vehicle; updating the motion state data predicted by the dynamic graph convolution-long short-term memory network model (GCN-LSTM) for the own vehicle based on the actual driving state data recorded during the natural driving of the own vehicle within a first preset time; updating the discrete time dynamic graph based on the actual driving state data updated by the own vehicle ; Update-based discrete-time dynamic graph The motion state of the own vehicle within a second preset time period is predicted.
[0009] The present invention also provides an electronic device, which includes: a memory for storing non-temporary computer-readable instructions; and a processor for running the computer-readable instructions, so that when the computer-readable instructions are executed by the processor, the above-mentioned driving status testing method is implemented.
[0010] The present invention also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a device, the device executes the above-mentioned driving status testing method.
[0011] Compared with the prior art, the present invention has obvious advantages and beneficial effects. By means of the above technical solution, the present invention has at least one of the following advantages and beneficial effects:
[0012] 1. The present invention provides a method for processing driving data, which includes: processing the collected natural driving video data based on feature point matching; georeferencing the processed video data; obtaining the rotation detection frame of the vehicle's natural driving based on the target detection algorithm; tracking the heading angle of the vehicle's natural driving and the pixel coordinates of the video data; and trajectory smoothing the world coordinates converted from the pixel coordinates to obtain natural driving data. The present invention collects and processes natural driving data through a drone, and extracts data information for training a GCN-LSTM model. When sampling, the spatial state is adjusted quickly and conveniently by controlling the drone, the data jitter is small, and the stability of the processed data is high, thereby improving the utilization rate and accuracy of natural driving data.
[0013] Second, the present invention also provides a method for obtaining training data, the method comprising: obtaining a discrete time dynamic graph arranged in the order of image frames based on the continuous time dynamic graph; ; Dynamic graph based on each discrete time Create a node set based on each vehicle detected in the process, the adjacency relationship between each vehicle and its surrounding vehicles, and the node feature elements of each vehicle. , adjacency matrix and the feature matrix ; Set each vehicle detected in natural driving data as a node, and the vehicles detected and identified in the same frame as a node set ; Safety requirements and test decision node sets based on road scenarios The adjacency relationship between each vehicle and its surrounding vehicles, and based on the node set Create an adjacency matrix based on the adjacency relationship of all vehicles in ; Create a feature matrix based on the node feature elements of the vehicle , the node set The node features of each vehicle in the feature matrix A row of; based on the dynamic graph The image frames are sampled to obtain the adjacency matrix of vehicle training data in natural driving video data and the feature matrix ; Wherein, the node feature elements of each vehicle include the current lane ID, vehicle speed, vehicle acceleration, vehicle heading angle and bias; wherein t represents the duration of the tth preset cycle, τ represents an integer and satisfies 1≤t≤τ. The present invention collects and processes natural driving data and extracts data information for training the GCN-LSTM model, and converts the data information into a discrete-time dynamic graph for neural network training of the dynamic graph. The present invention realizes the application of the dynamic graph (static graph) converted from the continuous time domain to the discrete-time dynamic graph in the GCN-LSTM model for time series regression of the dynamic graph nodes, enhances the local structural information of the dynamic graph nodes before and after the current moment, improves the accuracy of the vehicle state prediction based on the discrete-time dynamic graph, and the accuracy of predicting the motion state of vehicles around the vehicle.
[0014] 3. The present invention also provides a method for predicting a driving state, which includes: constructing a dynamic graph convolution-long short-term memory network model (GCN-LSTM) based on the node feature element data of each vehicle obtained; training the dynamic graph convolution-long short-term memory network model (GCN-LSTM) using the node feature element data of each vehicle obtained by the driving data processing method and the training data acquisition method until each node feature in the training data is time-series regressed; converting the dynamic graph data continuously collected and processed within a first preset time length into a dynamic graph of discrete time , the discrete-time dynamic graph Input to the trained dynamic graph convolution-long short-term memory network model (GCN-LSTM); based on the trained dynamic graph convolution-long short-term memory network model (GCN-LSTM), the state of the node feature elements of all current vehicles is predicted. The present invention converts the continuous time domain into a discrete time dynamic graph (static graph) and applies it to the GCN-LSTM model to perform temporal regression of the dynamic graph nodes, thereby enhancing the local structural information of the dynamic graph nodes before and after the current moment, reducing the parameters required for learning through the convolution operation of shared weights, reducing the complexity of its model, improving the generalization ability of feature representation learning based on dynamic graph nodes in the discrete time domain, and improving the accuracy of vehicle state prediction based on discrete time dynamic graphs.
[0015] Fourth, the present invention also provides a driving state simulation test method, which applies the above-mentioned driving state prediction method, and the driving state simulation test method includes: simulating the movement of all current vehicles based on the state results of the node feature elements of all current vehicles predicted by the simulator in the simulation test; simulating the movement state of vehicles around the own vehicle based on the movement state of the own vehicle. The present invention predicts and simulates the movement state of surrounding vehicles by using the GCN-LSTM neural network of the node time series regression of the dynamic graph, and reduces the parameters required for learning through the convolution operation of shared weights, reduces the complexity of its model, improves the generalization ability of feature representation learning based on dynamic graph nodes in discrete time domain, and improves the accuracy of vehicle state prediction based on discrete time dynamic graph.
[0016] 5. The present invention also provides a driving state testing method, which applies the above-mentioned driving state simulation testing method and introduces the driver's confrontation control. The driving state testing method includes: in the simulation test, obtaining the lateral and / or longitudinal control data of the driver controlling the own vehicle; updating the motion state data predicted by the dynamic graph convolution-long short-term memory network model (GCN-LSTM) for the own vehicle based on the actual driving state data recorded during the natural driving of the own vehicle within a first preset time; updating the discrete time dynamic graph based on the actual driving state data updated by the own vehicle ; Update-based discrete-time dynamic graph Predicting the motion state of the own vehicle within a second preset time period. The present invention further improves the accuracy of predicting the motion state of vehicles around the own vehicle by updating the own vehicle state based on driver control within a preset cycle time period.
[0017] Sixth, the present invention also provides an electronic device, which includes: a memory for storing non-temporary computer-readable instructions; and a processor for running computer-readable instructions, so that the computer-readable instructions are executed by the processor to implement the above-mentioned driving state test method. The electronic device of the present invention adjusts the spatial state quickly and conveniently during sampling, with small data jitter and high stability; realizes the application of the dynamic graph (static graph) converted from the continuous time domain to the discrete time to the GCN-LSTM model for the time series regression of the dynamic graph nodes, enhances the local structural information of the dynamic graph nodes before and after the current moment, reduces the parameters required for learning through the convolution operation of shared weights, reduces the complexity of its model, improves the generalization ability of feature representation learning based on dynamic graph nodes in the discrete time domain, and improves the accuracy of vehicle state prediction based on discrete time dynamic graphs; by updating the vehicle state based on driver control within a preset cycle duration, the accuracy of predicting the motion state of vehicles around the vehicle is improved.
[0018] VII. The present invention also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a device, the device executes the driving state test method described above. The computer-readable storage medium of the present invention is quick and convenient to adjust the spatial state during sampling, with small data jitter and high stability; it realizes the application of the dynamic graph (static graph) converted from the continuous time domain to the discrete time to the GCN-LSTM model for the time series regression of the dynamic graph nodes, enhances the local structural information of the dynamic graph nodes before and after the current moment, reduces the parameters required for learning through the convolution operation of shared weights, reduces the complexity of its model, improves the generalization ability of feature representation learning based on the dynamic graph nodes in the discrete time domain, and improves the accuracy of the vehicle state prediction based on the discrete time dynamic graph; by updating the vehicle state based on the driver control within the preset cycle duration, the accuracy of predicting the motion state of the vehicles around the vehicle is improved.
[0019] The above description is only an overview of the technical solution of the present invention. In order to more clearly understand the technical means of the present invention, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present invention more obvious and easy to understand, the following specifically cites a preferred embodiment and describes it in detail with the accompanying drawings as follows. BRIEF DESCRIPTION OF THE DRAWINGS
[0020] Figure 1 is a flowchart of a method for processing driving data according to an embodiment of the present invention;
[0021] Figure 2 is a schematic structural diagram of a driving data processing device according to an embodiment of the present invention;
[0022] Figure 3 is a flow chart of a method for acquiring training data according to an embodiment of the present invention;
[0023] Figure 4 is a structural schematic diagram of a training data acquisition device according to an embodiment of the present invention;
[0024] Figure 5 is a flow chart of a method for predicting a driving state according to an embodiment of the present invention;
[0025] Figure 6 is a schematic structural diagram of a driving state prediction system according to an embodiment of the present invention;
[0026] Figure 7 It is a flowchart of a driving state simulation test method according to an embodiment of the present invention;
[0027] Figure 8 It is a structural schematic diagram of a driving state simulation test device according to an embodiment of the present invention;
[0028] Fig. 9 It is a flowchart of a driving state testing method according to an embodiment of the present invention;
[0029] Fig.10 It is a structural schematic diagram of a driving state testing system according to an embodiment of the present invention;
[0030] Fig.11 It is a schematic structural diagram of an electronic device according to an embodiment of the present invention.
[0031] Fig.12 The present invention is a schematic diagram of a road traffic identification and detection system based on ISO 34502 according to an embodiment of the present invention.
[0032] Fig.13 It is a schematic diagram of a discrete-time dynamic graph obtained by converting natural driving data according to an embodiment of the present invention.
[0033] Fig.14 It is a structural diagram of a GCN-LSTM model according to an embodiment of the present invention.
[0034] Fig.15 It is a structural diagram of LSTM model calculation according to one embodiment of the present invention;
[0035] Fig.16 The present invention is a flowchart of a lateral and longitudinal control process based on driver control of a vehicle according to an embodiment of the present invention. DETAILED DESCRIPTION
[0036] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects proposed according to the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments.
[0037] The present invention provides a method for processing driving data, such as Figure 1 As shown, the method for processing the driving data includes:
[0038] S1, processing the collected natural driving video data based on feature point matching.
[0039] It should be noted that in S1, the drone will be affected by air flow and will jitter during the video collection process. The purpose of S1 is to eliminate the impact of jitter when the drone collects video data, and it is achieved by combining the feature point matching algorithm with the random sample consensus algorithm (RANSAC) and homography transformation. Among them, the feature point matching algorithm is not limited to one and / or a combination of multiple SIFT, SURF, and ORB. The feature point matching algorithm generally includes the following three main steps: such as feature point detection, feature descriptor extraction, and matching of feature points with feature descriptors. In the present invention, the feature point matching algorithm uses the first frame of the video data as the reference frame, and the feature points of all images are matched with the first frame. In the present invention, the RANSAC algorithm is used to eliminate the mismatch between feature points and feature descriptors, and then the points of the image plane are mapped to the reference frame plane through homography transformation, that is, the points of the image plane are mapped to the plane of the first frame.
[0040] S2, video data after georeferencing.
[0041] It should be noted that the present invention converts the pixel coordinates into world coordinates through the video data processed by georeferencing. In an embodiment, multiple points such as four reference points selected on the plane of the first frame can be used, or multiple reference points can be selected on the plane of the first frame. For example, the method for obtaining the world coordinates of the four reference points in the above embodiment can adopt the following steps: first obtain the longitude and latitude of the four points from the satellite map, and then convert them into world coordinates through the WGS84 coordinate system.
[0042] S3, based on the target detection algorithm, obtains the rotation detection frame of the natural driving of the vehicle; tracks the heading angle of the natural driving of the vehicle and the pixel coordinates of the video data.
[0043] It should be noted that the target detection algorithm uses a rotating target detection algorithm to detect vehicles, for example, single-stage rotating target detection (R3Det) can be selected. The rotating detection frame obtained during the R3Det detection can be used as the length and width of the vehicle and applied to the calculation of the heading angle. In addition, the tracking algorithm is not limited to using a multi-target tracking algorithm based on target detection (DeepSort). In addition to the target detection of the vehicle, it is also necessary to detect the target of the lane line, determine the lane through the lane line, and assign an ID to the determined lane, so as to detect and track the vehicle and lane in the collected natural driving video data.
[0044] S4, trajectory smoothing processes the world coordinates converted from pixel coordinates to obtain natural driving data.
[0045] It should be noted that the pixel coordinates output by the above tracking algorithm need to be converted into world coordinates before smoothing, and the coordinates of its unit pixel (pixel) need to be converted into pixels (m). In the embodiment of the present invention, by selecting trajectory smoothing processing, not limited to the use of adaptive traceless Kalman smoothing (RTS Smoother), the tracking algorithm calculates all trajectory points in the natural driving video data, and uses a bidirectional filtering method for the tracked trajectory curve to obtain the speed, acceleration and coordinate data that conforms to the continuous motion changes of the vehicle target.
[0046] In the above embodiment of the present invention, after completing the above steps S1 to S4, the natural driving data of the vehicle target can be obtained, and its format is not limited to being adjusted to: vehicle ID, current lane ID, vehicle speed (m / s), vehicle acceleration (m / s^2), heading angle (°), offset (m), surrounding vehicle ID. For example, when the vehicle ID of the target vehicle target feature description is 3, the above natural driving data with the vehicle ID of 3 can be adjusted to: 3, 2, 5, 0.5, 3, 0.3, [2, 6, 0, 4, 0, 1, 0, 5] and other forms, or other array forms that are recorded, stored or calculated respectively, with reference to the prior art, which will not be described in detail one by one here.
[0047] In the above-mentioned embodiment of the present invention, as an optional implementation manner, before processing the collected natural driving video data, the method for processing the driving data also includes: using a fixed-state drone on a road with heavy traffic to collect video data of vehicles and registration points on the road.
[0048] It should be noted that the fixed state drone collects the video data of vehicles and registration point targets on the road (such as landmark buildings, landmark road signs, etc.). In the embodiments of the present invention, the drone collects the video data of vehicles and registration point targets on the road, and obtains more comprehensive video data of the target area through high-resolution image acquisition sensors, intelligent flight control systems, and multi-sensor data fusion. For example, infrared remote sensing detection is not limited to the data fusion method of visible light sensors and infrared sensors to identify vehicle targets, which improves the accuracy of the vehicle's natural driving video data.
[0049] In the above-mentioned embodiment of the present invention, as an optional implementation mode, the processing of the collected natural driving video data based on the video stabilization method includes: based on the feature point matching algorithm, taking the first frame of the video data as the reference frame, the feature points of all images are matched with the first frame, and the feature points in the video data are detected; based on the random sample consensus algorithm (RANSAC), the wrong matching of feature points is eliminated; based on the homography transformation, the feature points of all images are mapped to the reference frame plane.
[0050] It should be noted that four reliable reference points are selected on the first base frame, and the RANSAC algorithm is used to obtain the homography matrix matching the first base frame. The feature points of all other images are mapped to the reference frame plane through the homography matrix corresponding to the first base frame. When mapping, it is not limited to image correction, perspective transformation, image stitching, augmented reality (AR), camera pose estimation, visual SLAM, etc. for all the other images mentioned above. Subsequently, it is not limited to selection through RANSAC, or unconstrained or constrained nonlinear optimization of the obtained vehicle targets and traffic roads, and landmark reference points through the nonlinear least squares optimization algorithm (LM).
[0051] In the above embodiment of the present invention, as an optional implementation, the video data after geo-reference processing includes: based on the satellite map, selecting n reference points on the basic frame of the mapping feature points, obtaining the longitude and latitude of the n reference points on the basic frame and their converted world coordinates; based on the pixel coordinates and world coordinates of the selected n reference points, and the state parameters of the camera, obtaining the camera's extrinsic parameter matrix k1 based on formula (1);
[0052] (1)
[0053] The pixel coordinates of the base frame are converted to world coordinates based on formula (1); where (X w , Y w , Z w ) is the world coordinate of the reference point on the base frame; (u, v) is the pixel coordinate of the reference point; k1 is the external parameter matrix of the camera; k2 is the internal parameter matrix of the camera; the state parameters of the camera include Z c , Z c is the vertical height from the optical center of the drone camera to the ground; n is a positive integer not less than 4.
[0054] It should be noted that when performing the above-mentioned geo-registration, n reference points are selected on the basic frame of the mapping feature points based on the satellite map, and the value of n is an integer not less than 4, so that when the homography matrix of the vehicle target obtained based on the first frame of the basic frame is used for image correction, the above-mentioned conversion of the pixel coordinates of the basic frame into world coordinates can be achieved through the minimum required four corresponding point pairs.
[0055] In the above embodiment of the present invention, as an optional implementation, checkerboard calibration is performed on the base frame based on the selected reference points to obtain the pixel focal length of the main reference point on the base frame, and the intrinsic parameter matrix k2 of the camera is obtained according to formula (2).
[0056] (2)
[0057] Among them, f xis the length of the pixel focal length in the x-axis direction; f y is the length of the pixel focal length in the y-axis direction; (u0, v0) is the actual pixel coordinate of the main reference point.
[0058] It should be noted that the internal parameter matrix k2 of the above camera can be adjusted based on the distance between the drone and the shooting target, and can also be adjusted based on the ultra-high definition of the captured video. x 、f y The two coordinate axis deflection parameters u0, u0 and v0 caused by manufacturing errors are usually small, and are used to represent the intrinsic parameters inherent to the four cameras when the camera's internal parameter characteristics are captured in a fixed state video.
[0059] In the above embodiment of the present invention, as an optional implementation, the driving data processing method further includes selecting n reference points to obtain a rotation matrix and a translation matrix of the camera's extrinsic matrix according to formula (3);
[0060] (3)
[0061] Among them, R represents the rotation matrix and T represents the translation matrix.
[0062] It should be noted that the n reference points selected above can represent the minimum 4 reference points of a plane. In the camera's extrinsic parameter matrix k1, the camera's extrinsic parameters R and T can be solved using the P4P method, which requires selecting four reference points on the base frame and giving the world coordinates of the four reference points. In addition, it is not limited to selecting more than four reference points on the base frame, and the camera's extrinsic parameter matrix k1 is solved using the PnP method, where n represents an integer greater than 4.
[0063] In the above-mentioned embodiment of the present invention, as an optional implementation mode, the target detection algorithm selects a rotating target detection algorithm, and uses the rotation detection frame obtained when the vehicle is driven naturally as the length and width of the vehicle.
[0064] It should be noted that in the present invention, the rotation detection frame of the vehicle's natural driving is obtained as the length and width of the vehicle, and the longitude and latitude of these four points can be obtained on the satellite map to convert them into their world coordinates.
[0065] In the above-mentioned embodiment of the present invention, as an optional implementation, the pixel coordinates of the heading angle and video data of the natural driving of the tracking vehicle and / or drone camera include: video data of the natural driving of the tracking vehicle based on a tracking algorithm.
[0066] It should be noted that the above method for tracking the heading angle of the vehicle's natural driving can be based on the camera calibration of the vanishing point estimation, the vehicle positioning data preprocessing, the road global direction angle calculation, and the drone heading angle calculation. At the same time, the road direction is indirectly inferred with the help of the vehicle GNSS positioning information, and the camera global heading angle is calculated by combining the calibrated horizontal field of view angle and the first clearing point position. Among them, the calibration of the camera's internal and external parameters can be completed by the conversion of the camera coordinate system, the conversion of the pixel coordinate system, and the conversion of the pinhole imaging to reduce the distortion of the camera when shooting video. The method based on the vanishing point vehicle calibration obtains the first vanishing point position and the camera's internal and external parameters in the scene; the vehicle GNSS positioning data is processed, and the camera's global heading angle is calculated by combining the pixel position of the first vanishing point, the image center position, and the calibrated camera horizontal field of view angle. In addition, the vehicle's natural driving heading angle can be calculated based on the vehicle's driving direction, speed, acceleration, and lane data corresponding to the lane ID to track the vehicle's natural driving heading angle. The pixel coordinates of the video data can be obtained based on the heading angle of the above-mentioned vehicle and / or drone and the coordinate position of the captured video data.
[0067] In the above-mentioned embodiment of the present invention, as an optional implementation, the method for processing driving data also includes: acquiring lane lines detected from natural driving video data based on a target detection algorithm, and assigning ID data to lanes based on safety requirements and tests of road scenes.
[0068] It should be noted that the above target detection algorithm is not limited to using Hough transform to detect straight or curved lane lines when detecting lane lines from natural driving video data. The color threshold method is used based on the color features of the lane lines, and the RGB or HSV color space is usually used to quickly detect and identify the lane lines. The above two methods are combined with convolutional neural networks (CNNs) to perform end-to-end lane detection by using the deep learning model of convolutional neural networks (CNNs). The features of the shape, color and chromaticity of the lane lines in each image frame can be automatically learned for end-to-end lane recognition. In addition, a sliding window can be used in the image to search for the position of the lane line so as to quickly detect the lane line features in the picture. In order to ensure the correctness of the detected lanes and eliminate the errors in lane line detection, the multi-lane features are fitted to exclude the lane lines with large differences in detection features. In an embodiment of the present invention, a tracking algorithm is used to track the position of the lane line between consecutive frames to reduce the amount of repeated calculations, and combined with the data of the laser radar and the camera, the accuracy and robustness of the lane line detection are improved, and the errors in the lane line detection are reduced. For example, it is also possible to find the geometric structure of the road in the image or video, determine the position of the vehicle on the road, and support the driver assistance system. Lane detection usually uses image processing techniques such as edge detection, color segmentation, Hough transform, etc. to detect the white or yellow lane lines on the road. For example, in an embodiment of the present invention, Fig.12 As shown, along the driving direction of the 5 lanes in the figure, the test vehicle is located at the center of all the above vehicles. The test vehicle is the position of the vehicles around the ego vehicle. Positions 1 to 8 are the observable areas of the test vehicle, and positions 9 to 24 of the vehicles around the ego vehicle in the above lanes are the blind spots of the test vehicle. It should also be noted that the vehicle ID cannot be 0, it starts from 1; the current lane ID is the lane ID where the center of the detection frame is located; the heading angle can be estimated by the angle between the long side of the detection frame output by the rotating target detection algorithm and the lane line on the right side of the vehicle; the bias represents the offset of the center point of the vehicle detection frame relative to the center line of the lane, which is positive downward and negative upward (the rules are unified); the surrounding vehicle ID is an array containing 8 values, and the 8 positions considered are referenced Fig.12 If there is no vehicle at a certain position, fill it with 0.
[0069] The present invention also provides a driving data processing device, such as Figure 2As shown, the driving data processing device 900 includes: a stability processing unit 910, which is used to process the collected natural driving video data based on a feature point matching method; a geographic registration unit 920, which is used to geo-register the processed video data; a detection and tracking unit 930, which is used to obtain a rotation detection frame of the vehicle's natural driving based on a target detection algorithm; track the heading angle of the vehicle's natural driving and the pixel coordinates of the video data; a trajectory smoothing unit 940, which is used to smooth the trajectory of the world coordinates converted from the pixel coordinates to obtain natural driving data.
[0070] It should be noted that the driving data processing device 900 of the present invention includes a stability processing unit 910, a geographic registration unit 920, a detection and tracking unit 930 and a trajectory smoothing unit 940. When running, it can collect and process natural driving data through a drone, and extract data information for training the GCN-LSTM model. When sampling, it is quick and convenient to adjust the spatial state by controlling the drone, the data jitter is small, and the stability of the processed data is high, thereby improving the utilization rate and accuracy of natural driving data.
[0071] The present invention also provides a method for obtaining training data, such as Figure 3 As shown, the method for obtaining training data includes: S5, based on the continuous time dynamic graph, obtaining the discrete time dynamic graph arranged in the order of image frames ; S6, dynamic graph based on each discrete time Create a node set based on each vehicle detected in the process, the adjacency relationship between each vehicle and its surrounding vehicles, and the node feature elements of each vehicle. , adjacency matrix and the feature matrix ; S7, based on dynamic graph The image frames are sampled to obtain the adjacency matrix of vehicle training data in natural driving video data and the feature matrix ; Wherein, t represents the duration of the tth preset cycle, τ represents an integer and satisfies 1≤t≤τ.
[0072] It should be noted that a discrete-time dynamic graph is a sequence of graph snapshots. , where each graph Contains a collection of nodes , adjacency matrix and the feature matrix In an embodiment of the present invention, the discrete time dynamic graph can be converted from the natural driving data obtained in step 2 (eg Fig.13 ). Node collection Contains all vehicles in the same frame, each vehicle is a node. Adjacency matrix Indicates whether the vehicle has an adjacent relationship with surrounding vehicles (refer to Fig.12 Positions 1-8 in ). Feature matrix Each row of represents a node feature of a vehicle, wherein the elements are the current lane ID, vehicle speed, vehicle acceleration, heading angle and offset. and The data in can use natural driving data belonging to continuous frames, or the sampling step s can be set (for example, s = 10 frames). Fig.13 As shown, in the three lanes, vehicles 3 and 4 are located in the top lane, vehicle 1 is located in the middle lane, and vehicle 2 is located in the bottom lane. Vehicle 2 accelerates from the bottom lane to the front position of vehicle 1 in the middle lane. The above three node sets , adjacency matrix and the feature matrix The calculation is as follows Fig.13 , based on the dynamic graph The present invention collects and processes natural driving data and extracts data information for training the GCN-LSTM model, and converts the data information into a discrete-time dynamic graph for neural network training of the dynamic graph. The present invention realizes the conversion of the continuous time domain into a discrete-time dynamic graph (static graph) and applies it to the GCN-LSTM model for time series regression of the dynamic graph nodes, enhances the local structural information of the dynamic graph nodes before and after the current moment, improves the accuracy of vehicle state prediction based on discrete-time dynamic graphs, and the accuracy of predicting the motion state of vehicles around the vehicle.
[0073] In the above embodiment of the present invention, as an optional implementation, the node set is created Including: setting each vehicle detected in natural driving data as a node, and setting the vehicles detected and identified in the same frame as a node set .
[0074] It should be noted that if Fig.13 As shown in the figure, each vehicle detected by natural driving data is set as a node, such as vehicle position numbers 1 to 24, and the number position of the test vehicle (located at the center of the area), and the vehicles detected and identified in the same frame are set as a node set. For example, vehicle 1 is adjacent to vehicles 2, 3, and 4 respectively. When vehicle 2 enters the middle lane from the lowest lane, vehicle 2 is only adjacent to vehicle 1. After vehicle 2 enters the middle lane, any of vehicles 1, 2, 3, and 4 are adjacent to each other.
[0075] In the above embodiment of the present invention, as an optional implementation, the adjacency matrix is created Includes: Safety requirements based on road scenarios and test decision node sets The adjacency relationship between each vehicle and its surrounding vehicles, and based on the node set Create an adjacency matrix based on the adjacency relationship of all vehicles in .
[0076] It should be noted that the safety requirements and test decision node sets based on road scenarios The value of t is 3, and its node v (1) ={v1, v2, v3, v4} Based on different time periods, the positional relationships between vehicles and their lanes are different, such as v (2) ={v1,v2,v3,v4},v (3) ={v1, v2, v3, v4} have different positional relationships. Thus, the corresponding adjacency matrix A is obtained (1) To A (3) It is expressed as follows:
[0077] .
[0078] In the above embodiment of the present invention, as an optional implementation, the feature matrix is created Includes: Creating a feature matrix based on the vehicle's node feature elements , the node set The node features of each vehicle in the feature matrix A row of; wherein the node feature elements of each vehicle include the current lane ID, vehicle speed, vehicle acceleration, vehicle heading angle and offset.
[0079] It should be noted that the node feature elements of each vehicle include the current lane ID, vehicle speed, vehicle acceleration, vehicle heading angle and offset data such as Fig.13 Medium X (1) To X (3) As shown, for example, X (1) The current lane ID, vehicle speed, vehicle acceleration, vehicle heading angle, and offset data of vehicle 1 are 1, 4.0, 0.2, 2, and 0.1 respectively; the current lane ID, vehicle speed, vehicle acceleration, vehicle heading angle, and offset data of vehicle 2 are 2, 5.0, 1, 3, and 0.2 respectively; vehicle 3..., and so on, and X (1) To X (2) , and X (2) To X (3) The feature matrix of each vehicle like Fig.13 As shown. The present invention uses the characteristic matrix of the vehicle Accuracy of vehicle status detection based on ISO 34502 road traffic recognition.
[0080] In the above embodiment of the present invention, as an optional implementation, Fig.13 As shown, the dynamic graph based on The image frames are sampled by: sampling based on the natural driving video data of continuous frames with a preset time period of adjacent frames; for example, X (1) and X (3) And X (2) If the sample is not sampled, it will be sampled at the preset time period. This sampling can reduce the sample size, highlight the sample features, facilitate fast sampling, and accelerate the learning and training of the model.
[0081] In an embodiment of the present invention, sampling may also be performed based on a preset number of frames of natural driving video data of consecutive frames. Fig.13 As shown, X can be used during sampling (1) To X (3) Sampling is performed, and X (1) and X (2) , and X (2) and X (3) That is, sampling is performed only at a preset number of consecutive frames, so that sampling can ensure that the vehicle 2 in a consecutive number of frames is not lost when it enters the middle lane from the lowest lane. (2) and X (2) For details with large feature changes in consecutive frames, the amount of data obtained by sampling is slightly larger, but the structural features, especially the local area features and edge structure features are more significant.
[0082] In an embodiment of the present invention, the feature matrix of the natural driving video data of the current continuous frame can also be used. Sampling is performed to see whether the data of the previous several frames and the next several frames have changed. For example, the corresponding A (1) To A (3) Corresponding X (1) and X (2) , and X (2) and X (3) A between (t) and X (t) When any value changes, sampling is performed once to obtain the node feature changes of each vehicle or each vehicle based on the lane to the maximum extent.
[0083] In the above embodiment of the present invention, as an optional implementation, the feature matrix of the natural driving video data based on the current continuous frame is Sampling whether the data changes between the previous several frames and the next several frames includes: if the feature matrix If there is no change within the preset time period, sampling is performed at the preset time period; otherwise, the feature matrix When data of several previous frames and several subsequent frames change, the previous frame before the change and the subsequent frame after the change are sampled.
[0084] It should be noted that such an approach can prevent sampling of vehicles whose states have not changed within a preset time period. For example, if any parameter of a moving convoy changes, such as when all vehicles stop, multiple sampling is not performed. When the preset time period ends, sampling is performed for the preset time period. Such an approach can reduce the number of samples and maximize the sampling of all structural features of each node in the video data when it changes, ensuring the generalization of the structural features of the video data used for learning and training.
[0085] The present invention also provides a device for acquiring training data, such as Figure 4 As shown, the training data acquisition device 800 includes: a dynamic image generation unit 810, which is used to acquire a discrete time dynamic image arranged in the order of image frames based on the continuous time dynamic image. ; Creating unit 820, for dynamic graph based on each discrete time Create a node set based on each vehicle detected in the process, the adjacency relationship between each vehicle and its surrounding vehicles, and the node feature elements of each vehicle. , adjacency matrix and the feature matrix ; Sampling unit 830, for dynamic graph based on The image frames are sampled to obtain the adjacency matrix of vehicle training data in natural driving video data and the feature matrix ; Wherein, t represents the duration of the tth preset cycle, τ represents an integer and satisfies 1≤t≤τ.
[0086] It should be noted that the training data acquisition device 800 of the present invention collects and processes natural driving data and extracts data information for training the GCN-LSTM model, and converts the data information into a discrete-time dynamic graph for neural network training of the dynamic graph. The present invention realizes the application of the dynamic graph (static graph) converted from the continuous time domain to the discrete-time dynamic graph to the GCN-LSTM model for time series regression of the dynamic graph nodes, enhances the local structural information of the dynamic graph nodes before and after the current moment, improves the accuracy of vehicle state prediction based on the discrete-time dynamic graph, and the accuracy of predicting the motion state of vehicles around the vehicle.
[0087] The present invention also provides a method for predicting a driving state. Figure 5 As shown, the driving state prediction method includes: S8, a dynamic graph convolution-long short-term memory network model (GCN-LSTM) constructed by acquiring node feature element data pairs of each vehicle.
[0088] S9, applying the driving data processing method and the training data acquisition method to obtain the node feature element data of each vehicle to train the dynamic graph convolution-long short-term memory network model (GCN-LSTM) until each node feature in the training data is regressed in time series. S10, converting the dynamic graph data continuously collected and processed within the first preset time length into a dynamic graph of discrete time , the discrete-time dynamic graph The input is given to the trained dynamic graph convolution-long short-term memory network model (GCN-LSTM); based on the trained dynamic graph convolution-long short-term memory network model (GCN-LSTM), the status of the node feature elements of all current vehicles is predicted.
[0089] It should be noted that the present invention Apply dynamic graph convolutional neural network (GCN) to obtain implicit representation matrix . For the i-th node , get the embedded sequence To obtain temporal embedding, The output is fed into a recurrent convolutional neural network model (RNN), which is used Feature matrix in alternative training data acquisition method , and use the implicit representation of the long short-term memory network model (LSTM) as the node In addition, the temporal embedding of other nodes can also be obtained by feeding the embedding sequence generated by the GCN model into an RNN model, and using this to construct the mathematical model of GCN-LSTM (see Fig.14 and Fig.15 ).
[0090] In the above embodiment of the present invention, as an optional implementation, the training of the constructed dynamic graph convolution-long short-term memory network model (GCN-LSTM) includes: applying the method for obtaining training data and the method for processing driving data, applying the dynamic graph convolution neural network model (GNC) based on the currently processed and obtained training data to obtain the node set according to formula (4): Implicit representation matrix of midpoint features , and each node obtained by training data The implicit representation matrix Input to the recurrent neural network model (RNN) to get training data for each node The embedding sequence ; Hidden state model based on the embedding sequence of previous training data and memory unit model The output data of the current i-th node The embedding sequence Data, apply the long short-term memory network model (LSTM) to get the current i-th node The hidden state model of the embedding sequence and memory unit model , where the hidden state model The implicit state of the embedding sequence corresponding to the current first i training values; Get the node set based on the dynamic graph convolutional neural network model (GCN) and recurrent neural network model (RNN) The temporal embedding sequence of each node in , based on the node set The hidden state model of the embedding sequence of each previous node in and memory unit model , and each current node The embedding sequence Input the long short-term memory network model (LSTM) to obtain the hidden state model corresponding to the embedding sequence of the current first t nodes and memory unit model .
[0091] (4)
[0092] Among them, the node set represents the number of nodes within the tth preset cycle length, i represents the node set The i-th node of , where i is an integer greater than 1 and t is an integer greater than 1.
[0093] It should be noted that the training data set can use an existing data set, or use the training data acquisition method to obtain it from the collected video data. A GCN unit and an LSTM unit are combined to form a GCN-LSTM model. The temporal embedding sequence and hidden state model of multiple groups of nodes and memory unit model The GCN-LSTM model within the next preset cycle duration is input as input data to predict the feature elements of the next node, so as to obtain the motion state of the vehicle within the next preset cycle duration.
[0094] In the above embodiment of the present invention, as an optional implementation, the constructed dynamic graph convolution-long short-term memory network model (GCN-LSTM) includes: obtaining the input gate model of the node , the forget gate model of the node , memory unit model Output gate model of the node , and the hidden state model Formula (5) to (9):
[0095] (5)
[0096] (6)
[0097] (7)
[0098] (8)
[0099] (9)
[0100] Wherein, σ(·) represents an activation function; t represents the duration of the t-th preset cycle, and τ represents an integer and satisfies 1≤t≤τ; Represents the input implicit representation matrix The weight of Indicates the implicit state of the loop The weight of The hidden state of the embedding sequence corresponding to the first t observations; is the bias data.
[0101] It should be noted that the general activation function is not limited to sigmoid. Specifically, Represents the input gate model middle The weight of Represents the input gate model middle The weight of Represents the input gate model The bias data in ; similarly, Represents the forget gate model middle The weight of Represents the forget gate model middle The weight of Represents the forget gate model The bias data in Representation memory unit model middle The weight of Representation memory unit model middle The weight of Representation memory unit model The bias data in Represents the output gate model middle The weight of Represents the output gate model middle The weight of Represents the output gate model In addition, and It can be initialized with 0 or a random value, or a set initialization value. The above weights are obtained by random initialization method to obtain the training weights of the above model. The present invention adopts stochastic gradient descent (SGD) to optimize the loss function and update the model parameters. Unlike the traditional gradient descent method, SGD only uses one sample at a time to calculate the gradient and update the parameters, instead of using the entire data set for training. This randomness makes SGD more efficient on large data sets because it only needs to process one sample in each iteration.
[0102] The present invention also provides a driving state prediction system, such as Figure 6 As shown, the driving state prediction system 700 includes: a model creation unit 710, which is used to obtain node feature element data of each vehicle to construct a dynamic graph convolution-long short-term memory network model (GCN-LSTM). A training unit 720, based on the node feature element data of each vehicle obtained by the application driving data processing method and the training data acquisition method, trains the dynamic graph convolution-long short-term memory network model (GCN-LSTM) until each node feature in the training data is regressed in time series. A prediction unit 730, which is used to convert the dynamic graph data continuously collected and processed within a first preset time length into a dynamic graph of discrete time. , the discrete-time dynamic graph The input is given to the trained dynamic graph convolution-long short-term memory network model (GCN-LSTM); based on the trained dynamic graph convolution-long short-term memory network model (GCN-LSTM), the status of the node feature elements of all current vehicles is predicted.
[0103] It should be noted that when the driving state prediction system 700 of the present invention is in operation, the dynamic graph (static graph) converted from the continuous time domain into discrete time is applied to the GCN-LSTM model for time series regression of the dynamic graph nodes, thereby enhancing the local structural information of the dynamic graph nodes before and after the current moment, and reducing the parameters required to be learned through the convolution operation of shared weights, thereby reducing the complexity of the model, improving the generalization ability of feature representation learning based on the dynamic graph nodes in the discrete time domain, and improving the accuracy of vehicle state prediction based on the discrete time dynamic graph.
[0104] The present invention also provides a driving state simulation test method, such as Figure 7 As shown, the driving state simulation test method applies the above-mentioned driving state prediction method, including: S11, simulating the movement of all current vehicles based on the state results of the node characteristic elements of all current vehicles predicted by the simulator in the simulation test. S12, simulating the movement state of the vehicles around the own vehicle based on the movement state of the own vehicle.
[0105] It should be noted that, first, the hypothetical states of the vehicle and surrounding vehicles within the first preset time step are pre-given, and the node data of the above vehicles are converted into a discrete time dynamic graph; the discrete time dynamic graph is input into the trained GCN-LSTM model; and the states of all vehicles are predicted. In addition, a simulator unit can be set up to simulate the movement of vehicles around the vehicle according to the prediction results in the simulation test through the simulator unit, so that the movement of vehicles around the vehicle can be displayed and output in real time in the form of images. The simulation test method of the present invention can improve the accuracy of vehicle state prediction simulation based on discrete time dynamic graphs.
[0106] The present invention also provides a driving state simulation test device, such as Figure 8 As shown, the driving state simulation test device 600 applies the driving state prediction system, including: a first simulation unit 610, which is used to simulate the movement of all current vehicles based on the state results of the node feature elements of all current vehicles predicted by the simulator in the simulation test. A second simulation unit 620, which is used to simulate the movement state of the vehicles around the own vehicle based on the movement state of the own vehicle.
[0107] It should be noted that when the driving state simulation test device 600 of the present invention is running, it can simulate the movement of vehicles around the vehicle according to the prediction results through the simulation unit and display, analyze and output them, thereby improving the accuracy of vehicle state prediction simulation based on discrete time dynamic graphs.
[0108] The present invention also provides a method for testing a driving state, such as Fig. 9 As shown, the driving state testing method applies the above-mentioned driving state simulation testing method and introduces the driver's confrontation control, including: S13, in the simulation test, obtaining the lateral and / or longitudinal control data of the driver controlling the own vehicle. S14, based on the recorded actual driving state data of the own vehicle naturally driving within the first preset time, updating the motion state data predicted by the dynamic graph convolution-long short-term memory network model (GCN-LSTM) for the own vehicle. S15, based on the actual driving state data updated by the own vehicle, updating the discrete time dynamic graph S16, based on the updated discrete-time dynamic graph The motion state of the own vehicle within a second preset time period is predicted.
[0109] It should be noted that in the simulation test, the driver will control the lateral and longitudinal control of the vehicle, and use the recorded actual driving state of the vehicle in the first time step to update the vehicle parameters in the GCN-LSTM prediction data; the updated discrete-time dynamic graph is input to the GCN-LSTM to predict the movement of vehicles around the vehicle in the second time step. By analogy, the movement of vehicles around the vehicle in other time steps is achieved. The present invention introduces driver confrontation control, enhances the local structural information of the dynamic graph node before and after the current moment based on the driver's confrontation control, reduces the parameters required for learning through the convolution operation of shared weights, reduces the complexity of its model, improves the generalization ability of feature representation learning based on dynamic graph nodes in the discrete time domain, and improves the accuracy of vehicle state prediction, simulation and testing based on discrete-time dynamic graphs; by updating the vehicle state based on driver control within a preset cycle duration, the accuracy of predicting the motion state of vehicles around the vehicle is further improved.
[0110] In the above-mentioned embodiment of the present invention, as an optional implementation, the driving status testing method also includes: updating the motion state data of the own vehicle predicted by the dynamic graph convolution-long short-term memory network model (GCN-LSTM) based on the actual driving state data of the own vehicle naturally driving within the tth preset time length; updating the discrete-time dynamic graph based on the updated actual driving state data of the own vehicle; predicting the motion state of the own vehicle within the t+1th preset time length after the tth preset time length based on the updated discrete-time dynamic graph.
[0111] It should be noted that the present invention further improves the accuracy of predicting the motion state of vehicles around the vehicle within each preset cycle duration by updating the vehicle state in real time based on driver control within each consecutive preset cycle duration.
[0112] The present invention also provides a driving state testing system, such as Fig.10 As shown, the driving state test system 500 applies the above-mentioned driving state simulation test method and introduces the driver's confrontation control, including: a control acquisition unit 510, which is used to obtain the lateral and / or longitudinal control data of the driver controlling the own vehicle in the simulation test; a first updating unit 520, which is used to update the motion state data predicted by the dynamic graph convolution-long short-term memory network model (GCN-LSTM) of the own vehicle based on the actual driving state data recorded during the natural driving of the own vehicle within a first preset time period; a second updating unit 530, which is used to update the discrete time dynamic graph based on the actual driving state data updated by the own vehicle ; Adversarial prediction unit 540, for discrete time dynamic graph based on update The motion state of the own vehicle within a second preset time period is predicted.
[0113] It should be noted that when the driving status test system 500 of the present invention is running, the vehicle status is updated in real time based on the driver's control within each consecutive preset cycle time, thereby further improving the accuracy of predicting the movement status of vehicles around the vehicle within each preset cycle time.
[0114] The present invention also provides a vehicle, which includes the driving data processing device mentioned above, the training data acquisition device mentioned above, the driving state prediction system mentioned above, the driving state simulation test device mentioned above, and the driving state test system mentioned above.
[0115] It should be noted that the above-mentioned vehicle includes a driving data processing device, a training data acquisition device, a driving state prediction system, a driving state simulation test device and a driving state test system. When the vehicle is running, it can update its own vehicle state in real time based on driver control within each consecutive preset cycle time, thereby further improving the accuracy of predicting the movement state of vehicles around the vehicle within each preset cycle time.
[0116] The present invention also provides an electronic device, such as Fig.11 As shown, the electronic device 400 includes: a memory 410 for storing non-temporary computer-readable instructions 430; and a processor 420 for running the computer-readable instructions 430, so that when the computer-readable instructions 430 are executed by the processor 420, the above-mentioned driving status testing method is implemented.
[0117] It should be noted that when the electronic device 400 of the present invention is running, it can adjust the spatial state quickly and conveniently during sampling, with small data jitter and high stability; it can convert the continuous time domain into a dynamic graph (static graph) of discrete time and apply it to the GCN-LSTM model to perform time series regression of the dynamic graph nodes, thereby enhancing the local structural information of the dynamic graph nodes before and after the current moment, reducing the parameters required to be learned through the convolution operation of shared weights, reducing the complexity of its model, improving the generalization ability of feature representation learning based on dynamic graph nodes in the discrete time domain, and improving the accuracy of vehicle state prediction based on discrete time dynamic graphs; by updating the vehicle state based on driver control within a preset cycle duration, the accuracy of predicting the motion state of vehicles around the vehicle is improved.
[0118] The present invention also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on a device, the device executes the above-mentioned driving status testing method.
[0119] It should be noted that, when the computer-readable storage medium of the present invention is running, it can adjust the spatial state quickly and conveniently during sampling, with small data jitter and high stability; it can convert the continuous time domain into a dynamic graph (static graph) of discrete time and apply it to the GCN-LSTM model to perform time series regression of the dynamic graph nodes, thereby enhancing the local structural information of the dynamic graph nodes before and after the current moment, reducing the parameters required for learning through the convolution operation of shared weights, reducing the complexity of the model, improving the generalization ability of feature representation learning based on dynamic graph nodes in the discrete time domain, and improving the accuracy of vehicle state prediction based on discrete time dynamic graphs; by updating the vehicle state based on driver control within a preset cycle duration, the accuracy of predicting the motion state of vehicles around the vehicle is improved.
[0120] In an embodiment of the present invention, Fig.16 As shown, first, the assumed states of the vehicle and surrounding vehicles in the first time step s0 are given in advance, and the data is converted into a discrete time dynamic graph ;Will Input to the trained GCN-LSTM model; predict the status of all vehicles through the GCN-LSTM model, and simulate the movement of vehicles around the self-vehicle according to the prediction results in the simulation test; introduce the driver's lateral and longitudinal control of the self-vehicle in the simulation test, and use the recorded actual driving status of the self-vehicle in the first time step s0 to update the self-vehicle parameters in the GCN-LSTM prediction data; update the discrete time dynamic graph The data is input to GCN-LSTM to predict the movement of the vehicles around the ego vehicle in the second time step. The above steps are repeated to obtain the movement state data of the vehicles around the ego vehicle in subsequent different time steps, and the movement state data of the vehicles around the ego vehicle is used to control the operation state of the ego vehicle.
[0121] It should be noted that any process or method description in the flowchart or otherwise described herein may be understood to represent a module, fragment or portion of a code comprising one or more executable instructions for implementing the steps of a specific logical function or process, and that the scope of the preferred embodiments of the present invention includes alternative implementations in which functions may not be performed in the order shown or discussed, including performing functions in a substantially simultaneous manner or in reverse order depending on the functions involved, which should be understood by those skilled in the art to which the embodiments of the present invention belong.
[0122] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in combination with these instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in combination with these instruction execution systems, devices or apparatuses. More specific examples (non-exhaustive list) of computer-readable media include the following: an electrical connection with one or more wires (electronic device), a portable computer disk box (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium and then editing, interpreting or otherwise processing in a suitable manner if necessary, and then stored in a computer memory.
[0123] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or a combination thereof: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.
[0124] A person skilled in the art may understand that all or part of the steps included in the method for implementing the above-mentioned embodiment may be completed by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, which, when executed, includes one of the steps of the method embodiment or a combination thereof.
[0125] In addition, each functional unit in each embodiment of the present invention may be integrated into a processing module, or each unit may exist physically separately, or two or more units may be integrated into one module. The above-mentioned integrated module may be partially implemented in the form of hardware, or may be implemented in the form of a software functional module. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.
[0126] The storage medium mentioned above may be a read-only memory, a magnetic disk or an optical disk, etc. Although the embodiments of the present invention have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as limiting the present invention. A person of ordinary skill in the art may change, modify, replace and modify the above embodiments within the scope of the present invention.
[0127] The above are only preferred embodiments of the present invention, and are not intended to limit the present invention in any form. Although the present invention has been disclosed as a preferred embodiment as above, it is not intended to limit the present invention. Any technician familiar with the profession can make some changes or modify the technical contents disclosed above into equivalent embodiments without departing from the scope of the technical solution of the present invention. However, any simple modification, equivalent change and modification made to the above embodiments based on the technical essence of the present invention without departing from the content of the technical solution of the present invention still fall within the scope of the technical solution of the present invention.
Claims
1. A method for processing driving data, characterized in that: include: Based on the video stabilization method, the collected natural driving video data is processed, and the feature point matching includes feature point detection, feature descriptor extraction and matching of feature points and feature descriptors; Based on the feature point matching algorithm, the first frame of the video data is used as the reference frame, and the feature points of all images are matched with the first frame to detect the feature points in the video data; Eliminate false matching of feature points based on random sample consensus algorithm; map feature points of all images to the reference frame plane based on homography transformation; Video data after georeferencing; Obtain the rotating detection frame of the vehicle's natural driving based on the target detection algorithm; Track the heading angle of the vehicle's natural driving and the pixel coordinates of the video data, obtain the lane lines detected from the natural driving video data based on the object detection algorithm, and assign ID data to the lanes based on the safety requirements and tests of the road scene; The trajectory is smoothed to obtain the world coordinates converted from pixel coordinates to obtain natural driving data.
2. The method for processing driving data according to claim 1, characterized in that: Before processing the collected natural driving video data, it also includes: Based on the use of fixed-state drones on roads with heavy traffic, video data of vehicles and registration points on the road are collected.
3. The method for processing driving data according to claim 1, characterized in that: The video data of natural driving collected by processing in a video stabilization manner includes: Based on the feature point matching algorithm, the first frame of the video data is used as the reference frame, and the feature points of all images are matched with the first frame to detect the feature points in the video data; Eliminate false matching of feature points based on random sample consensus algorithm; The feature points of all images are mapped to the reference frame plane based on homography transformation.
4. The method for processing driving data according to claim 3, characterized in that: The video data after the geo-reference processing includes: Based on the n reference points selected on the basic frame of the mapping feature points based on the satellite map, the longitude and latitude of the n reference points on the basic frame and their converted world coordinates are obtained; Based on the pixel coordinates and world coordinates of the selected n reference points and the state parameters of the camera, the camera's extrinsic parameter matrix k1 is obtained based on formula (1); (1) Based on formula (1), the pixel coordinates of the base frame are converted into world coordinates; Among them, (X w , Y w , Z w ) is the world coordinate of the reference point on the base frame; (u, v) is the pixel coordinate of the reference point; k1 is the external parameter matrix of the camera; k2 is the internal parameter matrix of the camera; the state parameters of the camera include Z c , Z c is the vertical height from the optical center of the drone camera to the ground; n is a positive integer not less than 4.
5. The method for processing driving data according to claim 4, characterized in that: Perform checkerboard calibration on the base frame based on the selected reference points to obtain the pixel focal length of the main reference point on the base frame, and obtain the camera's intrinsic parameter matrix k2 according to formula (2); (2) Among them, f x is the length of the pixel focal length in the x-axis direction; f y is the length of the pixel focal length in the y-axis direction; (u0, v0) is the actual pixel coordinate of the main reference point.
6. The method for processing driving data according to claim 4, characterized in that: It also includes selecting n reference points to obtain the rotation matrix and translation matrix of the camera's extrinsic matrix according to formula (3); (3) Among them, R represents the rotation matrix and T represents the translation matrix.
7. The method for processing driving data according to claim 1, characterized in that: The target detection algorithm uses a rotating target detection algorithm, and uses the rotation detection frame of the vehicle driven naturally as the length and width of the vehicle.
8. The method for processing driving data according to claim 7, characterized in that: The pixel coordinates of the heading angle and video data of the natural driving of the tracking vehicle include: Video data of natural driving of the vehicle is tracked based on the tracking algorithm.
9. A driving data processing device, characterized in that: include: A stability processing unit, used for processing the collected natural driving video data based on a feature point matching method, wherein the feature point matching includes feature point detection, feature descriptor extraction, and matching of feature points and feature descriptors; Based on the feature point matching algorithm, the first frame of the video data is used as the reference frame, and the feature points of all images are matched with the first frame to detect the feature points in the video data; Eliminate false matching of feature points based on random sample consensus algorithm; map feature points of all images to the reference frame plane based on homography transformation; A geo-reference unit, used for geo-referenced video data; A detection and tracking unit, used to obtain a rotation detection frame of a vehicle driving naturally based on a target detection algorithm; Track the heading angle of the vehicle's natural driving and the pixel coordinates of the video data, obtain the lane lines detected from the natural driving video data based on the object detection algorithm, and assign ID data to the lanes based on the safety requirements and tests of the road scene; The trajectory smoothing unit is used for trajectory smoothing processing of world coordinates converted from pixel coordinates to obtain natural driving data.
10. A method for obtaining training data, characterized in that: include: Based on the continuous time dynamic graph, the discrete time dynamic graph is obtained in the order of image frames. ; Dynamic graph based on each discrete time Create a node set based on each vehicle detected in the process, the adjacency relationship between each vehicle and its surrounding vehicles, and the node feature elements of each vehicle. , adjacency matrix and the feature matrix ; Based on dynamic graph The image frames are sampled to obtain the adjacency matrix of vehicle training data in natural driving video data and the feature matrix ; Wherein, t represents the duration of the t-th preset cycle, and τ represents an integer and satisfies 1≤t≤τ.
11. The method for obtaining training data according to claim 10, characterized in that: Creating a node set include: Set each vehicle detected in natural driving data as a node, and the vehicles detected and identified in the same frame as a node set .
12. The method for obtaining training data according to claim 10, characterized in that: The adjacency matrix is created include: Safety requirements and test decision node sets based on road scenarios The adjacency relationship between each vehicle and its surrounding vehicles, and based on the node set Create an adjacency matrix based on the adjacency relationship of all vehicles in .
13. The method for obtaining training data according to claim 10, characterized in that: The creation of the feature matrix include: Create a feature matrix based on the node feature elements of the vehicle , the node set The node features of each vehicle in the feature matrix a line of; The node feature elements of each vehicle include the current lane ID, vehicle speed, vehicle acceleration, vehicle heading angle and offset.
14. The method for obtaining training data according to claim 10, characterized in that: The dynamic graph based The image frame sampling includes: The natural driving video data based on continuous frames is sampled with a preset time period of adjacent frames; or The natural driving video data based on continuous frames is sampled at a preset number of frames; or Feature matrix of natural driving video data based on the current continuous frames Sampling is performed to determine whether the data of the previous several frames and the next several frames have changed.
15. The method for obtaining training data according to claim 14, characterized in that: The feature matrix of the natural driving video data based on the current continuous frame Sampling whether the data changes between the previous and next frames includes: If the characteristic matrix If there is no change within the preset time period, sampling is performed at the preset time period; otherwise With feature matrix When data of several previous frames and several subsequent frames change, the previous frame before the change and the subsequent frame after the change are sampled.
16. A device for acquiring training data, characterized in that: include: A dynamic image generation unit is used to obtain a discrete time dynamic image arranged in the order of image frames based on the continuous time dynamic image. ; Create units for dynamic graphs based on each discrete time Create a node set based on each vehicle detected in the process, the adjacency relationship between each vehicle and its surrounding vehicles, and the node feature elements of each vehicle. , adjacency matrix and the feature matrix ; Sampling unit, used for dynamic graph The image frames are sampled to obtain the adjacency matrix of vehicle training data in natural driving video data and the feature matrix ; Wherein, t represents the duration of the t-th preset cycle, and τ represents an integer and satisfies 1≤t≤τ.
17. A method for predicting a driving state, characterized in that: include: A dynamic graph convolution-long short-term memory network model constructed by acquiring node feature element data of each vehicle; Applying the driving data processing method according to any one of claims 1 to 8 and the training data acquisition method according to any one of claims 10 to 15 to obtain the node feature element data of each vehicle to train the dynamic graph convolution-long short-term memory network model until each node feature in the training data is time-series regressed; Convert the dynamic image data collected and processed continuously within the first preset time period into a dynamic image of discrete time , the discrete-time dynamic graph The input is given to the trained dynamic graph convolution-long short-term memory network model; based on the trained dynamic graph convolution-long short-term memory network model, the status of the node feature elements of all current vehicles is predicted.
18. The method for predicting driving status according to claim 17, characterized in that: The training of the constructed dynamic graph convolution-long short-term memory network model includes: Apply the method for obtaining training data and the method for processing driving data, and apply the dynamic graph convolutional neural network model based on the currently processed and obtained training data to obtain the node set according to formula (4): Implicit representation matrix of midpoint features , and each node obtained by training data The implicit representation matrix Input to the recurrent neural network model to get training data for each node The embedding sequence ; Hidden state model based on the embedding sequence of previous training data and memory unit model The output data of the current i-th node The embedding sequence Data, apply the long short-term memory network model to get the current i-th node The hidden state model of the embedding sequence and memory unit model , where the hidden state model The hidden state corresponding to the embedding sequence of the first i training values; Obtaining node sets based on dynamic graph convolutional neural network models and recurrent neural network models The temporal embedding sequence of each node in , based on the node set The hidden state model of the embedding sequence of each previous node in and memory unit model , and each current node The embedding sequence Input the long short-term memory network model (LSTM) to obtain the hidden state model corresponding to the embedding sequence of the current first t nodes and memory unit model ; (4) Among them, the node set represents the number of nodes within the tth preset cycle length, i represents the node set The i-th node of , where i is an integer greater than 1 and t is an integer greater than 1.
19. The method for predicting driving status according to claim 18, characterized in that: The constructed dynamic graph convolution-long short-term memory network model includes: Get the input gate model of the node , the forget gate model of the node , memory unit model Output gate model of the node , and the hidden state model Formula (5) to (9): (5) (6) (7) (8) (9) in, () represents the activation function; t represents the duration of the t-th preset cycle, τ represents an integer and satisfies 1≤t≤τ; Represents the input implicit representation matrix The weight of Indicates the implicit state of the loop The weight of The hidden state of the embedding sequence corresponding to the first t observations; is the bias data.
20. A driving state prediction system, characterized in that: include: A model creation unit, used to construct a dynamic graph convolution-long short-term memory network model based on the acquired node feature element data of each vehicle; A training unit, which trains the dynamic graph convolution-long short-term memory network model based on node feature element data of each vehicle obtained by applying the driving data processing method and the training data acquisition method, until each node feature in the training data is time-series regressed; The prediction unit is used to convert the dynamic image data continuously collected and processed within the first preset time into a dynamic image of discrete time. , the discrete-time dynamic graph The input is given to the trained dynamic graph convolution-long short-term memory network model; based on the trained dynamic graph convolution-long short-term memory network model, the status of the node feature elements of all current vehicles is predicted.
21. A driving state simulation test method, characterized in that: The method for predicting a driving state according to any one of claims 17 to 19 is applied, comprising: In the simulation test, the movement of all current vehicles is simulated based on the simulator according to the predicted state results of the node characteristic elements of all current vehicles; The motion state of the vehicles around the own vehicle is simulated based on the motion state of the own vehicle.
22. A driving state simulation test device, characterized in that: The driving state prediction system according to claim 20 comprises: A first simulation unit, used for simulating the movement of all current vehicles based on the state results of the node characteristic elements of all current vehicles predicted by the simulator in the simulation test; The second simulation unit is used to simulate the motion state of vehicles around the own vehicle based on the motion state of the own vehicle.
23. A driving state testing method, characterized in that: The driving state simulation test method as claimed in claim 21 is applied to introduce the driver's confrontation control, including: In the simulation test, obtaining the lateral and / or longitudinal control data of the driver controlling his own vehicle; Based on the recorded actual driving state data of the own vehicle during natural driving within a first preset time period, the dynamic graph convolution-long short-term memory network model updates the motion state data predicted by the own vehicle; Update the discrete time dynamic graph based on the actual driving status data updated by the own vehicle ; Update-based discrete-time dynamic graphs The motion state of the own vehicle within a second preset time period is predicted.
24. The driving state testing method according to claim 23, characterized in that: Also includes: Based on the recorded actual driving state data of the own vehicle during the t-th preset time, the dynamic graph convolution-long short-term memory network model updates the motion state data predicted by the own vehicle; Update the discrete-time dynamic graph based on actual driving status data updated by the own vehicle; The motion state of the own vehicle within the t+1th preset time period after the tth preset time period is predicted based on the updated discrete-time dynamic graph.
25. A driving state testing system, characterized in that: The driving state simulation test method as claimed in claim 21 is applied to introduce the driver's confrontation control, including: A control acquisition unit, used to acquire lateral and / or longitudinal control data of the driver controlling his own vehicle in a simulation test; A first updating unit is used to update the motion state data predicted by the dynamic graph convolution-long short-term memory network model for the own vehicle based on the recorded actual driving state data of the own vehicle naturally driving within a first preset time period; The second updating unit is used to update the discrete time dynamic graph based on the actual driving state data updated by the own vehicle ; Adversarial prediction unit for update-based discrete-time dynamic graphs The motion state of the own vehicle within a second preset time period is predicted.
26. A vehicle, characterized in that: It includes a driving data processing device as described in claim 9, a training data acquisition device as described in claim 16, a driving state prediction system as described in claim 20, a driving state simulation test device as described in claim 22, and a driving state test system as described in claim 25.
27. An electronic device comprising: a memory for storing non-transitory computer-readable instructions; as well as A processor is used to run computer-readable instructions so that when the computer-readable instructions are executed by the processor, a driving status testing method as described in any one of claims 23 to 24 is implemented.
28. A computer-readable storage medium, characterized in that: It comprises computer instructions, which, when executed on a device, enable the device to execute the driving state testing method as claimed in any one of claims 23 to 24.
