Driving scene data enhancement method based on track editing and image translation
Through the method based on trajectory editing and image translation, the problems of instability and insufficient diversity of the GAN data enhancement method generation process are solved, the diversity and recognition accuracy of driving scenario data are improved, and the accuracy of the forward collision warning system is improved. Responsiveness.
Patent Information
- Application Number
- CN202411127481.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-16
- Publication Date
- 2024-12-03
AI Technical Summary
The data enhancement method of driving scenarios based on GAN in the prior art has problems such as unstable generation process, insufficient detail capture and insufficient diversity, and it is difficult to meet the diversity and richness requirements of deep learning models for high-risk driving scenarios, affecting forward collisions. Accuracy of early warning system.
Using a method based on track editing and image translation, traffic participants are detected by obtaining vehicle forward videos, forming an object trajectory map, and using an image translation model to convert it into a motion profile map, combining trajectory editing methods (such as trajectory erasing and replacement , translation) enhancement data set to generate driving scenario data with clear trajectory semantics and controllable.
The trajectory semantic clarity and traffic environment diversity of the motion profile diagram are improved, the accuracy of dangerous driving scenario recognition is improved, the diversity and richness of the model is enhanced, and the foresight and real-time response capabilities of the forward collision warning system are improved.
Smart Images

Figure CN119068459A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of vehicle automatic driving, and in particular to a driving scene data enhancement method based on trajectory editing and image translation. Background Art
[0002] Forward collision is the dominant type of traffic accident. To meet this severe challenge, the research community has focused on optimizing road infrastructure, upgrading traffic management systems, and developing advanced driver assistance systems (ADAS). In particular, many automakers have considered the integration of forward collision warning systems (FCW) as standard equipment in their vehicles, and regard it as one of the key technical paths to effectively curb collision accidents.
[0003] The core of the early FCW algorithm lies in the use of radar data, but it is limited by the high cost and limited detection range of radar. With the leap of computer vision and deep learning technology, vision-based ADAS has made breakthrough progress. Existing companies have achieved a warning accuracy rate of more than 80% with video data-driven FCW technology. In addition, the policy orientation of many countries, such as requiring commercial vehicles to install on-board recorders, has laid a solid foundation for the widespread popularization of FCW systems, heralding a broad prospect for the application of forward collision warning technology.
[0004] At present, the mainstream research on visual assisted driving systems focuses on building a vehicle collision risk assessment model using the output of the target detection algorithm. This method analyzes the target detection bounding box in the video frame, extracts key parameters such as the relative distance and speed change of traffic participants, and then calculates immediate indicators such as collision time (TTC) to trigger the warning mechanism. However, the inherent omission and false detection phenomena of the target detection algorithm directly affect the accuracy of the TTC calculation, and this method is limited to responding to immediate risk indicators and lacks predictive assessment of future risk events. In addition, it is highly dependent on the accuracy of the target detection algorithm and requires intensive processing of each frame of the video sequence, which significantly increases the burden on the on-board computing platform and poses a challenge to the real-time response capability of the forward collision warning system.
[0005] Therefore, a lightweight risk scene recognition method is proposed in the prior art. The motion contour map has the advantages of simple calculation, low computing power requirements, and complete extraction of driving scene information. It is suitable as the input of the lightweight dangerous driving scene recognition model. The motion features of the objects in the motion contour map are learned and extracted through a lightweight deep neural network, so as to evaluate and predict the collision risk. However, since some lightweight risk scene recognition methods do not rely on the target detection algorithm, and the pixel average operation is simple to calculate, the terminal computing power requirements are low; at the same time, the method uses a deep neural network to extract risk features instead of calculating specific risk indicators (such as TTC), which has a higher tolerance for data noise. However, there are also disadvantages. Due to the use of deep neural network modeling, the method has extremely high requirements for the diversity and richness of modeling samples. In fact, the integrity and richness of driving scenes collected from natural driving data are limited. In addition, collisions are rare occasional events, and the high-risk driving scenes that can be extracted from them are not able to meet the requirements of deep learning in terms of quantity, richness and diversity. Therefore, exploring effective driving scene data enhancement methods has become the key to the success of this method. Patent CN117746098A discloses a high-risk driving scene recognition method based on GAN data enhancement. This method uses a GAN model for data enhancement. Although the GAN-based visual generation technology performs well in image clarity, it does not capture enough details and is prone to mode collapse and homogenization, resulting in an unstable GAN training process. In addition, due to its characteristic of generating images from noise, the generation process is uncontrollable, which limits the diversity of the generated motion contour map, thereby affecting the performance of the model in complex scenarios. Therefore, it is necessary to propose a data enhancement method that can obtain data with clear trajectory semantics and a controllable generation process. Summary of the invention
[0006] The purpose of the present invention is to provide a driving scene data enhancement method based on trajectory editing and image translation, which can realize the purpose of controllable generation process and effective motion contour map data enhancement.
[0007] The purpose of the present invention can be achieved by the following technical solutions:
[0008] A driving scene data enhancement method based on trajectory editing and image translation comprises the following steps:
[0009] Acquire the vehicle-mounted forward video, detect traffic participants from the video image, and extract and combine the trajectories of the traffic participants according to the time sequence relationship to form an object trajectory map;
[0010] Expanding the object trajectory map by using a trajectory editing method to obtain an extended object trajectory map;
[0011] constructing an object trajectory atlas based on the object trajectory map and the extended object trajectory map, inputting the atlas into a pre-trained image translation model, and generating an enhanced motion contour atlas;
[0012] The training steps of the image translation model include:
[0013] Generate motion contour map based on vehicle-mounted forward video;
[0014] A dataset of object trajectory map-motion contour map pairs is established based on the motion contour map and the object trajectory map, and is input into the image translation model for training until the training is completed.
[0015] Furthermore, the step of forming the object trajectory graph includes:
[0016] Traffic object detection: Assume that there are n frames of vehicle-mounted forward video. For any frame of video image I i (1≤i≤n), all traffic objects are detected using target detection algorithm;
[0017] Expression of traffic objects: The traffic objects that actually interact are denoted as o j (1≤j≤m), and take the horizontal line segment in the middle of each traffic object detection area represents traffic objects, where m is the number of traffic objects interacting in the video image, and x j_l ,y j 、x j_r ,y j Represents a horizontal line segment The coordinates of the left and right endpoints;
[0018] Division of regions: The vehicle's forward direction is divided into the front area, the left front area and the right front area. The front area is the conflict area, while the left front area and the right front area are the non-conflict areas.
[0019] Partition information organization: Create a one-dimensional array of length N based on the front area, left front area and right front area Traverse all horizontal line segments in each area Set up a one-dimensional array The coordinates are in [x j_l ,x j_r ], where c represents the category and takes l, f or r, l, f, r represent the left front, front, and right front respectively, and the pixel value is set as:
[0020]
[0021] In the formula, Represents a one-dimensional array The pixel value of the kth pixel, O c Movement trajectories of traffic objects formed for each area;
[0022] Traffic object trajectory extraction: Arranged from top to bottom in the order of the video image frames, the motion trajectories of traffic objects in the left front area, the front area, and the right front area of the vehicle are respectively formed, which can be expressed as:
[0023]
[0024] In the formula, OT l ,OT f ,OT r The motion trajectories of traffic objects in the left front area, the front area, and the right front area of the vehicle respectively;
[0025] Object trajectory map generation: OT l ,OT f ,OT r Use G, R, and B channels to superimpose each other to form the final object trajectory map;
[0026] Calibration of object trajectory map: Data calibration of object trajectory map is performed, which is divided into two label types: non-conflict driving scenario and high-risk driving scenario.
[0027] Furthermore, the track editing method includes one or more of track erasing, track replacement and track translation.
[0028] Furthermore, the step of erasing the track includes:
[0029] Get the object trajectory map I in RGB format, and the minimum value of the preset number n n and the maximum value max n , randomly select a certain number of rectangular regions {R1,R2,…,R n};
[0030] For each selected rectangular area R i , the minimum value of the preset length w is min w and the maximum value max w , set the height of the rectangle to h, where the length w is a random value between the preset minimum and maximum values;
[0031] For each selected rectangular area R i , remove the blue channel B and the green channel G, and keep only the red channel R, that is: I(R i )=I(R i )(R).
[0032] Furthermore, the step of replacing the track includes:
[0033] Dividing the object trajectory graph into two groups: non-dangerous driving scene S1 and dangerous driving scene S2;
[0034] Extract the pixel vector x of each object trajectory map i , using Euclidean distance as the metric distance, the kNN algorithm is applied to identify the K nearest neighbor images {x i1 ,x i2 ,…,x iK}, where the Euclidean distance d(x i ,x j ) is expressed as follows:
[0035]
[0036] Randomly select any nearest neighbor image x ik , and compare it with the original image x i The blue channel B and green channel G are replaced by trajectories, namely:
[0037] Furthermore, the step of translating the trajectory includes:
[0038] Convert the object trajectory map I into a tensor T, preset the minimum value min and the maximum value max of the number of pixels to be translated, and randomly determine the direction d and the pixel distance p of the image translation, where min≤p≤max, and d includes up, down, left, and right;
[0039] The tensor T is expanded using edge copy filling technology to obtain the filled tensor T padded ;
[0040] Based on the padded tensor T padded Perform translation operation to obtain tensor T shifted , and is converted to PIL image format I shifted , where the tensor T shifted It is expressed as:
[0041] T shifted =shift(T padded ,d,p)
[0042] In the formula, shift represents the translation operation.
[0043] Furthermore, the image translation model is constructed based on the pix2pixHD model.
[0044] Furthermore, the step of generating a motion profile graph comprises:
[0045] Focus area setting: Assume that there are n frames of video images in the vehicle forward video. For any frame of video image I i (1≤i≤n), take the rectangular area in the middle and record it as image
[0046] Image information compression: image It is only a vertical compression, taking the average of the pixels in the same vertical direction to get the pixel line
[0047] Object motion trajectory extraction: pixel line Arrange the video image frames from top to bottom in order to form a motion profile map MP m ;
[0048] Modeling sample calibration: motion profile MP m Data calibration is carried out and divided into two label types: non-conflict driving scenarios and high-risk driving scenarios.
[0049] Furthermore, the step of establishing a data set of object trajectory graph-motion profile graph pairs includes:
[0050] The object trajectory map and motion contour map from the same driving scene are combined into a map pair;
[0051] Perform m changes in environmental factors on the motion contour map in each image pair to expand 1 image pair into m+1 image pairs;
[0052] Perform n geometric transformations on the motion contour graph in each image pair to expand 1 image pair into n+1 image pairs;
[0053] Based on the expanded image pairs, an image pair dataset of the object trajectory map-motion contour map is formed.
[0054] Further, it also includes taking evaluation steps, including:
[0055] Build a risk identification model;
[0056] The enhanced motion contour atlas is input into the risk identification model to obtain the predicted risk classification probability, and the AUC value, model accuracy, precision, recall rate and F1 value are used for evaluation to obtain the evaluation result.
[0057] Compared with the prior art, the present invention has the following beneficial effects:
[0058] (1) The present invention aims to address the limited accuracy of dangerous driving scene recognition caused by unclear semantics of lines in motion contour maps, and combines the clear semantics of trajectories in object trajectory maps to translate object trajectory maps into motion contour maps using an image translation model. Through image translation, the motion contour map can effectively inherit the trajectory subjects of traffic participants in the object trajectory map, improve the clarity of the trajectory semantics of the motion contour map, enhance the diversity of the traffic environment, and achieve the purpose of motion contour map data enhancement.
[0059] (2) The present invention makes full use of the respective advantages of the object trajectory map and the motion contour map, and adopts three trajectory editing methods to expand the object trajectory map, including trajectory erasure, trajectory replacement and trajectory translation. Trajectory erasure simulates the failure of the object detection algorithm in a harsh traffic environment; trajectory replacement replaces the surrounding non-critical traffic participants while retaining the key conflicting objects, so as to enhance the richness and diversity of the driving scene; trajectory translation is mainly used to simulate the change of the installation position of the vehicle-mounted camera. By performing trajectory editing on the object trajectory map, the driving scene can be enriched and expanded to a certain extent.
[0060] (3) The present invention translates the object trajectory map into a motion contour map through an image translation model, and can also add various rich traffic background details such as the sky, roads, and roadside buildings, which can further enhance the diversity of the traffic environment and thus improve the accuracy of dangerous driving scene recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0061] Figure 1 It is a schematic diagram of the method flow of the present invention;
[0062] Figure 2 A schematic diagram of a motion profile generation process of the present invention;
[0063] Figure 3 A schematic diagram of the object trajectory diagram generation process of the present invention;
[0064] Figure 4 It is a schematic diagram of three-channel superposition of the object trajectory diagram of the present invention;
[0065] Figure 5 This is a schematic diagram of comparing the trajectory graphs of an object before and after replacing the trajectory using the kNN algorithm of the present invention;
[0066] Figure 6 A schematic diagram showing the comparison of object trajectory diagrams before and after trajectory erasure of the present invention;
[0067] Figure 7 It is a schematic diagram comparing the object trajectory diagrams before and after all trajectories of the present invention are translated to the left and right sides;
[0068] Figure 8 It is a demonstration of the pair of graphs of the object trajectory graph and the motion profile graph of the present invention;
[0069] Fig. 9 It is a schematic diagram for comparing the original motion profile map and the enhanced motion profile map of the present invention. DETAILED DESCRIPTION
[0070] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. This embodiment is implemented based on the technical solution of the present invention, and provides a detailed implementation method and specific operation process, but the protection scope of the present invention is not limited to the following embodiments.
[0071] Motion contour maps are simple to calculate, require low computing power, and extract complete driving scene information. They are suitable as input for lightweight dangerous driving scene recognition models. Lightweight deep neural networks are used to learn and extract the motion features of objects in motion contour maps, thereby evaluating and predicting collision risks. However, due to the use of deep neural network modeling, the method has extremely high requirements for the diversity and richness of modeling samples. In fact, the completeness and richness of driving scenes collected from natural driving data are limited. In addition, collisions are rare and sporadic events. The high-risk driving scenes that can be extracted from them are even less in quantity, richness, and diversity. They cannot meet the requirements of deep learning. Therefore, exploring effective driving scene data enhancement methods and generating more diverse motion contour maps have become the key to the success of lightweight dangerous driving scene recognition methods.
[0072] To this end, this embodiment provides a driving scene data enhancement method based on trajectory editing and image translation to enhance the motion contour map, such as Figure 1 As shown, the method comprises the following steps:
[0073] Step 1: Generate a motion contour map for the driving scene based on the onboard forward video.
[0074] This embodiment predicts dangerous scenarios based on regular driving video records of a commercial truck fleet over a two-year period.
[0075] The video captured by the on-board camera is converted into a motion profile (MP). The specific method involves extracting visual information from the middle area of each frame of the video image, compressing the area vertically, and averaging the pixel values of each column to form a single pixel line representing the frame information. Finally, these pixel lines arranged in time series are integrated to form a motion profile, which condenses the temporal dynamic information of the video while highlighting the motion trajectory characteristics of all objects in the driving scene, providing a targeted data basis for subsequent model training.
[0076] The detailed process of step 1 is as follows:
[0077] Step 1.1: Focus area setting: Assume that the forward video captured by the in-car camera has a total of n frames. For any frame image I i (1≤i≤n), take the rectangular area in the middle of the image (between 0.75 and 0.85 times the image height), denoted as
[0078] Step 1.2: Image information compression: compress the image in the area vertically and take the average value of the pixels in the same vertical direction. Compressed into pixel lines
[0079] Step 1.3: Object motion trajectory extraction: convert horizontal pixel lines Arrange the frames from top to bottom in order to form a motion profile map MP m In this case, a motion profile is generated as shown below: Figure 2 shown.
[0080]
[0081] Step 1.4: Modeling sample calibration: Watch the vehicle forward video, calibrate the motion profile data based on the thresholds of speed, acceleration, etc. combined with expert experience, and divide the data into two label types: "non-conflict driving scene" and "high-risk driving scene". The original motion profile and its labels are divided into training set Θ according to the ratio of training set: test set: validation set = 8:1:1. train , validation set Θ valid With the test set Θ test .
[0082] Step 2: Generate an object trajectory map for the driving scene based on the onboard forward video.
[0083] For each frame of video image, all traffic elements are marked by the target detection algorithm, covering actual interactive objects and non-interactive environmental details. Subsequently, experts intervened to identify traffic objects that interact with the vehicle and divided the forward video into three front areas: left, front, and right. For interactive traffic objects in each area, the object box is compressed into a horizontal line segment, and then mapped to the pixel array of the corresponding category (left, front, and right). These arrays are stacked vertically through the frame sequence to form three images. Finally, these three monochrome images are combined according to the red, green, and blue channels (red represents the interactive object directly in front, green for the left front, and blue for the right front) to generate a color object trajectory map (Objects Trajectory, OT), which eliminates unnecessary environmental interference and concentrates on the dynamic trajectory changes of traffic participants in the driving scene.
[0084] The detailed process of step 2 is as follows:
[0085] Step 2.1: Detection of traffic objects: Only for the training set Θ train The vehicle video corresponding to the motion contour map is processed. Assume that the forward video captured by the in-vehicle camera has a total of n frames. For any frame image I i (1≤i≤n), use the target detection algorithm to detect all traffic objects in the frame image.
[0086] Step 2.2: Expression of traffic objects: Experts in professional fields evaluate and identify traffic objects that interact with the vehicle in the driving scene. The traffic objects that actually interact with the vehicle are denoted as j (1≤j≤m), where m is the number of traffic objects that actually interact in the frame image. Take the horizontal line segment in the middle of each traffic object detection area (Bounding box) To represent traffic objects, the length of the line segment represents the distance of the traffic object. The shorter the line segment, the farther the object is.
[0087] Step 2.3: Division of conflict areas: Divide the area in front of the vehicle into three areas: left front, front, and right front. The front is the conflict area, and the objects in this area are considered the main interactive objects. The left and right sides are non-conflict areas. The conflict area can be obtained through lane line detection. When the lane line detection algorithm fails, such as in rural roads, intersections, etc. where lane lines are missing or difficult to identify at night, the line connecting the vanishing point and the left and right fifth points below the video frame can be used as the default lane line. In this case, Figure 3 The triangle area formed by the two yellow lines is the conflict area.
[0088] Step 2.4: Partition information organization: For each partition, create a one-dimensional array of length N Where N is the number of pixel columns in the horizontal direction of the frame image, c represents the category, which can be l, f or r, representing the left front area, the front area and the right front area of the vehicle respectively. Traverse all horizontal line segments in the partition The one-dimensional array L coordinates in [x j_l ,x j_r ]The pixel values in the interval are set to 255, and the remaining positions are set to 0. Represents the pixel value of the kth pixel in the one-dimensional array L.
[0089]
[0090] Step 2.5: Extraction of the trajectory of traffic objects: Arrange from top to bottom in the order of frame time to form three pictures OT l ,OT f ,OT r, respectively represent the motion trajectories of traffic objects in front of the left, front, and right of the vehicle:
[0091]
[0092] Step 2.6: Object trajectory map generation: OT l ,OT f ,OT r The three channels G, R, and B are superimposed together to form the final object trajectory map (OT). In this case, the object trajectory map generation process and the three-channel superposition process are as follows: Figure 3 and Figure 4 shown.
[0093] Step 2.7: Calibration of object trajectory map: The calibration rules are the same as those in step 1.4. The object trajectory map is calibrated for "non-conflict driving scenarios" and "high-risk driving scenarios" based on the video.
[0094] Step 3: Driving scenario extension based on trajectory editing of object trajectory graph.
[0095] Based on the object trajectory map samples generated in step 2, a series of data enhancement strategies are adopted to expand the driving scene, mainly including ① trajectory replacement, ② trajectory erasure, and ③ trajectory translation. ① Trajectory replacement is to use the kNN algorithm to realize trajectory replacement to simulate the diversified changes of traffic objects in the left and right lanes; ② Trajectory erasure simulates the abnormal unrecognized areas of the object to enhance the model's tolerance to the errors of the object recognition algorithm; ③ Trajectory translation refers to moving the trajectory of traffic objects in different directions, such as up, down, left, and right, to simulate the changes in the viewing angles and installation positions of different on-board cameras, and to improve the diversity of driving scenes.
[0096] The detailed process of step 3 is as follows:
[0097] Step 3.1: Extension based on trajectory replacement: Perform trajectory replacement based on the kNN algorithm, and replace the traffic objects in the left and right lanes for similar object trajectories in the same category to increase the richness of the driving scene. The logic of trajectory replacement data enhancement is that whether the driving scene is dangerous is mainly determined by the state of the traffic object in front, and has little to do with the traffic objects on the side. Therefore, replacing the traffic object information in the left and right lanes usually does not change the scene label. The detailed process is as follows:
[0098] 1) The object trajectory graph is divided into two groups: "non-dangerous driving scene" (S1) and "dangerous driving scene" (S2) to avoid the traffic objects belonging to the dangerous driving scene being mistakenly replaced with the non-dangerous driving scene, or vice versa, in the subsequent data enhancement process;
[0099] 2) Extract the pixel vector x of each image i, using the Euclidean distance ∥x i -x j ∥2 is used as the distance metric and the kNN algorithm is applied to identify the K nearest neighbor images {x i1 ,x i2 ,…,x iK}. For two image feature vectors x of length n i and x j , the Euclidean distance d(x i ,x j ) is expressed as follows:
[0100]
[0101] 3) If Figure 5 As shown, by randomly selecting a nearest neighbor image x ik , and compare it with the original image x i The blue channel B and green channel G of , complete the track replacement.
[0102]
[0103] Step 3.2: Extension based on track erasure: Perform rectangular track erasure on the left and right lane traffic object tracks to simulate the situation where the target detection algorithm fails to successfully identify the target object during the construction of the object track map. In order to ensure the label accuracy of the extended image, the red channel R is kept unchanged, and track erasure is performed only on the blue channel B and the green channel G. The detailed process is as follows:
[0104] 1) Read the RGB format image I and preset the minimum value of the number n n and the maximum value max n , randomly select a certain number of rectangular regions {R1,R2,…,R n}, the number n is between the preset minimum and maximum values, that is, min n ≤n≤max n ; In this case, take the min n is 1, take max n 7, which has the greatest improvement on model performance;
[0105] 2) For each selected rectangular area R i , the minimum value of the preset length w is min w and the maximum value max w , set the height of the rectangle to h, and the length w to a random value between the preset minimum and maximum values, that is, min w ≤w≤max w ; In this case, h is taken as 20, min w is 150, take maxw is 250;
[0106] 3) If Figure 6 As shown, for each selected removal rectangular area R i , remove the blue channel B and the green channel G, and keep only the red channel R, that is:
[0107] I(R i )=I(R i )(R)
[0108] Step 3.3: Extension based on trajectory translation: The trajectories of traffic objects in all partitions are translated to the same side at the same time to simulate the movement of the camera position or the offset of the shooting angle in the actual driving scene. The detailed process is as follows:
[0109] 1) Convert the input image I into a PyTorch tensor T, preset the minimum and maximum values of the number of pixels to be translated, and randomly determine the direction d and pixel distance p of the image translation, where min≤p≤max, and d includes up, down, left, and right;
[0110] 2) If Figure 7 As shown in the figure, the tensor T is expanded by using edge copy padding technology to prevent the phenomenon that the edge is completely empty after the translation trajectory. The tensor after padding is denoted as T padded ;
[0111] 3) Complete the translation operation T shifted =shift(T padded ,d,p), the generated tensor T shifted is converted back to PIL image format I shifted .
[0112] Step 4: Motion contour map sample enhancement based on image translation technology.
[0113] First, based on steps 1 and 2, we construct a dataset of object trajectory maps and motion contour maps, and build an image translation model M from the object trajectory map domain to the motion contour map domain based on the pix2pixHD model. i2i ; Then construct the object trajectory atlas S sp , including the original object trajectory map and the extended object trajectory map generated in step 3, in the object trajectory map set S sp Using the image translation model M i2i Generate motion profile atlas
[0114] The detailed process of step 4 is as follows:
[0115] Step 4.1: Model parameter adjustment:
[0116] Adjust some parameters of the pix2pixHD model (including loadSize, etc., see Table 1 for details) to make it more suitable for the dataset used for training;
[0117] Table 1 Some parameters of the pix2pixHD model recommended for adjustment
[0118]
[0119] Step 4.2: Enhance color brightness, etc.:
[0120] The object trajectory map and motion contour map from the same driving scene are combined into a map pair. For each pair of motion contour maps, m changes in color, brightness, etc. are taken (as shown in Table 2) to simulate the changes in environmental factors in the actual driving environment, and 1 map pair is expanded to m+1 map pairs;
[0121] Table 2 Recommended image color transformation parameters
[0122]
[0123] Step 4.3: Enhancements such as rotation and translation:
[0124] A relatively conservative geometric transformation data ratio and transformation range (as shown in Table 3) are adopted for the motion contour map part of the image pair to simulate the position change and camera parameter change of the vehicle-mounted camera in the actual driving environment. If n changes are made on one image pair, then one image pair is expanded to n+1 image pairs.
[0125] Table 3. Recommended image geometric transformation parameters
[0126]
[0127] Step 4.4: Image translation model training:
[0128] Train the image translation model M on the object trajectory map-motion contour map pair dataset i2i In this case, the object trajectory map-motion contour map pair is demonstrated as follows Figure 8 shown.
[0129] Step 4.5: Motion contour map enhancement:
[0130] Construct object trajectory atlas S sp , including the original object trajectory map and the extended object trajectory map generated in step 3, in the object trajectory map set S sp Using M i2i Model-generated motion profile atlas In this case, Fig. 9 : is a comparison diagram of the original motion contour map and the enhanced motion contour map. It can be seen that the generated motion contour map and the original motion contour map are similar in structure and diverse in content.
[0131] Step 5: Construct dangerous driving recognition model and evaluate data enhancement effect.
[0132] Based on deep neural networks (such as ResNet-18 models), a dangerous driving recognition model is constructed, and the motion contour map after data enhancement (including the original motion contour map S mp and the resulting motion profile ) as the training data of the model, the real motion profile map as the verification data and test data, and the binary classification data of whether it is a high-risk driving scene (0 is a non-conflict scene, 1 is a high-risk scene) as the classification label to train the model, carry out model training and evaluate the model performance;
[0133] The detailed process of step 5 is as follows:
[0134] Step 5.1: Design a risk identification model:
[0135] Construct the network structure of the dangerous driving recognition model. The detailed process is as follows:
[0136] 1) First, the input layer has an input image size of 3*224*224. If the image size does not match, the size is transformed first.
[0137] 2) Connect to the ResNet-18 network, set the parameters of the fully connected layer, and output a 2*1 matrix. The two values in the matrix correspond to the predicted values of the probability of belonging to the risky category and the risk-free category. Then use Softmax to process the predicted values so that the sum of the probabilities of the two categories is 1.
[0138] Step 5.2: Training sample enhancement:
[0139] The motion contour map generated in step 4.5 The training dataset Θ integrated into the model train In order to achieve the purpose of training sample enhancement;
[0140] Step 5.3: Risk identification model training:
[0141] After the enhanced training data set Θ train On, after n epoch Iteration cycles, the model is trained on the training set Θ train The loss value on the validation set Θ converges to a smaller value and test The loss value on the M is basically no longer decreasing. At this time, save the model M TA ;
[0142] Step 5.4: Model testing and data enhancement effect evaluation:
[0143] For the test set Θ test (including e c non-conflict driving scenarios and e n Each scenario in the high-risk driving scenario) calls the trained M TA The model is constructed to obtain its predicted classification probability, the classification threshold is calculated according to the Youden index method, the confusion matrix is calculated, the model accuracy, precision, recall and F1 value are calculated, the AUC curve is drawn and the AUC value is calculated; the model performance indicators before and after data enhancement are compared to evaluate and identify the data enhancement effect.
[0144] Table 4 Model performance index results
[0145]
[0146] In this case, the performance indicators of the model under different data enhancement strategies are shown in Table 4, where the basic enhancement refers to directly transforming the object trajectory map through M i2i The model is translated into a motion contour map for enhancement, and the trajectory operation on the right means that the object trajectory map is first subjected to the specified trajectory operation and then enhanced by M i2i The model translation is enhanced into a motion contour map. The experimental results show that the AUC of the basic model without data enhancement is 0.8788. After enhancement, the AUC index is 0.9407, an increase of 7%. In addition, the continuous improvement of the model AUC under different enhancement strategy combinations also shows that the trajectory editing and image translation technology proposed in this patent effectively improves the accuracy of the model by increasing the diversity of driving scenarios.
[0147] In summary, this embodiment proposes a driving scene data enhancement method based on trajectory editing and image translation. First, traffic participants are detected, and the trajectories of traffic participants are extracted and combined according to the temporal relationship to form an object trajectory map (Objects Trajectory), which represents the dynamic change of the relative position of the traffic participants in front of the vehicle in a driving scene segment in the temporal sequence. The object trajectory map realizes the projection of the driving scene video on the two-dimensional space of the image, and is mainly composed of the trajectories of traffic participants. Both the motion contour map and the object trajectory map are compressions of the driving scene. The motion contour map is simple to calculate, but the semantics of the lines in the image are unclear; the trajectory semantics of the object trajectory map are clear, but the accuracy depends on the accuracy of the target detection algorithm and has high computing power requirements. This embodiment makes full use of the respective advantages of these two types of maps. First, three trajectory editing methods are proposed to expand the object trajectory map, including trajectory erasure, trajectory replacement and trajectory translation. Trajectory erasure is to simulate the failure of the object detection algorithm in a harsh traffic environment; trajectory replacement is to replace the surrounding non-critical traffic participants while retaining the key conflicting objects, so as to improve the richness and diversity of the driving scene; trajectory translation is mainly used to simulate the changes in the installation position of the vehicle-mounted camera. By editing the trajectory of the object trajectory map, the driving scene can be enriched and expanded to a certain extent; on this basis, a dataset of object trajectory map-motion contour map pairs is established, and the object trajectory map is translated into the motion contour map by using image translation technology. Through image translation, the motion contour map can effectively inherit the trajectory of traffic participants in the object trajectory map, and at the same time add various rich traffic background details such as the sky, roads, and roadside buildings, which can further enhance the diversity of the traffic environment. The experimental results show that after trajectory editing and image translation, the diversity and richness of driving scene samples represented by the motion contour map are effectively enhanced, and the accuracy AUC index of the dangerous driving scene recognition model trained based on the enhanced samples increases from 0.8788 to 0.9407, and the performance is improved by 7%.
[0148] If the above functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program code.
[0149] It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes. The schemes in the embodiments of the present invention may be implemented in various computer languages, for example, object-oriented programming language Java and literal scripting language JavaScript, etc.
[0150] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0151] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0152] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.
[0153] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0154] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A driving scene data enhancement method based on trajectory editing and image translation, characterized in that: The following steps are involved: Acquire the vehicle-mounted forward video, detect traffic participants from the video image, and extract and combine the trajectories of the traffic participants according to the time sequence relationship to form an object trajectory map; Expanding the object trajectory map by using a trajectory editing method to obtain an extended object trajectory map; constructing an object trajectory atlas based on the object trajectory map and the extended object trajectory map, inputting the atlas into a pre-trained image translation model, and generating an enhanced motion contour atlas; The training steps of the image translation model include: Generate motion contour map based on vehicle-mounted forward video; A dataset of object trajectory map-motion contour map pairs is established based on the motion contour map and the object trajectory map, and is input into the image translation model for training until the training is completed.
2. The driving scene data enhancement method based on trajectory editing and image translation according to claim 1, characterized in that: The step of forming the object trajectory graph comprises: Traffic object detection: Assume that there are n frames of vehicle-mounted forward video. For any frame of video image I i (1≤i≤n), all traffic objects are detected using target detection algorithm; Expression of traffic objects: The traffic objects that actually interact are denoted as o j (1≤j≤m), and take the horizontal line segment in the middle of each traffic object detection area represents traffic objects, where m is the number of traffic objects interacting in the video image, and x j_l ,y j 、x j_r ,y j Represents a horizontal line segment The coordinates of the left and right endpoints; Division of regions: The vehicle's forward direction is divided into the front area, the left front area and the right front area. The front area is the conflict area, while the left front area and the right front area are the non-conflict areas. Partition information organization: Create a one-dimensional array of length N based on the front area, left front area and right front area Traverse all horizontal line segments in each area Set up a one-dimensional array The coordinates are in [x j_l ,x j_r ], where c represents the category and takes l, f or r, l, f, r represent the left front, front, and right front respectively, and the pixel value is set as: In the formula, Represents a one-dimensional array The pixel value of the kth pixel, O c Movement trajectories of traffic objects formed for each area; Traffic object trajectory extraction: Arranged from top to bottom in the order of the video image frames, the motion trajectories of traffic objects in the left front area, the front area, and the right front area of the vehicle are respectively formed, which can be expressed as: In the formula, OT l ,OT f ,OT r The motion trajectories of traffic objects in the left front area, the front area, and the right front area of the vehicle respectively; Object trajectory map generation: OT l ,OT f ,OT r Use G, R, and B channels to superimpose each other to form the final object trajectory map; Calibration of object trajectory map: Data calibration of object trajectory map is performed, which is divided into two label types: non-conflict driving scenario and high-risk driving scenario.
3. The driving scene data enhancement method based on trajectory editing and image translation according to claim 1, characterized in that: The track editing mode includes one or more of track erasing, track replacement and track translation.
4. The driving scene data enhancement method based on trajectory editing and image translation according to claim 3 is characterized in that: The step of erasing the track includes: Get the object trajectory map I in RGB format, and the minimum value of the preset number n n and the maximum value max n , randomly select a certain number of rectangular regions {R1, R2, ..., R n }; For each selected rectangular area R i , the minimum value of the preset length w is min w and the maximum value max w , set the height of the rectangle to h, where the length w is a random value between the preset minimum and maximum values; For each selected rectangular area R i , remove the blue channel B and the green channel G, and keep only the red channel R, that is: I(R i )=I(R i )(R).
5. The driving scene data enhancement method based on trajectory editing and image translation according to claim 3 is characterized in that: The step of trajectory replacement includes: Dividing the object trajectory graph into two groups: non-dangerous driving scene S1 and dangerous driving scene S2; Extract the pixel vector x of each object trajectory map i , using Euclidean distance as the metric distance, the kNN algorithm is applied to identify the K nearest neighbor images {x i1 , x i2 , ..., x iK }, where the Euclidean distance d(x i , x j ) is expressed as follows: Randomly select any nearest neighbor image x ik , and compare it with the original image x i The blue channel B and green channel G are replaced by trajectories, namely:
6. The driving scene data enhancement method based on trajectory editing and image translation according to claim 3 is characterized in that: The step of trajectory translation comprises: Convert the object trajectory map into a tensor T, preset the minimum value min and the maximum value max of the number of pixels to be translated, and randomly determine the direction d and the pixel distance p of the image translation, where min≤p≤max, and d includes up, down, left, and right; The tensor T is expanded using edge copy filling technology to obtain the filled tensor T padded ; Based on the padded tensor T padded Perform translation operation to obtain tensor T shifted , and is converted to PIL image format I shifted , where the tensor T shifted It is expressed as: T shifted =shift(T padded ,d,p) In the formula, shift represents the translation operation.
7. The driving scene data enhancement method based on trajectory editing and image translation according to claim 1, characterized in that: The image translation model is built based on the pix2pixHD model.
8. The driving scene data enhancement method based on trajectory editing and image translation according to claim 1, characterized in that: The step of generating a motion profile diagram comprises: Focus area setting: Assume that there are n frames of video images in the vehicle forward video. For any frame of video image I i (1≤i≤n), take the rectangular area in the middle and record it as image Image information compression: image It is only a vertical compression, taking the average of the pixels in the same vertical direction to get the pixel line Object motion trajectory extraction: pixel line Arrange the video image frames from top to bottom in the order of the frames to form a motion profile map Mpmm; Modeling sample calibration: The motion profile map Mpmm is calibrated and divided into two label types: non-conflict driving scenarios and high-risk driving scenarios.
9. The driving scene data enhancement method based on trajectory editing and image translation according to claim 1, characterized in that: The step of establishing a graph pair data set of an object trajectory graph and a motion profile graph comprises: The object trajectory map and motion contour map from the same driving scene are combined into a map pair; Perform m changes in environmental factors on the motion contour map in each image pair to expand 1 image pair into m+1 image pairs; Perform n geometric transformations on the motion contour graph in each image pair to expand 1 image pair into n+1 image pairs; Based on the expanded image pairs, an image pair dataset of the object trajectory map-motion contour map is formed.
10. The driving scene data enhancement method based on trajectory editing and image translation according to claim 1, characterized in that: It also involves taking an assessment step, including: Build a risk identification model; The enhanced motion contour atlas is input into the risk identification model to obtain the predicted risk classification probability, and the AUC value, model accuracy, precision, recall rate and F1 value are used for evaluation to obtain the evaluation result.
Citation Information
Patent Citations
High-risk driving scene identification method based on GAN data enhancement
CN117746098A
Cited By
Scene reconstruction method and device, electronic equipment, storage medium and program product
CN120635331A
