A prediction method for near-field vehicle jamming behavior based on image input

Abstract

The present invention relates to a method for predicting the behavior of a near-field vehicle cutting in based on image input, the method comprising: (1) collecting image sequence information based on a forward panoramic image in a real structured road scene, and annotating the position and behavior information of the vehicle target in the image sequence by an artificial method; (2) constructing a near-field vehicle detection and tracking model suitable for detecting and tracking near-field vehicles in structured roads; (3) constructing a lane line detection network suitable for detecting lane lines in structured roads and a corresponding loss function; (4) based on the vehicle ID obtained by the near-field vehicle detection and tracking model established in step (2) and the bounding box position data of the corresponding target, and the lane line obtained by the lane line detection network established in step (3), the relative position deviation between the target and the lane line is obtained, and according to the formulation of a priori rules, the prediction result of the cutting in behavior of the near-field vehicle is obtained. Compared with the prior art, the present invention has high prediction accuracy and high efficiency.

Description

Technical Field

[0001] The present invention relates to the field of intelligent driving technology, and in particular to a method for predicting near-field vehicle lane-cutting behavior based on image input. Background Art

[0002] Behavior prediction is a further development based on behavior recognition. As one of the basic tasks in the field of computer vision, with the rapid development of deep learning technology in recent years, behavior prediction algorithms also have algorithms that formulate prior rules and end-to-end prediction technologies based on deep neural networks. The methods of behavior recognition and prediction have evolved from the initial methods based on physical motion features to the SlowFast network based on visual video input, the TSN network based on bimodal input action recognition network, and the 3D convolutional neural network based on dilated three-dimensional convolution (I3D). Many good algorithm technologies have emerged. These algorithms have excellent detection effects and performance on open human behavior recognition datasets. However, for the near-field vehicle jamming behavior prediction task, there are the following shortcomings in practical applications:

[0003] First, among the existing public datasets, there is a lack of ego-vehicle perspective datasets for predicting near-field vehicle jamming behavior. Compared with target detection datasets, there are many difficulties in data collection and annotation, which limits the further development of this technology.

[0004] Second, in the dual-modal input technology, optical flow is a hand-crafted feature and is trained separately from RGB input, which cannot achieve end-to-end training. The accuracy of the system needs to be improved, and the complex algorithm reduces the real-time operation of the system.

[0005] Third, the hardware cost and maintenance cost of the LiDAR-based method are relatively high. At the same time, there is currently a lack of ego vehicle perspective behavior prediction methods that use forward panoramic image videos as input.

[0006] Fourth, the end-to-end method based on deep learning has problems such as poor generalization of the training model, unclear specific principles, and high requirements for mobile hardware, making it difficult to quickly implement in actual scenarios. Summary of the invention

[0007] The purpose of the present invention is to overcome the defects of the above-mentioned prior art and to provide a near-field vehicle squeezing behavior prediction method based on image input. The method can use the forward panoramic image video data provided by the on-board high-definition camera within a period of time as input, and utilize the target detection and tracking algorithm based on image input to obtain the perception and tracking of the forward target vehicle's area of ​​interest from the perspective of the ego vehicle (Ego Vehicle), and then predict the behavior of the area of ​​interest based on an algorithm that establishes prior rules. While ensuring the reasoning speed, the actual deployment software and hardware costs are greatly reduced, and finally a more accurate prediction of the nearby vehicle squeezing behavior is obtained, which provides sufficient time for the intelligent driving system to avoid risks and improves the overall safety of the intelligent driving system.

[0008] The purpose of the present invention can be achieved by the following technical solutions:

[0009] A method for predicting near-field vehicle jamming behavior based on image input, the method comprising:

[0010] (1) Collect image sequence information based on forward panoramic images in real structured road scenes, and annotate the position and behavior information of vehicle targets in the image sequence through manual methods;

[0011] (2) Construct a near-field vehicle detection and tracking model suitable for near-field vehicle detection and tracking on structured roads;

[0012] (3) Construct a lane detection network and corresponding loss function suitable for lane detection on structured roads;

[0013] (4) Based on the vehicle ID and the bounding box position data of the corresponding target obtained by the near-field vehicle detection and tracking model established in step (2), and the lane line obtained by the lane line detection network established in step (3), the relative position deviation between the target and the lane line is obtained, and according to the formulation of prior rules, the prediction result of the near-field vehicle's lane-cutting behavior is obtained.

[0014] Preferably, step (1) specifically includes:

[0015] (11) Calibrate the internal and external parameters of the camera, where the external parameters include the rotation matrix R and the translation vector T, and the internal parameters include the internal parameter matrix K and the camera distortion coefficient;

[0016] (12) Using a data collection vehicle equipped with a camera to collect video data in a real road scene, and recording the category of vehicle targets in the image during collection;

[0017] (13) Use annotation tools to annotate the collected video data. The annotation methods include vehicle target tracking ID annotation, vehicle target category annotation, target object bounding box annotation, key frame annotation of the start of vehicle lane-cutting, vehicle crossing the midpoint of the lane line, and vehicle completion of lane-cutting behavior, and vehicle lane-cutting behavior category annotation. The annotation content must at least include the position of the near-field vehicle, key frames, and lane-cutting behavior category information.

[0018] Preferably, step (2) specifically includes:

[0019] (21) Construct a near-field vehicle target detection network based on the improved Yolov5, input the input video slices as image time series into the near-field vehicle target detection network, and perform feature extraction and feature encoding on the input image information after multi-layer convolution and downsampling operations to obtain a multi-dimensional feature tensor that divides the image;

[0020] (22) Construct a classification network and use non-maximum suppression operations to finally obtain the location information and classification confidence information of each target, including the classification probability and positioning probability of the object;

[0021] (23) A near-field vehicle target tracking network based on the improved Deep-SORT is constructed. The target object bounding box information and classification information obtained by target detection are used as input. Multiple objects in the video are simultaneously located and tracked, and their ID and trajectory information are recorded. In particular, the transformation of object ID is reduced under occlusion conditions. The tracking ID, target category, and target object bounding box information of the target vehicle are output.

[0022] Preferably, step (3) specifically includes:

[0023] (31) Construct a lane feature extraction backbone network based on a convolutional neural network, output features based on a shallow residual connection network, and improve the inference speed of the model while ensuring the detection effect by using a larger receptive field;

[0024] (32) Construct a lane line semantic segmentation network. During network training, multi-scale features are upsampled to the same scale and subjected to transposed convolution. The semantic segmentation loss is calculated to enhance the visual feature extraction capability of the backbone network. Finally, an enhanced lane line detection backbone network based on residual connection is obtained.

[0025] (33) The features extracted by the backbone network are used to calculate candidate points through a classifier on a global scale according to the a priori specified longitudinal candidate anchor frame of the image, and finally the lane line position node of the lane where the ego vehicle is located is obtained;

[0026] (34) Construct the loss function of the lane detection network, including multi-classification loss, segmentation loss and lane structuring loss.

[0027] Preferably, the loss function of the lane detection network is expressed as L total :

[0028] L total =L cls +L seg +ηL lane

[0029] L cls is the multi-classification loss, L seg is the segmentation loss, L lane is the lane structured loss and η is a hyperparameter.

[0030] Preferably, the multi-classification loss L cls It is expressed as:

[0031]

[0032] Among them, L CE (·) represents the cross entropy loss function, P i，j，： represents the prediction results of all (w+1) lane line units for the i-th lane line and the j-th lateral anchor box, T i，j，： represents the true distribution of all (w+1) lane line units for the i-th lane line and the j-th lateral anchor box, c i，j，： Indicates P i，j，： With T i，j，： , C and h represent the number of lane line classes and the number of lane longitudinal anchor points respectively, and γ and α are hyper parameters.

[0033] Preferably, the lane structure loss L lane It is expressed as:

[0034] L lane =L sim +λ Lshp

[0035]

[0036]

[0037] Among them, L sim is the similarity loss, L shp is the shape loss, λ is the hyperparameter representing the loss weight, P i,j,k represents the predicted probability of the i-th lane line at position (j, k), and w is the number of division units in each row.

[0038] Preferably, the steps of training the network and formulating prior rules in step (4) are as follows:

[0039] (41) Preprocessing the collected image sequence, including: randomly flipping the image horizontally, cropping it, and uniformly scaling it to a fixed size, and flipping, cropping, and scaling the annotated data accordingly, and then normalizing the obtained image by channel;

[0040] (42) Using the lane line position nodes of the lane where the ego vehicle is located obtained by lane line detection, a lane line fitting model of the lane where the ego vehicle is located is obtained by using a highly robust regression model;

[0041] (43) Establishing an area of ​​interest for the jamming behavior based on the lane line model of the lane where the ego vehicle is located, and calculating the position deviation between the target vehicle boundary box information and the area of ​​interest for the jamming behavior, and establishing the expected number of jamming behaviors and the vehicle state symbol dictionary for each target based on the target vehicle tracking ID;

[0042] (44) By setting the expected number of lane-cutting behaviors and combining the vehicle status symbol to determine the specific behavior of the target vehicle, the parameters of the established lane-cutting behavior region of interest are updated, and the ideal network parameters are obtained after iteration.

[0043] Preferably, the lane line fitting model of the lane where the ego vehicle is located is a linear model, and the left and right lanes are fitted according to the lane line prediction position nodes respectively.

[0044] Preferably, the key value in the dictionary of expected number of lane-cutting behaviors and vehicle status symbols for each target is the target tracking ID, and the value is the expected number and the vehicle status symbol.

[0045] Compared with the prior art, the present invention has the following advantages:

[0046] (1) The present invention proposes a new network structure to address the problem that current neural networks are not good at predicting the behavior of vehicles cutting in. Under the premise of limited computing power on the mobile terminal, the performance of real-time prediction of the behavior of nearby vehicles is greatly improved. The network structure can be easily deployed in the existing intelligent driving system, so that the intelligent driving car can respond to the behavior of nearby vehicles cutting in in a timely manner, thereby improving the safety during driving.

[0047] (2) The present invention first extracts the video clips of lane-changing behaviors and the bounding box information of the target, and selects the data set of the adjacent vehicle jamming behaviors that meets the definition above, and finally establishes a "nearby object typical jamming behavior database" data set containing annotations and video data;

[0048] (3) The present invention obtains forward-view images with a large field of view and high resolution, including the appearance features of the target and the dependencies between targets, and develops a nearby vehicle detection and tracking model and a lane-cutting behavior prediction algorithm. The target detection module is deeply improved based on the latest One-Stage target detection algorithm Yolov5, which has a high detection speed while maintaining a certain detection accuracy;

[0049] (4) After outputting the bounding box and category information of the target, the present invention uses the Deep-SORT multi-target tracking algorithm to obtain a frame sequence of interest corresponding to each target ID. Considering that the traditional dual-modal input network has high requirements on the system's computing resources, in order to ensure the real-time performance of the algorithm, this paper does not use optical flow as the extraction of time domain features, but uses the target sequence as the spatiotemporal feature input;

[0050] (5) In the lane-cutting behavior prediction module of the present invention, starting from the characteristic information of the lane-cutting behavior, based on the algorithm for formulating prior rules and the target sequence output obtained in the previous step, a real-time and highly robust prediction method for the adjacent object lane-cutting behavior in an interpretable vehicle driving scenario is proposed. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 It is a flow chart of a method for predicting near-field vehicle jamming behavior based on image input according to the present invention;

[0052] Figure 2 It is a schematic diagram of the structure of the near-field vehicle target detection network of the present invention;

[0053] Figure 3 The first part of the algorithm flow for predicting the lane-cutting behavior based on a priori rules;

[0054] Figure 4 The second part of the algorithm flow for predicting the lane-cutting behavior based on the prior rules;

[0055] Figure 5 This is the third part of the algorithm flow for predicting lane-cutting behavior based on prior rules. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with the accompanying drawings and specific embodiments. Note that the following embodiments are merely illustrative in nature, and the present invention is not intended to limit its applicable objects or uses, and the present invention is not limited to the following embodiments.

[0057] Example

[0058] The main steps of the behavior prediction method of the present invention include collecting image sequence information based on forward panoramic images in real structured road scenes, annotating the position and behavior information of vehicle targets in the image sequence by artificial methods; constructing a deep convolutional neural network suitable for near-field vehicle detection and tracking in structured roads; constructing a deep neural network suitable for lane line detection in structured roads and a corresponding loss function; obtaining the relative position deviation between the target and the lane line from the bounding box position data of the vehicle ID and the corresponding target obtained from the established near-field vehicle detection and tracking model, and the lane line model obtained from the established lane line detection network, and obtaining the prediction result of the near-field vehicle's jamming behavior according to the formulation of prior rules. The present invention proposes a new jamming behavior prediction algorithm to address the problem that the current neural network is not good at predicting the jamming behavior of vehicles. Under the premise of limited computing power on the mobile terminal, the performance of real-time behavior prediction of near-field vehicles is greatly improved, and it can be conveniently deployed in the existing intelligent driving system, so that the intelligent driving car can respond to the jamming behavior of near-field vehicles in a timely manner, thereby improving the safety during driving.

[0059] like Figure 1 As shown, this embodiment provides a method for predicting near-field vehicle jamming behavior based on image input, the method comprising:

[0060] (1) Collect image sequence information based on forward panoramic images in real structured road scenes, and annotate the position and behavior information of vehicle targets in the image sequence through manual methods;

[0061] (2) Construct a near-field vehicle detection and tracking model suitable for near-field vehicle detection and tracking on structured roads;

[0062] (3) Construct a lane detection network and corresponding loss function suitable for lane detection on structured roads;

[0063] (4) Based on the vehicle ID and the bounding box position data of the corresponding target obtained by the near-field vehicle detection and tracking model established in step (2), and the lane line obtained by the lane line detection network established in step (3), the relative position deviation between the target and the lane line is obtained, and according to the formulation of prior rules, the prediction result of the near-field vehicle's lane-cutting behavior is obtained.

[0064] Specifically, step (1) includes:

[0065] (11) Calibrate the internal and external parameters of the camera, where the external parameters include the rotation matrix R and the translation vector T, and the internal parameters include the internal parameter matrix K and the camera distortion coefficient;

[0066] (12) Using a data collection vehicle equipped with a camera to collect video data in a real road scene, and recording the category of vehicle targets in the image during collection;

[0067] (13) Use annotation tools to annotate the collected video data. The annotation methods include vehicle target tracking ID annotation, vehicle target category annotation, target object bounding box annotation, key frame annotation of the start of vehicle lane-cutting, vehicle crossing the midpoint of the lane line, and vehicle completing lane-cutting behavior, and vehicle lane-cutting behavior category annotation. The annotation content must at least include the position of the near-field vehicle, key frame, and lane-cutting behavior category information. The vehicle target ID required to be recorded in this process is unique.

[0068] Step (2) specifically includes:

[0069] (21) A near-field vehicle target detection network based on improved Yolov5 is constructed. The input video slices are input as image time series into the near-field vehicle target detection network. After multi-layer convolution and downsampling operations, the input image information is subjected to feature extraction and feature encoding to obtain a multi-dimensional feature tensor that divides the image. The network structure of this part of the near-field vehicle target detection network is as follows: Figure 2 As shown, it consists of Backbone, FPN, PAN and other structures;

[0070] (22) Construct a classification network and use non-maximum suppression operations to finally obtain the location information and classification confidence information of each target, including the classification probability and positioning probability of the object;

[0071] (23) A near-field vehicle target tracking network based on the improved Deep-SORT is constructed. The target object bounding box information and classification information obtained by target detection are used as input. Multiple objects in the video are simultaneously located and tracked, and their ID and trajectory information are recorded. In particular, the transformation of object IDs is reduced under occlusion conditions. The tracking ID, target category, and target object bounding box information of the target vehicle are output. The ReID module in the Deep-SORT network is trained on the new vehicle re-identification dataset Compcars that has been reclassified.

[0072] Step (3) specifically includes:

[0073] (31) A lane feature extraction backbone network based on a convolutional neural network is constructed. The features are output based on a shallow residual connection network. By using a larger receptive field, the inference speed of the model is improved while ensuring the detection effect. The shallow residual connection convolutional neural network output has four scales, namely 112×112×3, 56×56×64, 28×28×128, and 14×14×256:

[0074] (32) Construct a lane line semantic segmentation network. During network training, multi-scale features are upsampled to the same scale, and after transposed convolution, the semantic segmentation loss is calculated to enhance the visual feature extraction capability of the backbone network. Finally, an enhanced lane line detection backbone network based on residual connection and a lane line semantic segmentation network are obtained. During network training, multi-scale features are upsampled to the same scale, but not used in the actual inference process.

[0075] (33) The features extracted by the backbone network are used to calculate candidate points through a classifier on a global scale according to the longitudinal candidate anchor boxes of the image specified a priori, and finally the lane line position node of the lane where the ego vehicle is located is obtained. In this embodiment, the value of the number of lane line classes C is 4, and the value of the number of longitudinal candidate anchor boxes h is 18;

[0076] (34) Construct the loss function of the lane detection network, including multi-classification loss, segmentation loss and lane structure loss. The loss function of the lane detection network is expressed as L total :

[0077] L total =L cls +L seg +ηL lane

[0078] L cls is the multi-classification loss, L seg is the segmentation loss, L lane is the lane structured loss and η is a hyperparameter.

[0079] Among them, the multi-classification loss L cls and segmentation loss L seg The cross entropy loss function is used, and the segmentation loss L seg A multi-classification loss with 2 classification numbers was used.

[0080] Specifically, the multi-classification loss L cls It is expressed as:

[0081]

[0082] Among them, L CE (·) represents the cross entropy loss function, P i，j，： represents the prediction results of all (w+1) lane line units for the i-th lane line and the j-th lateral anchor box, T i，j，： represents the true distribution of all (w+1) lane line units for the i-th lane line and the j-th lateral anchor box, c i，j，： Indicates P i，j，： With T i，j，： , C and h represent the number of lane line classes and the number of lane longitudinal anchor points respectively, and γ and α are hyper parameters.

[0083] Lane structure loss L lane It is expressed as:

[0084] L lane =L sim +λL shp

[0085]

[0086]

[0087] Among them, L sim is the similarity loss, L shp is the shape loss, λ is the hyperparameter representing the loss weight, P i，j，k represents the predicted probability of the i-th lane line at position (j, k), and w is the number of lane line units in each row.

[0088] In the embodiment, the value of η is 1, the value of γ is 2, the value of α is 0.25, and the value of λ is 1.25.

[0089] Step (4) The steps of training the network and formulating prior rules are as follows:

[0090] (41) Preprocessing the collected image sequence, including: randomly flipping the image horizontally, cropping it, and uniformly scaling it to a fixed size, and flipping, cropping, and scaling the annotated data accordingly, and then normalizing the obtained image by channel;

[0091] (42) The lane line position nodes of the lane where the ego vehicle is located obtained by lane line detection are used to obtain a lane line fitting model of the lane where the ego vehicle is located using a high robustness regression model. The lane line fitting model of the lane where the ego vehicle is located is a linear model, and the left and right lanes are fitted according to the lane line prediction position nodes respectively;

[0092] (43) Establishing an area of ​​interest for the jamming behavior based on the lane line model of the lane where the ego vehicle is located, and calculating the position deviation between the target vehicle boundary box information and the area of ​​interest for the jamming behavior, and establishing a dictionary of the expected number of jamming behaviors and vehicle status symbols for each target based on the target vehicle tracking ID, wherein the key value of the dictionary of the expected number of jamming behaviors and vehicle status symbols for each target is the target tracking ID, and the value is the expected number of times and the vehicle status symbol;

[0093] (44) By setting the expected number of lane-cutting behaviors and combining the vehicle status symbol to determine the specific behavior of the target vehicle, the parameters of the established lane-cutting behavior region of interest are updated, and the ideal network parameters are obtained after iteration.

[0094] According to the formulation of the priori rules, the specific process of the algorithm for predicting the jamming behavior of the near-field vehicle is as follows: Figure 3 , Figure 4 , Figure 5 As shown in the figure. First, after each adjacent vehicle is detected and tracked, a target state descriptor is constructed with the target tracking ID as the key and the initial state [0, 'keep'] as the value, where 0 is the count value and 'keep' is the historical state descriptor. According to the different areas where the adjacent vehicle detection bounding box (Bounding Box) is located, the initial lane of the adjacent vehicle is judged to be the left lane or the right lane; at the same time, according to the lane line model obtained by the lane line detection module, the width of both sides of the lane line is appropriately increased, and the vehicle jam prediction region of interest (Cut-In RoI) is set. According to the relative position relationship between the adjacent vehicle target and the vehicle jam prediction region of interest (Cut-InRoI), it can be divided into two position states: ① When the adjacent vehicle target is across the edge of the region; ② When the adjacent vehicle target is in the region. The latter is relatively simple, and the target state descriptor is set to [2, 'follow']. The former state mostly appears at the two moments of the adjacent vehicle cutting in and cutting out. Under this condition, it can be distinguished according to the historical state descriptor {'keep', 'cut_in', 'follow'}.

[0095] If the target state descriptor is [count<Threshold, 'keep'], it is considered that the target adjacent vehicle is in the lane keeping state at the previous moment and has a potential trend from lane keeping to jamming behavior. At the same time, in order to reduce the false detection rate of the jamming behavior warning logic of the adjacent vehicle, the jamming warning prediction time can be appropriately delayed, and only the target that has been tracked for a long enough time to jam is considered. Specifically, only the points of the adjacent vehicle detection bounding box (Bounding Box) far from the jamming prediction region of interest (Cut-In RoI) are located outside the region, and at the same time, when the number of times the bounding box key points in the jamming prediction region of interest (Cut-In RoI) are detected exceeds the threshold, the target state descriptor is [count=count+1, 'keep'] and keeps updating until count=Threshold, and the target state descriptor is updated to [count=Threshold, 'cut_in']. In this project, α=0.4 is set, and the threshold of the number of jamming times detected is Threshold=3. At this time, the jam-in behavior of the adjacent vehicle in the current frame changes from action = 'keep' to action = 'cut_in', and the adjacent vehicle detection bounding box (Bounding Box) and the jam-in prediction region of interest (Cut-In RoI) become red warnings.

[0096] If the target state descriptor of the current frame is [count=Threshold, 'cut_in'], it can be considered that the target is continuing the jam-in behavior of the previous frame, so the target state descriptor is kept unchanged as [count=Threshold, 'cut_in']. The jam-in behavior of the adjacent vehicle in the current frame changes from action='cut_in', and the adjacent vehicle detection bounding box (BoundingBox) and the jam-in prediction region of interest (Cut-In RoI) continue to remain red, indicating that the adjacent vehicle is jamming.

[0097] If the target state descriptor of the current frame is [count=Threshold, 'cut_in'], but at this time the adjacent vehicle detection bounding box (Bounding Box) completely enters the Cut-In RoI, it can be considered that the target has completed the cutting-in behavior, so the target state descriptor is updated to [count=Threshold, 'follow'], the cutting-in behavior of the adjacent vehicle in the current frame changes from action 'keep', the adjacent vehicle detection bounding box (Bounding Box) and the Cut-In RoI become green, indicating that the cutting-in behavior of the adjacent vehicle has been completed and the current frame is in a safe state.

[0098] If the target state descriptor of the current frame is [count=Threshold,′folow′], and the adjacent vehicle detection bounding box (Bounding Box) is across the edge of the lane-cutting prediction region of interest (Cut-In RoI), it can be considered that the target has completed the lane-cutting behavior and is leaving the current lane, so the target state descriptor is updated to [count=Threshold,′keep′], the lane-cutting behavior of the adjacent vehicle in the current frame changes from action=′keep′, the adjacent vehicle detection bounding box (BoundingBox) and the lane-cutting prediction region of interest (Cut-In RoI) continue to remain green, and the current frame is in a safe state.

[0099] In any case, if the adjacent vehicle detection bounding box (Bounding Box) completely leaves the range of the cut-in prediction region of interest (Cut-In RoI), the target state descriptor is immediately updated to [count=0, 'keep'], and the above steps are repeated in the next frame. Using the above algorithm flow, there is an additional 3-frame delay for the cut-in situation. Since the FPS of the input stream is 30, this delay is equivalent to 0.1s; under real conditions, human drivers usually need at least 0.7s to brake when encountering an unforeseen cut-in situation. Therefore, even if there is a time delay from detection to sending a cut-in warning, it is not expected to cause serious problems, which meets the actual technical requirements.

[0100] Through testing, it can be known that the network proposed in the present invention can successfully predict the behavior of the adjacent vehicle to cut in and issue a warning, and can also cancel the warning after the cutting in is completed. It can successfully distinguish in the above two similar scenarios.

[0101] In summary, the present invention proposes a method for predicting the near-field vehicle cutting in behavior based on image input, which greatly improves the performance of real-time behavior prediction of near-field vehicles under the premise of limited computing power of the mobile terminal. It can be easily deployed in the existing intelligent driving system, so that the intelligent driving car can respond to the near-field vehicle cutting in behavior in a timely manner, thereby improving safety during driving.

[0102] The above embodiments are merely examples and do not limit the scope of the present invention. These embodiments can also be implemented in various other ways, and various omissions, substitutions, and changes can be made without departing from the technical concept of the present invention.

Claims

1. A near-field vehicle jamming behavior prediction method based on image input, characterized in that: The method includes: (1) Collect image sequence information based on forward panoramic images in real structured road scenes, and annotate the position and behavior information of vehicle targets in the image sequence through manual methods; (2) Construct a near-field vehicle detection and tracking model suitable for near-field vehicle detection and tracking on structured roads; (3) Construct a lane detection network and corresponding loss function suitable for lane detection on structured roads; (4) Based on the vehicle ID and the bounding box position data of the corresponding target obtained by the near-field vehicle detection and tracking model established in step (2), and the lane line obtained by the lane line detection network established in step (3), the relative position deviation between the target and the lane line is obtained, and according to the formulation of the prior rules, the prediction result of the near-field vehicle's lane-cutting behavior is obtained; The loss function of the lane detection network is expressed as L total : L total =L cls +L seg +ηL lane L cls is the multi-classification loss, L seg is the segmentation loss, L lane is the lane structure loss, η is a hyperparameter; The multi-classification loss L cls It is expressed as: Among them, L CE (·) represents the cross entropy loss function, P i，j，： represents the prediction results of all (w+1) lane line units for the i-th lane line and the j-th lateral anchor box, T i，j，： represents the true distribution of all (w+1) lane line units for the i-th lane line and the j-th lateral anchor box, c i，j，： Indicates P i，j，： With T i，j，： , C and h represent the number of lane line classes and the number of lane longitudinal anchor points respectively, and γ and α are hyper parameters; The lane structure loss L lane It is expressed as: THE lane =L sim +λL shp Among them, L sim is the similarity loss, L shp is the shape loss, λ is the hyperparameter representing the loss weight, P i,j,k represents the predicted probability of the i-th lane line at position (j, k), and w is the number of division units in each row.

2. The method for predicting near-field vehicle jamming behavior based on image input according to claim 1, characterized in that: Step (1) specifically includes: (11) Calibrate the internal and external parameters of the camera, where the external parameters include the rotation matrix R and the translation vector T, and the internal parameters include the internal parameter matrix K and the camera distortion coefficient; (12) Using a data collection vehicle equipped with a camera to collect video data in a real road scene, and recording the category of vehicle targets in the image during collection; (13) Use annotation tools to annotate the collected video data. The annotation methods include vehicle target tracking ID annotation, vehicle target category annotation, target object bounding box annotation, key frame annotation of the start of vehicle lane-cutting, vehicle crossing the midpoint of the lane line, and vehicle completion of lane-cutting behavior, and vehicle lane-cutting behavior category annotation. The annotation content must at least include the position of the near-field vehicle, key frames, and lane-cutting behavior category information.

3. The method for predicting near-field vehicle jamming behavior based on image input according to claim 1, characterized in that: Step (2) specifically includes: (21) Construct a near-field vehicle target detection network based on the improved Yolov5, input the input video slices as image time series into the near-field vehicle target detection network, and perform feature extraction and feature encoding on the input image information after multi-layer convolution and downsampling operations to obtain a multi-dimensional feature tensor that divides the image; (22) Construct a classification network and use non-maximum suppression operations to finally obtain the location information and classification confidence information of each target, including the classification probability and positioning probability of the object; (23) A near-field vehicle target tracking network based on the improved Deep-SORT is constructed. The target object bounding box information and classification information obtained by target detection are used as input. Multiple objects in the video are simultaneously located and tracked, and their ID and trajectory information are recorded. In particular, the transformation of object ID is reduced under occlusion conditions. The tracking ID, target category, and target object bounding box information of the target vehicle are output.

4. The method for predicting near-field vehicle jamming behavior based on image input according to claim 1, characterized in that: Step (3) specifically includes: (31) Construct a lane feature extraction backbone network based on a convolutional neural network and output features based on a shallow residual connection network; (32) Construct a lane line semantic segmentation network. During network training, multi-scale features are upsampled to the same scale and subjected to transposed convolution. The semantic segmentation loss is calculated to enhance the visual feature extraction capability of the backbone network. Finally, an enhanced lane line detection backbone network based on residual connection is obtained. (33) The features extracted by the backbone network are used to calculate candidate points through a classifier on a global scale according to the a priori specified longitudinal candidate anchor frame of the image, and finally the lane line position node of the lane where the ego vehicle is located is obtained; (34) Construct the loss function of the lane detection network, including multi-classification loss, segmentation loss and lane structuring loss.

5. The method for predicting near-field vehicle jamming behavior based on image input according to claim 1, characterized in that: Step (4) The steps of training the network and formulating prior rules are as follows: (41) Preprocessing the collected image sequence, including: randomly flipping the image horizontally, cropping it, and uniformly scaling it to a fixed size, and flipping, cropping, and scaling the annotated data accordingly, and then normalizing the obtained image by channel; (42) Using the lane line position nodes of the lane where the ego vehicle is located obtained by lane line detection, a lane line fitting model of the lane where the ego vehicle is located is obtained by using a highly robust regression model; (43) Establishing an area of ​​interest for the jamming behavior based on the lane line model of the lane where the ego vehicle is located, and calculating the position deviation between the target vehicle boundary box information and the area of ​​interest for the jamming behavior, and establishing the expected number of jamming behaviors and the vehicle state symbol dictionary for each target based on the target vehicle tracking ID; (44) By setting the expected number of lane-cutting behaviors and combining the vehicle status symbol to determine the specific behavior of the target vehicle, the parameters of the established lane-cutting behavior region of interest are updated, and the ideal network parameters are obtained after iteration.

6. The method for predicting near-field vehicle jamming behavior based on image input according to claim 5, characterized in that: The lane line fitting model of the lane where the ego vehicle is located is a linear model, and the left and right lanes are fitted according to the lane line prediction position nodes respectively.

7. The method for predicting near-field vehicle jamming behavior based on image input according to claim 5, characterized in that: The key value in the dictionary of expected number of lane-cutting behaviors and vehicle status symbols for each target is the target tracking ID, and the value is the expected number of times and the vehicle status symbol.

Images