DeepSort algorithm-based cart right turning must-stop detection method

Abstract

The invention discloses a cart right turn must stop detection method based on a DeepSort algorithm, and belongs to the technical field of computer vision target detection and tracking, and the method comprises the following steps: inputting a video stream, preprocessing the video stream, and inputting the preprocessed video frame into an improved YOLOV7 neural network for preliminary target detection; performing target tracking on the detected suspicious target through a DeepSort algorithm; and judging whether the suspicious target does not stop when turning right or not according to the change of the pixel gray value of the suspicious target, thereby realizing the detection and identification of the large vehicle which must stop when turning right in a complex traffic environment. According to the method, the improved YOLOV7 neural network and the DeepSort algorithm are applied, the peccancy detection accuracy is improved, the peccancy detection speed is increased, the method is applied to the problems of vehicle overlapping, reflection and the like in the complex traffic environment, and intelligent detection and recognition of the cart turning right and stopping in the complex environment are achieved.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision target detection and tracking, and in particular relates to a method for detecting a vehicle that must stop when turning right based on a DeepSort algorithm. Background Art

[0002] With the continuous progress of society, more and more families own cars, and traffic problems are becoming more and more prominent. Therefore, target detection and tracking are widely used in traffic roads. Vehicle detection and vehicle model recognition play a very important role in alleviating traffic problems. When large buses, large trucks and other large vehicles turn right, their long bodies form an inner wheel difference, which leads to blind spots in the driver's vision. In the visual blind spot, non-motor vehicles or pedestrians going straight in the same direction are easily rolled under the vehicle, causing inner wheel difference accidents. In order to prevent such accidents from happening, the transportation department has introduced traffic rules that require right turns to stop. However, there are still traffic problems where large vehicles violate the rules when turning right at intersections, posing a threat to people's lives. In order to avoid major traffic accidents caused by large vehicles turning right continuously, large vehicle detection systems based on computer vision have attracted people's attention.

[0003] At present, among the traditional vehicle detection technologies in computer vision, some technologies complete vehicle detection through the symmetry and edge characteristics in the image; some technologies complete vehicle detection through the HOG characteristics or Harr-ilke characteristics of the vehicle; and the vehicle detection algorithm based on deep learning proposes a neural network model with self-learning target features to address the defects of traditional vehicle detection that requires manual feature extraction and sliding window traversal, which has brought the accuracy and real-time performance of vehicle detection to a new level. At the same time, various target detections based on YOLO rectification and combination have emerged, and have achieved good results in accuracy and detection speed for large-scale target detection. However, when applied to traffic scenes with complex real backgrounds, existing problems have been exposed one after another, such as too many misjudgments, low frame rates, target overlap, and partial detection frame loss.

[0004] The existing patent with application number CN202210335536.3 proposes a multi-target detection and tracking method based on improved YOLO-V5s. By combining the improved YOLO-V5s target detection model with the DeepSort tracking algorithm, the backbone feature extraction network in the YOLO-V5s algorithm is improved, and the ordinary convolution is replaced with a lightweight convolution module Ghost Module. In addition, a convolutional attention mechanism CBAM is added to the last layer of the backbone feature extraction network to achieve real-time tracking of vehicles; however, the detection accuracy of YOLO-V5s used in this patent for vehicles in complex traffic scenarios needs to be improved.

[0005] The existing patent with application number CN202310175297.4 proposes an infrared vehicle detection method based on an improved YOLOv7 algorithm. The method first constructs an infrared vehicle dataset, then replaces the backbone feature extraction network of the YOLOv7 algorithm with a new backbone feature extraction network Conv31, constructs a new network model Conv31-YOLOv7, and then trains the new network model. Finally, the model is used for infrared vehicle detection; this patent increases the network depth by increasing the number of convolutional layers in YOLOv7, which improves the detection accuracy to a certain extent, but the detection precision and accuracy need to be improved when detecting overlapping targets.

[0006] In summary, the current computer vision large vehicle detection methods still have some problems, such as difficulty in tracking overlapping targets and target frame loss. Therefore, it is necessary to propose a large vehicle right turn must stop detection method based on the DeepSort algorithm. Summary of the invention

[0007] Purpose of the invention: In order to overcome the deficiencies in the prior art, the present invention provides a method for detecting a large vehicle that must stop when turning right based on the DeepSort algorithm. The method aims to improve detection accuracy, increase detection speed, and be applied to complex scenarios or scenarios where the target direction changes suddenly.

[0008] Technical solution: To solve the above technical problems, a method for detecting a vehicle turning right and stopping based on a DeepSort algorithm of the present invention comprises the following steps:

[0009] S1, obtain the video stream of the target area;

[0010] S2, preprocessing the video stream of the target area and extracting the corresponding video frames;

[0011] S3, input the extracted video frames into the improved YOLOV7 neural network;

[0012] S4, performs preliminary target detection on the extracted video frames by applying the improved YOLOV7 neural network. If a suspicious target is detected, the location information of all detected suspicious targets is retained and stored.

[0013] Furthermore, a residual structure is added between the feature extraction modules ELAN of the improved YOLOV7 neural network.

[0014] Furthermore, the improved YOLOV7 neural network applied in S4 uses a channel-coordinate attention module for preliminary target detection. The channel-coordinate attention module includes a channel attention module and a coordinate attention module, which specifically includes the following steps:

[0015] S41, inputting the feature map F of the extracted video frame into the channel-coordinate attention module, the size of the feature map F is C×H×W, where C is the number of feature channels of the image, H is the height of the image, and W is the width of the image;

[0016] S42, obtaining the channel attention weight M of each feature channel of the feature map F of the extracted video frame through the channel attention module c (F), M c The size of (F) is C×1×1, M c The calculation formula of (F) is as follows:

[0017] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) Formula (1)

[0018] In the above formula (1), AvgPool represents the average pooling extraction of features from the feature map F of the video frame, MaxPool represents the maximum pooling extraction of features from the feature map F of the video frame, MLP represents entering the shared multi-layer perceptron network for processing, and σ represents the Sigmoid activation function;

[0019] S43, the channel coordinate attention weight M of each feature channel c (F) is multiplied by the feature map F of the original extracted video frame to obtain the weighted feature map F′, the size of F′ is C×H×W, and the calculation formula is:

[0020]

[0021] In the above formula (2), Indicates element-by-element multiplication by channel;

[0022] S44, input the weighted feature map F′ into the coordinate attention module, and assume that the feature map of the input feature map F′ in each feature channel is represented as F′c(i, j), where i is the width direction, j is the height direction, and c is the cth feature channel. Perform global average pooling on the input feature map F′c(i, j) in both the width and height directions to obtain a sub-input feature map in the width direction. and

[0023] Sub-input feature map in height direction Its expression is as follows:

[0024]

[0025] In the above formula (3), The size is C×1×W, The size is C×H×1;

[0026] S45, input the sub-feature map in the width direction And the sub-input feature map in the height direction After concatenation and convolution and batch normalization, the feature map F1 ([z h ,z w ]), F1([z h , z w ]) has the original feature channel dimension C / r, where the convolution operation is performed through a convolution module with a shared convolution kernel of 1×1;

[0027] S46, the feature map F1 ([z h , z w ]) is processed to obtain the intermediate feature map f, the size of the intermediate feature map f is 1×(W+H)×C / r, and the calculation formula is as follows:

[0028] f=σ(F1([z h , z w ])) Formula (4)

[0029] In the above formula (4), σ represents the Sigmoid activation function;

[0030] S47, the intermediate feature map f includes a sub-intermediate feature map f in the height direction h and the sub-intermediate feature map f in the width direction w , and the sub-intermediate feature map f in the height direction is respectively convolved with a convolution kernel of 1×1 h and the sub-intermediate feature map f in the width direction w Perform convolution operations to obtain the height feature map components F h (f h ) and width feature map component F w (f w ), height feature map component F h (f h ) and width feature map component F w (f w ) has C channels;

[0031] S48, the above-mentioned height feature map components F are respectively activated by the Sigmoid activation function. h (f h ) and width feature map component F w (f w ) is processed to obtain the attention weight g in the height direction h and the attention weight g in the width direction w , the calculation formula is as follows:

[0032]

[0033] In the above formula (5), σ represents the Sigmoid activation function;

[0034] S49, using the above height direction attention weight g h and the attention weight g in the width direction w For the feature map F′ c (i, j) is weightedly calculated to obtain a feature map yc(i, j) with attention weights in the width and height directions. The feature map yc(i, j) is a refined feature map, and the calculation formula is as follows:

[0035]

[0036] In the above formula (6), and According to the formula, the size of yc(i, j) is C×H×W;

[0037] S410, outputting the refined feature map yc(i, j) obtained in the above step, wherein the content in the video frame corresponding to the refined feature map is the suspicious target.

[0038] Furthermore, the above detection method also includes the following steps:

[0039] S5, preliminary target detection is performed on the extracted video frames through the improved YOLOV7 neural network to determine whether suspicious targets are detected.

[0040] If a suspicious target is detected, the location information of all detected suspicious targets is retained and stored and the process proceeds to S6, otherwise the process proceeds to S9;

[0041] S6, obtain the location information of the suspicious target, and perform Kalman prior prediction on the suspicious target to obtain the Kalman filter of the suspicious target.

[0042] Wave prediction value;

[0043] S7, track the suspicious target, determine whether the current frame of the suspicious target matches the target detected in the previous frame of the current frame, if so, update the Kalman filter parameters and keep the coordinates of the particle of the suspicious target; otherwise, treat the suspicious target as

[0044] It is a new suspicious target and returns to S5;

[0045] S8, the tracked suspicious target is judged to turn right, and the coordinates of the particle of the suspicious target and the particle coordinates of the previous frame of the current frame are used to judge whether the suspicious target is turning right. If it is turning right, the suspicious target is used as the detected optimal target information, otherwise it is returned.

[0046] Return to S4;

[0047] S9, if no suspicious target is detected in S5, detect whether there is a suspicious target in the previous frame of the current frame. If so, use the Kalman filter method to fill the current frame target position to obtain the optimal target information, otherwise return to S2 to continue extraction

[0048] Video frames;

[0049] S10, mapping the obtained optimal target information to the original video frame, and judging whether it is a violation of traffic rules according to the optimal target information, if it is a violation of traffic rules,

[0050] Then the optimal target information is saved and then S11 is entered; otherwise, S11 is entered directly;

[0051] S11, determine whether the optimal target information is the last frame, if so, end; otherwise return to S2 to continue extracting video frames for detection until the end.

[0052] Furthermore, S7 uses an improved DeepSort algorithm to track the suspicious target. The target tracking process includes a process of cascading matching the target detected in the current frame of the suspicious target and the previous frame of the current frame and a process of IoU matching the target detected in the current frame of the suspicious target and the previous frame of the current frame.

[0053] Furthermore, the process of cascading matching the current frame of the suspicious target and the target detected in the previous frame of the current frame specifically includes the following steps:

[0054] First, the features of the suspicious target are extracted through the re-identification network;

[0055] Then, motion association matching is performed on the current frame of the suspicious target and the target detected in the previous frame of the current frame. The current frame is set as the target frame, and the previous frame of the current frame is set as the detection frame. The Mahalanobis distance d(1)(i, j) between the detection frame and the target frame is calculated. The calculation formula of the Mahalanobis distance d(1)(i, j) is:

[0056]

[0057] In the above formula (7), d j Represents the coordinates of the jth detection box, y i represents the coordinates of the i-th target box, Represents the covariance matrix between the target box and the average detection box;

[0058] Next, determine whether the current frame of the suspicious target and the target detected in the previous frame of the current frame have motion correlation matching. If d (1)If (i, j) is greater than the motion association threshold, the motion association matching of the target detected in the current frame of the suspicious target and the previous frame of the current frame is successful and proceeds to the next step, otherwise the motion association matching fails, and the target detected in the current frame of the suspicious target and the previous frame of the current frame continues to be matched by IoU;

[0059] Next, the appearance correlation matching is performed on the current frame of the suspicious target and the target detected in the previous frame of the current frame, and the cosine distance d(2)(i, j) between the current frame and the previous frame of the current frame is calculated. 2 The calculation formula of )(i, j) is:

[0060]

[0061] In the above formula (8), r j Represents each detection box d j The extracted feature vector, and ||r j ||=1,r j T Represents r j The transpose of Represents each target box r in the Kth frame of the video j The extracted feature vector, K represents the trajectory within the video frame, R k represents a vector library, and The vector library retains the most recent L k The feature vector of each track k in the frame;

[0062] Next, determine whether the appearance of the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame is associated with each other. If d (2) If (i, j) is less than the appearance association threshold, the appearance association matching of the target detected in the current frame of the suspicious target and the previous frame of the current frame is successful and the process goes to S76; otherwise, the appearance association matching fails and the target detected in the current frame of the suspicious target and the previous frame of the current frame continue to be matched by IoU;

[0063] Next, the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and matched to calculate the Mahalanobis distance d (1) (i, j) and the cosine distance d (2) (i, j) is the correlation matrix C(i, j), and the calculation formula of the correlation matrix C(i, j) is:

[0064] C(i, j) = λd (1) (i, j)+(1-λ)d (2) (i, j) Formula (9)

[0065] In the above formula (9), d (1)(i, j) represents the Mahalanobis distance between i and j, which is calculated by formula (7), d (2) (i, j) represents the cosine distance between i and j, calculated by formula (8), and λ is a hyperparameter for adjusting the association effect;

[0066] Finally, determine whether the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and associated to match. If C(i, j) is less than the combined association threshold, the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and associated to match successfully, that is, the cascade matching of the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame is successful, and the Kalman filter parameters are updated and the coordinates of the particle of the suspicious target are retained and stored and enter S8. Otherwise, the combined association matching fails, and the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame continue to be matched by IoU.

[0067] Furthermore, the process of performing IoU matching between the current frame of the suspicious target and the target detected in the previous frame of the current frame specifically includes the following steps:

[0068] First, let the current frame be the target frame, let the previous frame of the current frame be the detection frame, and calculate the position coordinates of the intersection of the target frame and the detection frame. The calculation formula is:

[0069]

[0070] In the above formula (10), max() is the maximum value function, min() is the minimum value function, (x a1 ,y a1 ) is the coordinate value of the upper left corner of the target frame a, (x a2 ,y a2 ) is the coordinate value of the lower right corner of the target frame a, (x b1 ,y b1 ) is the coordinate value of the upper left corner of the detection box b, (x b2 ,y b2 ) is the coordinate value of the lower right corner of the detection box b;

[0071] The (x1, y1) obtained by the above formula (10) is the coordinate value of the upper left corner of the intersection of the upper target frame and the detection frame, and the (x2, y2) is the coordinate value of the lower right corner of the intersection of the target frame and the detection frame;

[0072] Then, the area of ​​the intersection between the target frame and the detection frame is calculated. The formula for the area of ​​the intersection is:

[0073] intersection=max(x2-x1+1.0,0)×max(y2-y1+1.0,0) Formula (11)

[0074] In the above formula (11), intersection is the area of ​​the intersection, max() is the maximum value function, (x1, y1) and (x2, y2) are both calculated by the above formula (10), x1 is the x-axis coordinate value of the upper left corner coordinate of the intersection of the target frame and the detection frame, x2 is the x-axis coordinate value of the lower right corner coordinate of the intersection of the target frame and the detection frame, y1 is the y-axis coordinate value of the upper left corner coordinate of the intersection of the target frame and the detection frame, and y2 is the y-axis coordinate value of the lower right corner coordinate of the intersection of the target frame and the detection frame;

[0075] Next, calculate the union area of ​​the target box and the detection box. The formula for the union area is:

[0076] union=S A +S B -intersection formula(12)

[0077] S A =(x a2 -x a1 +1.0)×(y a2 -y a1 +1.0) Formula (13)

[0078] S B =(x b2 -x b1 +1.0)×(y b2 -y b1 +1.0) Formula (14)

[0079] In the above formula (12), intersection is calculated by formula (11), S A Calculated by formula (13), S A Represents the area of ​​the target box, S B Calculated by formula (14), S B Indicates the area of ​​the detection box;

[0080] Next, calculate the IOU value of the target box and the detection box. The IOU calculation formula is:

[0081]

[0082] In the above formula (15), intersection is calculated by formula (11), and union is calculated by formula (12);

[0083] Finally, a threshold judgment is performed on the IOU of the target frame and the detection frame. If the IOU of the target frame and the detection frame is greater than the threshold, the target frame matches the detection frame, that is, the current frame matches the previous frame of the current frame, and the Kalman filter parameters are updated, and the coordinates of the particle of the suspicious target are retained and stored and enter S8, otherwise return to S6.

[0084] Furthermore, step S8 determines whether the tracked suspicious target should turn right, and specifically includes the following steps:

[0085] First, let the rectangular coordinate system XOY, the forward direction of the optimal suspicious target is the y-axis, and the left direction of the optimal suspicious target is the x-axis direction;

[0086] Then, let the coordinates of the center of mass of the best suspicious target in the current frame be (x, y), and the coordinates of the center of mass of the previous frame of the current frame be (x′, y′). The moving direction of the center of mass of the best suspicious target is used to determine whether the best suspicious target moves to the right front. The calculation formula is:

[0087]

[0088] In the above formula (16), dx is the movement of the optimal suspicious target mass center on the x-axis, and dy is the movement of the optimal suspicious target mass center on the y-axis; if dx>0, dy>0, the optimal suspicious target moves to the right front;

[0089] Finally, when dx>0 and dy>0, determine whether the value of dx / dy is greater than the set counter threshold. If so, start counting. When the number of consecutive counts is greater than the set counter threshold, the optimal suspicious target is judged to be a right-turning vehicle, and the optimal suspicious target is used as the detected optimal target information; otherwise, the optimal suspicious target is not judged to be a right-turning vehicle and returns to S4.

[0090] Furthermore, step S9 performs frame complementation on the target position of the current frame by using a Kalman filter method, specifically comprising the following steps:

[0091] First, assume the state equation and covariance equation, the calculation formula of the state equation is:

[0092]

[0093] In the above formula (17), A is the state transfer matrix, is the prior prediction of the current frame, X k-1 is the optimal prediction of the previous frame, B is the control variable matrix, u k-1 represents the observation noise, w k-1 represents process noise;

[0094] The formula for calculating the covariance equation is:

[0095]

[0096] In the above formula (18), is the covariance of the prior prediction of the current frame, A is the state transfer matrix, P k-1 is the covariance of the optimal prediction at the previous moment, A T is the transposed matrix of the state transfer matrix A, Q is the system noise covariance matrix;

[0097] Then, set the system state X as follows:

[0098] X = [x, y, w, h, dx, dy] T Formula (19)

[0099] In the above formula (19), x is the x-axis coordinate value of the center point of the video frame, y is the y-axis coordinate value of the center point of the video frame, dx is the change in the x-axis coordinate value of the center point of the current frame compared to the x-axis coordinate value of the center point of the previous frame, dy is the change in the y-axis coordinate value of the center point of the current frame compared to the y-axis coordinate value of the center point of the previous frame, w is the width of the video frame, h is the height of the video frame, [ ] T is the transpose of the matrix;

[0100] Set the control variable matrix B to a zero matrix, set the process noise p(w) to N(0, Q), and set the state transfer matrix A to:

[0101]

[0102] Next, the state equation (17) and the covariance equation (18) are updated, and the calculation formula of the updated state equation (17) is:

[0103]

[0104] In the above formula (20), Calculated by formula (17), K k is the Kalman gain at time k, Z k It is the observation value of the box selection information predicted by the improved YOLOV7 network for the target of the previous frame, and H is the conversion matrix from the state vector to the measurement vector;

[0105] The updated calculation formula of the covariance equation (18) is:

[0106]

[0107] In the above formula (21), I is the unit matrix, K k is the Kalman gain at time k, H is the transformation matrix from state vector to measurement vector, Calculated by formula (18);

[0108] The above K k is the Kalman gain at time k, and the calculation formula is:

[0109]

[0110] In the above formula (22), Calculated by formula (15), H T is the transposed matrix of H, H is the transformation matrix from state vector to measurement vector, H T is the transposed matrix of H, R is the covariance matrix of the measurement noise;

[0111] Finally, the X obtained through the above steps is k As the optimal prediction state, the optimal prediction state is the frame selection information of the system's optimal detection,

[0112] And enter S10; if the target of the previous frame of the current frame is not detected, return to S2 to continue extracting the video.

[0113] Furthermore, step S10 determines whether there is a traffic violation based on the change in the grayscale value of the image pixel of the optimal target information. If the same pixel value is maintained for a time longer than a retention threshold after the grayscale value changes, there is no traffic violation; otherwise, it indicates a traffic violation.

[0114] Further, the threshold is maintained at 3 seconds.

[0115] Furthermore, the motion association threshold is 0.5.

[0116] Furthermore, the appearance correlation threshold is 9.5.

[0117] Beneficial effects: Compared with the prior art, the present invention has the following advantages:

[0118] 1. Compared with the prior art, the present invention inputs the video frames after data preprocessing into the improved YOLOV7 neural network, and performs preliminary target detection on the video frames after data preprocessing through the improved YOLOV7 neural network, thereby improving the detection accuracy and detection speed in the target detection process.

[0119] 2. Compared with the prior art, the present invention adds a residual structure between the feature extraction modules ELAN of the improved YOLOV7 network, improves the parameter utilization efficiency, calculation efficiency and feature extraction speed, reduces the amount of calculation, saves a lot of computing resources without destroying the original gradient path, and improves the detection speed.

[0120] 3. Compared with the prior art, the present invention performs preliminary target detection on video frames by applying the improved YOLOV7 neural network, so that the spatial dimensional features of small targets in the video frames can be preserved, thereby improving the recognition rate of small targets when detecting suspicious targets.

[0121] 4. Compared with the prior art, the present invention addresses the problem of target loss when the direction of movement changes. The Kalman filter method is used to make a priori predictions on suspicious targets and the DeepSort algorithm is used to continuously track targets, thereby solving the problem of target loss when the vehicle changes its direction of travel and improving the accuracy of detection.

[0122] 5. Compared with the prior art, the present invention adopts the DeepSort algorithm to track suspicious targets, and improves the detection accuracy under the overlapping phenomenon caused by vehicle congestion in traffic congestion through the process of cascading matching the current frame of the suspicious target and the previous frame of the current frame, solves the problem of target loss when the vehicle changes its direction of travel, and improves the detection accuracy.

[0123] 6. Compared with the prior art, the present invention adopts the IOU method to perform IOU matching on the current frame of the suspicious target and the previous frame of the current frame, thereby optimizing the problem of too many misjudgments in the matching process, reducing excessive detection misjudgments caused by a single judgment, and improving the accuracy of large vehicle violation target detection.

[0124] 7. Compared with the prior art, the present invention uses the moving direction of the tracked center of mass of the suspicious target as a judgment criterion, and judges whether the suspicious target turns right based on the relationship between the center of mass coordinates of the current frame of the suspicious target and the center of mass coordinates of the previous frame of the current frame. The counter is used to further judge, thereby improving the precision and accuracy of the right turn judgment.

[0125] 8. Compared with the prior art, the present invention adopts a Kalman filter to perform frame supplement prediction to address the phenomenon of sudden appearance or disappearance of large vehicle targets in complex traffic environments, thereby solving the problem of loss of suspicious target frames during large vehicle target detection, ensuring accurate detection of targets and improving the detection speed in complex traffic environments.

[0126] 9. Compared with the prior art, the present invention determines whether a suspicious target enters the right-turn area according to the change in the grayscale value of the image pixel of the optimal target information, and further determines whether the suspicious target entering the right-turn area violates the traffic rules by setting a time threshold, thereby improving the detection accuracy of sudden vehicle changes in this scenario, and realizing accurate identification of violations such as large vehicles turning right and stopping in complex traffic conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 It is a flow chart of the present invention.

[0128] Figure 2It is the residual structure diagram.

[0129] Figure 3 This is the structure diagram of the channel-coordinate attention module.

[0130] Figure 4 It is the flow chart of DeepSort algorithm.

[0131] Figure 5 is the centroid shift discriminant map. Figure 6 It is a process diagram of using the method steps of the present invention to judge and identify the target object based on the real scene in Example 8. DETAILED DESCRIPTION

[0132] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0133] Example 1

[0134] The invention provides a method for detecting a vehicle that must stop when turning right based on a DeepSort algorithm, referring to Figure 1 , the method comprises the following steps:

[0135] S1, obtain the video stream of the target area;

[0136] S2, preprocessing the video stream of the target area and extracting the corresponding video frames;

[0137] S3, input the extracted video frames into the improved YOLOV7 neural network;

[0138] S4, performs preliminary target detection on the extracted video frames by applying the improved YOLOV7 neural network. If a suspicious target is detected, the location information of all detected suspicious targets is retained and stored.

[0139] After obtaining the video stream through video surveillance or other means, the video stream of the target area is preprocessed, and the video frame region of interest determination method is applied to extract the corresponding video frame, that is, a fixed region of interest is directly selected from the acquired video stream for cropping, and the cropped video frame is used as the target area, and the video frame is input into the improved YOLOV7 neural network for detection. After the detection is completed, the region of interest detection information is mapped to the original video frame. In this embodiment, the video frame region of interest determination method is used to preprocess the video stream, so that the calculation amount of the subsequent network is reduced, and the network operation speed and the real-time performance of the detection are improved.

[0140] In this embodiment, the video frames after data preprocessing are input into the improved YOLOV7 neural network, and preliminary target detection is performed on the video frames after data preprocessing through the improved YOLOV7 neural network, and the location information of all detected suspicious targets is retained and stored for viewing and later management; the detection method of this embodiment improves the detection accuracy and detection speed in the target detection process.

[0141] Example 2

[0142] The present invention provides a method for detecting a vehicle that must stop when turning right based on a DeepSort algorithm. Based on Example 1, a video frame after data preprocessing is input into an improved YOLOV7 neural network. This embodiment uses an improved YOLOV7 neural network for preliminary target detection. A residual structure is added between the feature extraction modules ELAN (efficient aggregation network) of the improved YOLOV7 neural network. ELAN uses the shortest gradient path and the longest gradient path of each layer in the entire neural network to design a layer aggregation architecture with an effective path propagation path. This residual structure is added between the neural networks. The residual structure helps solve the gradient vanishing and gradient exploding problems in the training project. At present, when using the YOLOV7 network for target detection, the detection speed will decrease due to the addition of too many parameters to the convolutional layer. This embodiment increases the detection speed by adding a residual structure to the YOLOV7 neural network so that the YOLOV7 neural network can be trained using a multi-branch model at the same time.

[0143] Reference Figure 2 , the residual structure consists of two parts: skip connection and 1*1 convolution structure. The workflow of the residual structure is as follows:

[0144] First, the extracted video frames are input into the residual structure;

[0145] Then, for the branch that extracts features from the input video frame, a 1*1 convolution and normalization branch is connected to the input end of the residual structure and before the activation function and added to it; this step changes the number of channels and reduces the number of parameters of the convolution layer through the 1*1 convolution structure, thereby reducing the amount of calculation and more effectively extracting features from the input video frame;

[0146] Finally, a skip branch is connected between the input end of the residual structure and before the activation function to obtain the feature map of the input video frame. This step effectively improves the problem of performance degradation of the model as the network depth increases through skip connections.

[0147] This embodiment adds a residual structure between the feature extraction modules ELAN of the YOLOV7 neural network, improves the parameter usage efficiency, calculation efficiency and feature extraction speed, reduces the amount of calculation, saves a lot of computing resources without destroying the original gradient path, and improves the detection speed.

[0148] Example 3

[0149] The present invention provides a method for detecting large vehicles that must stop when turning right based on the DeepSort algorithm. It is based on the fact that conventional convolutional neural networks are currently commonly used to detect larger targets. In a traffic environment, the camera is fixed, and the targets in the extracted video frames change from large to small. When conventional convolutional neural networks are used for very small target image tasks, the detection performance will drop rapidly. This embodiment optimizes the detection of small targets, so that small feature points that are easily overlooked can be preserved, thereby detecting small suspicious targets in video frames.

[0150] Based on Example 2, the video frames after data preprocessing are input into the improved YOLOV7 neural network. This embodiment uses the improved YOLOV7 neural network for preliminary target detection. The improved YOLOV7 neural network uses a channel-coordinate attention module to make the YOLOV7 neural network lighter and able to focus on the spatial dimensional characteristics of small targets.

[0151] Reference Figure 3 The channel-coordinate attention module includes a channel attention module and a coordinate attention module, which specifically includes the following steps:

[0152] S41, the feature map F of the extracted video frame is input into the channel-coordinate attention module, and the size of the feature map F is C×H×W, where

[0153] C is the number of feature channels of the image, H is the height of the image, and W is the width of the image;

[0154] S42, obtaining the channel attention weight M of each feature channel of the feature map F of the extracted video frame through the channel attention module c (F), M c The size of (F) is C×1×1, M c The calculation formula of (F) is as follows:

[0155] M c (F)=σ(MLP(AvgPool(F))+MLP(MaxPool(F))) formula (1)

[0156] In the above formula (1), AvgPool represents the average pooling extraction of features from the feature map F of the video frame, MaxPool represents the maximum pooling extraction of features from the feature map F of the video frame, MLP represents entering the shared multi-layer perceptron network for processing, and σ represents the Sigmoid activation function;

[0157] The above formula (1) indicates that the feature map F is extracted by average pooling AvgPool and maximum pooling MaxPool respectively, and then the feature

[0158] Send it to a shared multi-layer perceptron (MLP) network to generate the final channel attention weight M c ;

[0159] S43, the channel coordinate attention weight M of each feature channel c (F) is multiplied by the feature map F of the original extracted video frame to obtain the weighted feature map F′, the size of F′ is C×H×W, and the calculation formula is:

[0160]

[0161] In the above formula (2) Indicates element-by-element multiplication by channel;

[0162] S44, input the weighted feature map F′ into the coordinate attention module, and let the feature map of the input feature map F′ on each feature channel be represented as F′ c (i, j), where i is the width direction and j is the height direction and c is the cth feature channel, the input feature map F′ in both width and height directions c (i, j) perform global average pooling respectively to obtain the sub-input feature map in the width direction And the sub-input feature map in the height direction Its expression is as follows:

[0163]

[0164] In the above formula (3), The size is C×1×W, The size is C×Hx1;

[0165] S45, input the sub-feature map in the width direction And the sub-input feature map in the height direction After concatenation and convolution and batch normalization, the feature map F1 ([z h , z w ]), F1([z h , z w]) is the original C / r in feature channel dimension, where the convolution operation is performed through a convolution module with a shared convolution kernel of 1×1; where r refers to the ratio of channel compression, and the dimension is reduced to the original C / r through a 1×1 convolution module, which reduces the amount of computation and improves computational efficiency;

[0166] Preferably, the best value of r is obtained by experiment. In this embodiment, r=16;

[0167] S46, the feature map F1 ([z h , z w ]) is processed to obtain the intermediate feature map f, the size of the intermediate feature map f is 1×(W+H)×C / r, and the calculation formula is as follows:

[0168] f=σ(F1([z h , z w ])) Formula (4)

[0169] In the above formula (4), σ represents the Sigmoid activation function;

[0170] S47, the intermediate feature map f includes a sub-intermediate feature map f in the height direction h and the sub-intermediate feature map f in the width direction w , and the sub-intermediate feature map f in the height direction is respectively convolved with a convolution kernel of 1×1 h and the sub-intermediate feature map f in the width direction w Perform convolution operations to obtain the height feature map components F h (f h ) and width feature map component F w (f w ), height feature map component F h (f h ) and width feature map component F w (f w ) has C channels;

[0171] S48, the above-mentioned height feature map components F are respectively activated by the Sigmoid activation function. h (f h ) and width feature map component F w (f w ) is processed to obtain the attention weight g in the height direction h and the attention weight g in the width direction w , the calculation formula is as follows:

[0172]

[0173] In the above formula (5), σ represents the Sigmoid activation function;

[0174] S49, using the above height direction attention weight g h and the attention weight g in the width direction w For the feature map F′ c (i, j) is weighted to obtain the feature map y with attention weights in the width and height directions c (i, j), the feature map y c (i, j) is a refined feature map, and the calculation formula is as follows:

[0175]

[0176] In the above formula (6), and According to the formula, the size of yc(i, j) is C×H×W;

[0177] S410, outputting the refined feature map yc(i, j) obtained in the above step, wherein the content in the video frame corresponding to the refined feature map is the suspicious target.

[0178] In this embodiment, a channel-coordinate attention module is added after each convolution of the backbone network in the YOLOV7 neural network to obtain the channel attention weight M along the spatial dimension and the coordinate dimension from the feature map F of the video frame. c and coordinate attention weight M s , which improves the effective flow of feature information in the neural network; the addition of the channel-coordinate attention module makes the YOLOV7 neural network more lightweight; by applying the improved YOLOV7 neural network to perform preliminary target detection on video frames, the small target spatial dimensional features in the video frames can be preserved, which improves the recognition rate of small targets when detecting suspicious targets.

[0179] Example 4

[0180] The present invention provides a method for detecting a vehicle that must stop when turning right based on a DeepSort algorithm. Based on Example 3, an improved YOLOV7 neural network is used to perform preliminary target detection on a video frame after data preprocessing, and after detecting a suspicious target, the detected suspicious target is tracked, and a violation judgment is made on the tracked suspicious target. Specifically, the following steps are included:

[0181] S5, preliminary target detection is performed on the extracted video frames through the improved YOLOV7 neural network to determine whether suspicious targets are detected.

[0182] If a suspicious target is detected, the location information of all detected suspicious targets is retained and stored and the process proceeds to S6, otherwise the process proceeds to S9;

[0183] S6, obtaining the location information of the suspicious target, and performing Kalman prior prediction on the suspicious target to obtain the Kalman

[0184] Mann filter prediction value;

[0185] S7, tracking the suspicious target, determining whether the current frame of the suspicious target matches the target detected in the previous frame of the current frame, if so, updating the Kalman filter parameters and retaining the coordinates of the particle of the suspicious target; otherwise,

[0186] The suspicious target is taken as a new suspicious target and returns to S5;

[0187] S8, judging whether the tracked suspicious target is turning right, judging whether the suspicious target is turning right by the particle coordinates of the suspicious target and the particle coordinates of the previous frame of the current frame, if it is turning right, the suspicious target is used as the detected optimal target information,

[0188] Otherwise, return to S4;

[0189] S9, if no suspicious target is detected in S5, detect whether there is a suspicious target in the previous frame of the current frame. If so, use the Kalman filter method to fill the current frame target position to obtain the optimal target information, otherwise return to S2 to continue extraction

[0190] Video frames;

[0191] S10, mapping the obtained optimal target information to the original video frame, and judging whether it is a violation of traffic rules according to the optimal target information, if it is a violation of traffic rules,

[0192] Then the optimal target information is saved and then S11 is entered; otherwise, S11 is entered directly;

[0193] S11, determine whether the optimal target information is the last frame, if so, end; otherwise return to S2 to continue extracting video frames for detection until the end.

[0194] In this embodiment, the detected suspicious targets are tracked, and the tracked optimal target information is used to determine the violation of turning right without stopping, thereby achieving real-time detection of large vehicles turning right illegally and improving detection accuracy. At the same time, this embodiment uses Kalman filtering to supplement frames for undetected suspicious targets, thereby improving detection accuracy when the target is lost during the detection process.

[0195] Example 5

[0196] The present invention provides a method for detecting a vehicle that must stop when turning right based on a DeepSort algorithm. Based on Example 4, in step S6, after preliminary target detection is performed by an improved YOLOV7 neural network and the position information of a suspicious target is obtained, a priori prediction of the suspicious target is performed by a Kalman filter method. The Kalman filter predicts the state of the suspicious target in the current frame based on the state of the previous frame of the current frame of the suspicious target, and obtains the predicted state of the current frame of the suspicious target. Specifically:

[0197] First, the previous frame of the current frame of the suspicious target is input into the Kalman filter;

[0198] Then, let the predicted state be the target box, and let the state of the detected suspicious target be represented by the spatial coordinates. Among them, (u, v) is the coordinate of the center point of the detection box, γ is the aspect ratio, and h is the height of the detection box. are the corresponding velocities of (u, v, γ, h) in image coordinates;

[0199] Next, the Kalman filter of the uniform speed model and the linear observation model is used for prediction update. Its observation variables are (u, v, γ, h). The position of the suspicious target in the current frame is predicted based on the state of the previous frame of the suspicious target in the current frame, and the Kalman filter prediction value of the suspicious target is obtained, that is, the position of the suspicious target in the current frame;

[0200] Finally, the target trajectory generated by the previous iteration (i.e., the target detected in the previous frame of the current frame) is predicted by the Kalman filter, and the mean (average value) and covariance (covariance matrix) of the eight values ​​in the spatial coordinates of the current frame of the suspicious target input in this round are calculated.

[0201] In step S7, the DeepSort algorithm is used to perform target tracking and matching on the current frame of the predicted suspicious target and the previous frame of the current frame; the target tracking process includes a process of cascading matching between the current frame of the suspicious target and the previous frame of the current frame and a process of IoU matching between the current frame of the suspicious target and the previous frame of the current frame, specifically:

[0202] First, the Kalman filter prediction value state of the current frame of the suspicious target predicted by the above step S6 remains unchanged, including two states: Confirmed (confirmed match) and Unconfirmed (unconfirmed match);

[0203] Then, the current frame of the predicted suspicious target and the previous frame of the current frame are matched for the first time, and the Confirmed state part of the current frame of the above suspicious target (i.e., the Kalman filter prediction value) and the suspicious target detected by the current target detector (i.e., the previous frame of the current frame) are sent to the cascade for cascade matching, and three states of results are generated, namely Unmatched Tracks, Unmatched Detections, and Matched Tracks.

[0204] Next, the current frame of the predicted suspicious target and the previous frame of the current frame are matched for the second time. Since the detection in the above steps may be missed, the Unconfirmed state part of the current frame of the above suspicious target (i.e., the Kalman filter prediction value) is merged with the Unmatched Tracks and Unmatched Detections. The current frame of the predicted suspicious target and the previous frame of the current frame are matched for the second time using IOU Match, generating three state results, namely Unmatched Tracks, Unmatched Detections, and Matched Tracks.

[0205] Next, process the invalid objects and build new tracks for the above Unmatched Detections. For the above Unmatched Tracks, the unmatched Tracks include Confirmed and Unconfirmed. If the unmatched Tracks is Unconfirmed, delete it. If the unmatched Tracks is Confirmed, set age. If its age exceeds the threshold, delete it, that is, it is considered to have been removed from the surveillance camera, otherwise it is retained.

[0206] Finally, the Matched Tracks obtained from the first match, the Matched Tracks obtained from the second match, and the Confirmed Tracks retained in the invalid object that have not exceeded the age are merged together as the output of this round, which is also the input of the next iteration.

[0207] In order to solve the problem of target loss when the direction of movement is changed, this embodiment applies the Kalman filter method to make a priori predictions on suspicious targets and the DeepSort algorithm to continuously track targets, thus solving the problem of target loss when the vehicle changes its direction and improving the accuracy of detection.

[0208] Example 6

[0209] The present invention provides a method for detecting a vehicle that must stop when turning right based on a DeepSort algorithm. Based on Example 5, when a current frame of a predicted suspicious target and a previous frame of the current frame are cascade matched, that is, a process of cascade matching the current frame of the suspicious target and the previous frame of the current frame, that is, matching track (target frame) and detection (detection frame), the process of cascade matching the current frame and the previous frame of the current frame specifically includes the following steps:

[0210] First, the re-identification network extracts features of the suspicious target. By adding the appearance information extracted from the re-identification model, the IOU score and the spatial clues of the ReID feature are combined to make the tracking index more robust. The re-identification network extracts and saves the appearance features of the object after each detection and tracking. In each subsequent step, the similarity calculation between the appearance features of the object detected in the current frame and the previously stored appearance features is performed to avoid missed detections.

[0211] Then, motion association matching is performed on the current frame of the suspicious target and the previous frame of the current frame. The current frame is set as the target frame, and the previous frame of the current frame is set as the detection frame. The Mahalanobis distance d between the detection frame and the target frame is calculated. (1) (i, j), the association degree is expressed by calculating the Mahalanobis distance between the detection box and the target box. The calculation formula of the Mahalanobis distance is:

[0212]

[0213] In the above formula (7), d (1) (i, j) represents the Mahalanobis distance between i and j, d j Represents the coordinates of the jth detection box, y i represents the coordinates of the i-th target box, Represents the covariance matrix between the target box and the average detection box;

[0214] Next, determine whether the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame have motion correlation matching. (1)If (i, j) is greater than the motion association threshold, the motion association matching of the target detected in the current frame of the suspicious target and the previous frame of the current frame is successful and proceeds to the next step, otherwise the motion association matching fails, and the target detected in the current frame of the suspicious target and the previous frame of the current frame continues to be matched by IoU;

[0215] Preferably, the motion association threshold is 0.5. (1) If (i, j) is greater than 0.5, the target motion association matching between the current frame of the suspicious target and the target detected in the previous frame of the current frame is successful and proceeds to the next step, otherwise the motion association matching fails, and the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame continue to be matched by IoU;

[0216] Next, the appearance correlation matching is performed on the target detected in the current frame and the previous frame of the suspicious target, and the cosine distance d between the current frame and the previous frame of the current frame is calculated. (2) (i, j), that is, the apparent features and dimensional differences of the target box and the detection box are used to judge the correlation, and the cosine distance d (2) The calculation formula for (i, j) is:

[0217]

[0218] In the above formula (8), d (2) (i, j) the cosine distance between i and j, r j Represents each detection box d j The extracted feature vector, and ||r j ||=1, Represents r j The transpose of Represents each target box r in the Kth frame of the video j The extracted feature vector, K represents the trajectory within the video frame, R k represents a vector library, and The vector library retains the most recent L k The feature vector of each track k in the frame;

[0219] Next, determine whether the appearance of the target detected in the current frame and the previous frame of the current frame is associated with each other. If d (2) (i, j) is less than the appearance association threshold, then the appearance association matching of the target detected in the current frame of the suspicious target and the previous frame of the current frame is successful and enters S76, otherwise the appearance association matching fails, and the target detected in the current frame of the suspicious target and the previous frame of the current frame continue to perform IoU matching;

[0220] Preferably, the appearance correlation threshold is set to 9.5. (2)When (i, j) is less than 9.5, it means that the detection frame and the target frame are matched and associated, that is, the cascade matching of the current frame and the previous frame of the current frame is successful, otherwise the cascade matching of the current frame and the previous frame of the current frame fails; in this embodiment, since the target objects around the vehicle may have a relatively fast relative motion and cover with the vehicle in a complex road environment, the Mahalanobis distance is prone to misjudgment in judging the association, so the cosine distance is introduced to perform the appearance association matching judgment, which further improves the accuracy of the judgment;

[0221] Next, the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and matched to calculate the Mahalanobis distance d (1) (i, j) and the cosine distance d (2) (i, j) is the correlation matrix C(i, j), and the calculation formula of the correlation matrix C(i, j) is:

[0222] C(i, j) = λd (1) (i, j)+(1-λ)d (2) (i, j) Formula (9)

[0223] In the above formula (9), d (1) (i, j) represents the Mahalanobis distance between i and j, which is calculated by formula (7), d (2) (i, j) represents the cosine distance between i and j, calculated by formula (8), and λ is a hyperparameter for adjusting the association effect;

[0224] Finally, determine whether the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and associated to match. If C(i, j) is less than the combined association threshold, the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and associated to match successfully, that is, the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame are cascade matched successfully, update the Kalman filter parameters, retain and store the coordinates of the particle of the suspicious target, and enter S8. Otherwise, the combined association matching fails, and the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame continue to be matched by IoU.

[0225] Preferably, the combined association threshold is set to 9.5. If C(i, j) is less than 9.5, the combined association match between the target detected in the current frame of the suspicious target and the previous frame of the current frame is successful, that is, the cascade match between the target detected in the current frame of the suspicious target and the previous frame of the current frame is successful, the Kalman filter parameters are updated, the particle coordinates of the suspicious target are retained and stored, and S8 is entered. Otherwise, the combined association match fails, and the target detected in the current frame of the suspicious target and the previous frame of the current frame continue to be matched by IoU.

[0226] This embodiment uses the DeepSort algorithm to track suspicious targets. Through the process of cascading matching the current frame of the suspicious target and the previous frame of the current frame, the detection accuracy under overlapping phenomena caused by crowded vehicles in traffic congestion is improved, the problem of target loss when the vehicle changes its direction of travel is solved, and the detection accuracy is improved.

[0227] Example 7

[0228] The present invention provides a method for detecting a vehicle that must stop when turning right based on an improved YOLOV7 neural network and a DeepSort algorithm. Based on Example 6, when S7 uses the DeepSort algorithm to track the suspicious target, if the cascade matching of the current frame of the suspicious target and the previous frame of the current frame fails, the current frame of the suspicious target and the previous frame of the current frame continue to be matched for the second time, that is, IoU matching, so as to reduce misjudgment and improve detection accuracy. The process of performing IoU matching of the current frame of the suspicious target and the previous frame of the current frame specifically includes the following steps:

[0229] First, let the current frame be the target frame, let the previous frame of the current frame be the detection frame, and calculate the position coordinates of the intersection of the target frame and the detection frame. The calculation formula is:

[0230]

[0231] In the above formula (10), max() is the maximum value function, min() is the minimum value function, (x a1 ,y a1 ) is the coordinate value of the upper left corner of the target frame a, (x a2 ,y a2 ) is the coordinate value of the lower right corner of the target frame a, (x b1 ,y b1 ) is the coordinate value of the upper left corner of the detection box b, (x b2 ,y b2 ) is the coordinate value of the lower right corner of the detection box b;

[0232] The (x1, y1) obtained by the above formula (10) is the coordinate value of the upper left corner of the intersection of the upper target frame and the detection frame, and the (x2, y2) is the coordinate value of the lower right corner of the intersection of the target frame and the detection frame;

[0233] Then, the area of ​​the intersection between the target frame and the detection frame is calculated. The formula for the area of ​​the intersection is:

[0234] intersection=max(x2-x1+1.0,0)×max(y2-y1+1.0,0) (11)

[0235] In the above formula (11), intersection is the area of ​​the intersection, max() is the maximum value function, (x1, y1) and (x2, y2) are both calculated by the above formula (10), x1 is the x-axis coordinate value of the upper left corner coordinate of the intersection of the target frame and the detection frame, x2 is the x-axis coordinate value of the lower right corner coordinate of the intersection of the target frame and the detection frame, y1 is the y-axis coordinate value of the upper left corner coordinate of the intersection of the target frame and the detection frame, and y2 is the y-axis coordinate value of the lower right corner coordinate of the intersection of the target frame and the detection frame;

[0236] Next, calculate the union area of ​​the target box and the detection box. The formula for the union area is:

[0237] union=S A +S B -intersection (12)

[0238] S A =(x a2 -x a1 +1.0)×(y a2 -y a1 +1.0) (13)

[0239] S B =(x b2 -x b1 +1.0)×(y b2 -y b1 +1.0) (14)

[0240] In the above formula (12), intersection is calculated by formula (11), S A Calculated by formula (13), S A Represents the area of ​​the target box, S B Calculated by formula (14), S B Indicates the area of ​​the detection box;

[0241] Next, calculate the IOU value of the target box and the detection box. The IOU calculation formula is:

[0242]

[0243] In the above formula (15), intersection is calculated by formula (11), and union is calculated by formula (12);

[0244] Finally, a threshold judgment is performed on the IOU of the target frame and the detection frame. If the IOU of the target frame and the detection frame is greater than the threshold, the target frame matches the detection frame, that is, the current frame matches the previous frame of the current frame, and the Kalman filter parameters are updated, and the coordinates of the particle of the suspicious target are retained and stored and enter S8, otherwise return to S6.

[0245] Preferably, the above-mentioned IOU threshold is set to be greater than 0. When the IOU of the target box and the detection box is greater than the threshold, it indicates that the current frame of the suspicious target is successfully matched with the previous frame of the current frame; when the IOU of the target box and the detection box is less than the threshold, it indicates that the current frame of the suspicious target is not successfully matched with the previous frame of the current frame again. At this time, the tracking and matching of the suspicious target is automatically abandoned, the suspicious target is taken as a new suspicious target and returned to S6.

[0246] In this embodiment, in the case where the cascade matching of the current frame of the suspicious target and the previous frame of the current frame fails, the IOU method is used to perform IOU matching on the current frame of the suspicious target and the previous frame of the current frame, thereby optimizing the problem of too many misjudgments in the matching process, reducing excessive detection misjudgments caused by a single judgment, and improving the accuracy of large vehicle violation target detection.

[0247] Example 8

[0248] The present invention provides a method for detecting a large vehicle that must stop when turning right based on an improved YOLOV7 neural network and a DeepSort algorithm. Based on Example 7, the suspicious target is tracked to obtain the particle coordinates of the suspicious target, and then the right turn behavior of the moving vehicle (i.e., the suspicious target) is determined. The moving direction of the center of mass of the suspicious target tracked in the previous step is used as a judgment criterion, and whether the suspicious target turns right is determined based on the relationship between the center of mass coordinates of the current frame of the suspicious target and the center of mass coordinates of the previous frame of the current frame.

[0249] Reference Figure 5 , the right turn judgment of the tracked suspicious target specifically includes the following steps:

[0250] First, let the rectangular coordinate system XOY, the forward direction of the optimal suspicious target is the y-axis, the left direction of the optimal suspicious target is the x-axis direction, the suspicious target is located in the second quadrant of the rectangular coordinate system, that is, the quadrant of x<0, y>0, the suspicious target moves along the positive direction of the y-axis, and the positive direction of the x-axis is the right turn direction;

[0251] Then, let the coordinates of the center of mass of the best suspicious target in the current frame be (x, y), and the coordinates of the center of mass of the previous frame of the current frame be (x′, y′). The moving direction of the center of mass of the best suspicious target is used to determine whether the best suspicious target moves to the right front. The calculation formula is:

[0252]

[0253] In the above formula (16), dx is the movement amount of the optimal suspicious target mass center on the x-axis, and dy is the movement amount of the optimal suspicious target mass center on the y-axis;

[0254] If dy>0, the suspicious target moves along the positive direction of the y axis;

[0255] If dy<0, the suspicious target is moving along the negative direction of the y-axis, that is, the suspicious target is reversing;

[0256] If dx>0, the suspicious target is moving along the positive direction of the x-axis and has a tendency to turn right;

[0257] If dx < 0, the suspicious target is moving in the negative direction of the x-axis and has a tendency to turn left;

[0258] Therefore, if dx>0, dy>0, the optimal suspicious target moves to the right front;

[0259] Finally, determine whether the value of dx / dy is greater than the set counter threshold when dx>0 and dy>0. If it is greater than the counter threshold, the counter starts counting. When the number of consecutive counts is greater than the set counting threshold, the optimal suspicious target is judged to be a right-turning vehicle, and the optimal suspicious target is used as the detected optimal target information; otherwise, the optimal suspicious target is not judged to be a right-turning vehicle, and returns to S4.

[0260] Preferably, the counter threshold is set to 5. When the value of dx / dy is greater than 5, the counter starts counting. Otherwise, the counter starts not counting, the optimal suspicious target is not determined to be a right-turning vehicle, and the process returns to S4.

[0261] The counting threshold is set to 5. After the counter starts counting, if the consecutive count times are greater than 5, that is, there are 5 consecutive frames of the best suspicious target turning right, the best suspicious target is judged to be a right-turning vehicle, and the best suspicious target is used as the detected best target information, and then it is judged whether it is a violation based on the best target information; otherwise, the suspicious target is not judged as a right-turning vehicle, and returns to S4 for preliminary target detection.

[0262] Reference Figure 6 ,Depend on Figure 6 (a) It can be seen that the large vehicle selected by the frame is the suspicious target detected and tracked. The forward direction of the suspicious target is the y-axis, and the left direction of the optimal suspicious target is the x-axis direction. At this time, the direction of the suspicious target is straight, that is, moving forward along the positive direction of the y-axis;

[0263] Depend on Figure 6 (b) It can be seen that after the suspicious target enters the right turn waiting area, according to the above formula 16, dx>0 and dy>0, the suspicious target has a tendency to move to the right front, and it can be determined that the target turn of the suspicious target is a right turn;

[0264] Afterwards Figure 6 (c) It can be seen that when the counter starts counting, since the number of consecutive right-turn frames of the suspicious target is greater than 5, it is judged as a non-stop illegal vehicle.

[0265] This embodiment uses the moving direction of the tracked center of mass of the suspicious target as a discrimination criterion, and judges whether the suspicious target turns right based on the relationship between the center of mass coordinates of the suspicious target in the current frame and the center of mass coordinates of the previous frame of the current frame. A counter is used to further judge, thereby improving the precision and accuracy of the right turn judgment.

[0266] Example 9

[0267] The present invention provides a method for detecting a large vehicle that must stop when turning right based on an improved YOLOV7 neural network and a DeepSort algorithm. Based on Example 8, preliminary target detection is performed by applying the improved YOLOV7 network. If no suspicious target is detected, or when it is detected that the target object is lost, it is detected whether there is a suspicious target in the content of the previous frame of the current frame. If so, the target position of the current frame is predicted and supplemented by Kalman filtering to obtain the optimal target information, that is, the previous frame of the current frame obtained by the improved YOLOV7 network is used to predict the value of the current frame, and the observed value is used to correct the current value to obtain the optimal result of the current frame for frame supplement.

[0268] S9 uses the Kalman filter method to fill in the target position of the current frame, which specifically includes the following steps:

[0269] First, assume the state equation and covariance equation. The calculation formula of the state equation is:

[0270]

[0271] In the above formula (17), A is the state transfer matrix, is the prior prediction of the current frame, X k-1 is the optimal prediction of the previous frame, B is the control variable matrix, u k-1 represents the observation noise, W k-1 represents process noise;

[0272] The calculation formula of the covariance equation is:

[0273]

[0274] In the above formula (18), is the covariance of the prior prediction of the current frame, A is the state transfer matrix, P k-1 is the covariance of the optimal prediction at the previous moment, A T is the transposed matrix of the state transfer matrix A, Q is the system noise covariance matrix;

[0275] Then, set the system state X as follows:

[0276] X = [x, y, w, h, dx, dy] T (19)

[0277] In the above formula (19), x is the x-axis coordinate value of the center point of the video frame, y is the y-axis coordinate value of the center point of the video frame, dx is the change in the x-axis coordinate value of the center point of the current frame compared to the x-axis coordinate value of the center point of the previous frame, dy is the change in the y-axis coordinate value of the center point of the current frame compared to the y-axis coordinate value of the center point of the previous frame, w is the width of the video frame, h is the height of the video frame, [ ] T is the transpose of the matrix;

[0278] Set the control variable matrix B to a zero matrix, set the process noise p(w) to N(0, Q), and set the state transfer matrix A to:

[0279]

[0280] Next, the state equation (17) and the covariance equation (18) are updated, and the calculation formula of the updated state equation (17) is:

[0281]

[0282] In the above formula (20), Calculated by formula (17), K k is the Kalman gain at time k, Z k It is the observation value of the box selection information predicted by the improved YOLOV7 network for the target of the previous frame, and H is the conversion matrix from the state vector to the measurement vector;

[0283] The updated calculation formula of the covariance equation (18) is:

[0284]

[0285] In the above formula (21), I is the unit matrix, K k is the Kalman gain at time k, H is the transformation matrix from state vector to measurement vector, Calculated by formula (18);

[0286] The above K k is the Kalman gain at time k, and the calculation formula is:

[0287]

[0288] In the above formula (22), Calculated by formula (15), H Tis the transposed matrix of H, H is the transformation matrix from state vector to measurement vector, H T is the transposed matrix of H, R is the covariance matrix of the measurement noise;

[0289] Finally, the X obtained through the above steps is k As the optimal prediction state, the optimal prediction state is the system optimal target information, and enters S10; if the previous frame target of the current frame is not detected, return to S2 to continue extracting the video.

[0290] This embodiment uses a Kalman filter to perform frame prediction for the sudden appearance or disappearance of large vehicles in complex traffic environments. Before detection, the Kalman filter is hyperparameter adjusted to increase the weight of the prior value in the calculation, where R (the covariance matrix of the measurement noise) is set larger and (State equation at time 0) is 0, the covariance is set to 1, the value of the current frame is predicted by the optimal result of the previous frame, and the value of the current frame is corrected by the observed value of the current frame. The optimal prediction state calculated by the Kalman filter is used as the optimal target information of the system to obtain the optimal result of the current frame. This solves the problem of suspicious target frame loss during large vehicle target detection, ensures accurate detection of targets, and improves detection speed in complex traffic environments.

[0291] Example 9

[0292] The present invention provides a method for detecting a vehicle that must stop when turning right based on an improved YOLOV7 neural network and a DeepSort algorithm. Based on Example 8, the detected optimal target information is mapped to the original video, and the optimal target information is judged as being in violation of traffic rules. Whether a violation of traffic rules is determined based on the change in the grayscale value of the image pixel of the optimal target information is determined. If the same pixel value is maintained for a time exceeding the retention threshold after the grayscale value changes, then there is no violation of traffic rules; otherwise, it indicates illegal driving. This embodiment detects whether a suspicious target vehicle is in violation of traffic rules by using the pixel grayscale value, and specifically includes the following steps:

[0293] First, determine the region of interest, which is the right turn waiting area, which can be obtained by selecting the initial frame of the video with the mouse;

[0294] Then, the pixel grayscale value of the image of the optimal target information in the region of interest is detected to determine whether the pixel grayscale value of the background pixel point in the time series has a large change. If there is a change of ±50%, it means that a suspicious target has entered the region; otherwise, it means that no suspicious target has entered the region;

[0295] Next, if a suspicious target enters the region of interest, the violation of the suspicious target in the region of interest is judged based on the time the same background pixel value remains after the gray value changes:

[0296] When the background pixel value changes from an initial stable value to a new instantaneous value, and the new instantaneous value is maintained for a time longer than the retention threshold, and then slowly returns to the initial stable value, it indicates that the suspicious target vehicle has stopped and is considered as a non-violation target;

[0297] When the background pixel value changes from an initial stable value to a new instantaneous value, and the new instantaneous value is maintained for a time that does not exceed the retention threshold, but quickly returns to the initial stable value, it indicates that the suspicious target vehicle has not stopped and is considered a traffic violation target;

[0298] Preferably, the holding threshold can be set according to the actual traffic conditions. In this embodiment, the holding threshold is set to 3 seconds. When the new instantaneous value is held for more than 3 seconds and then slowly returns to the initial stable value, it indicates that the suspicious target vehicle has stopped and is regarded as a non-violation target; when the new instantaneous value is held for less than 3 seconds and quickly returns to the initial stable value, it indicates that the suspicious target vehicle has not stopped and is regarded as a violation target;

[0299] Finally, the current frame of the violation target image detected through the above steps is saved for viewing and later management.

[0300] In this embodiment, whether a suspicious target enters the right-turn area is judged according to the change in the grayscale value of the image pixel of the optimal target information, and whether the suspicious target entering the right-turn area is in violation of traffic rules is further judged by setting a time threshold, thereby improving the detection accuracy of sudden vehicle changes in this scenario, and realizing accurate identification of violations such as large vehicles turning right and stopping in complex traffic conditions.

[0301] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A method for detecting a vehicle that must stop when turning right based on the DeepSort algorithm, characterized in that: The following steps are involved: S1, obtain the video stream of the target area; S2, preprocessing the video stream of the target area and extracting the corresponding video frames; S3, input the extracted video frames into the improved YOLOV7 neural network; S4, performs preliminary target detection on the extracted video frames by applying the improved YOLOV7 neural network. If a suspicious target is detected, the location information of all detected suspicious targets is retained and stored.

2. The method for detecting a vehicle turning right and stopping based on the DeepSort algorithm according to claim 1, characterized in that: A residual structure is added between the feature extraction modules ELAN of the improved YOLOV7 neural network.

3. The method for detecting a vehicle turning right and stopping based on the DeepSort algorithm according to claim 2 is characterized in that: The improved YOLOV7 neural network described in S4 uses a channel-coordinate attention module for preliminary target detection. The channel-coordinate attention module includes a channel attention module and a coordinate attention module, which specifically includes the following steps: S41, inputting the feature map F of the extracted video frame into the channel-coordinate attention module, the size of the feature map F is C×H×W, where C is the number of feature channels of the image, H is the height of the image, and W is the width of the image; S42, obtaining the channel attention weight M of each feature channel of the feature map F of the extracted video frame through the channel attention module c (F), M c The size of (F) is C×1×1, M c The calculation formula of (F) is as follows: M c (F) = o(MLP(AvgPool(F)) + MLP(MaxPool(F))) Formula (1) In the above formula (1), AvgPool represents the average pooling extraction of features from the feature map F of the video frame, MaxPool represents the maximum pooling extraction of features from the feature map F of the video frame, MLP represents entering the shared multi-layer perceptron network for processing, and σ represents the Sigmoid activation function; S43, the channel coordinate attention weight M of each feature channel mentioned above c (F) is multiplied by the feature map F of the original extracted video frame to obtain the weighted feature map F′, the size of F′ is C×H×W, and the calculation formula is: In the above formula (2), Indicates element-by-element multiplication by channel; S44, input the weighted feature map F′ into the coordinate attention module, and let the feature map of the input feature map F′ on each feature channel be represented as F′ c (i, j), where i is the width direction and j is the height direction and c is the cth feature channel, the input feature map F′ in both width and height directions c (i, j) perform global average pooling respectively to obtain the sub-input feature map in the width direction And the sub-input feature map in the height direction Its expression is as follows: In the above formula (3), The size is C×1×W, The size is C×H×1; S45, input the sub-feature map in the width direction And the sub-input feature map in the height direction Splice and perform convolution and batch normalization to obtain the feature map after batch normalization The feature channel dimension of is the original C / r, where the convolution operation is performed through a convolution module with a shared convolution kernel of 1×1; S46, the feature map F1 ([z h , z w ]) is processed to obtain the intermediate feature map f, the size of the intermediate feature map f is 1×(W+H)×C / r, and the calculation formula is as follows: f=σ(F1([z h ,z w ])) Formula (4) In the above formula (4), σ represents the Sigmoid activation function; S47, the intermediate feature map f includes a sub-intermediate feature map f in the height direction h and the sub-intermediate feature map f in the width direction w , and the sub-intermediate feature map f in the height direction is respectively convolved with a convolution kernel of 1×1 h and the sub-intermediate feature map f in the width direction w Perform convolution operations to obtain the height feature map components F h (f h ) and width feature map component F w (f w ), height feature map component F h (f h ) and width feature map component f w (f w ) has C channels; S48, the above-mentioned height feature map components F are respectively activated by Sigmoid activation function. h (f h ) and width feature map component F w (f w ) is processed to obtain the attention weight g in the height direction h and the attention weight g in the width direction w , the calculation formula is as follows: In the above formula (5), σ represents the Sigmoid activation function; S49, using the above height direction attention weight g h and the attention weight g in the width direction w For the feature map F′ c (i, j) is weighted to obtain the feature map y with attention weights in the width and height directions c (i, j), the feature map y c (i, j) is the refined feature map, The calculation formula is as follows: In the above formula (6), and Calculated by the formula, y c The size of (i,j) is C×H×W; S410, output the refined feature map y obtained in the above steps c (i, j), the content in the video frame corresponding to the refined feature map is the suspicious target.

4. The method for detecting a vehicle that must stop when turning right based on the DeepSort algorithm according to claim 3 is characterized in that: The following steps are also included: S5, preliminary target detection is performed on the extracted video frames through the improved YOLOV7 neural network to determine whether suspicious targets are detected. If a suspicious target is detected, the location information of all detected suspicious targets is retained and stored and the process proceeds to S6, otherwise the process proceeds to S9; S6, obtaining the location information of the suspicious target, and performing Kalman priori prediction on the suspicious target to obtain a Kalman filter prediction value of the suspicious target; S7, target tracking is performed on the suspicious target to determine whether the current frame of the suspicious target matches the target detected in the previous frame of the current frame. If they match, the Kalman filter parameters are updated and the coordinates of the particle of the suspicious target are retained and stored; otherwise, the suspicious target is taken as a new suspicious target and the process returns to S5; S8, judging whether the tracked suspicious target is turning right, judging whether the suspicious target is turning right by the particle coordinates of the suspicious target and the particle coordinates of the previous frame of the current frame, if it is turning right, the suspicious target is used as the detected optimal target information, Otherwise, return to S4; S9, if no suspicious target is detected in S5, detect whether there is a suspicious target in the previous frame of the current frame. If so, fill the current frame target position with a Kalman filter method to obtain the optimal target information, otherwise return to S2 to continue extracting video frames; S10, mapping the obtained optimal target information to the original video frame, and judging whether it is a violation of traffic rules according to the optimal target information, if it is a violation of traffic rules, saving the optimal target information and entering S11, otherwise directly entering S11; S11, determine whether the optimal target information is the last frame, if so, end; otherwise return to S2 to continue extracting video frames for detection until the end.

5. The method for detecting a vehicle that must stop when turning right based on the DeepSort algorithm according to claim 4 is characterized in that: The S7 uses an improved DeepSort algorithm to track the suspicious target. The target tracking process includes a process of cascading matching the target detected in the current frame of the suspicious target and the previous frame of the current frame and a process of IoU matching the target detected in the current frame of the suspicious target and the previous frame of the current frame.

6. The method for detecting a vehicle that must stop when turning right based on the DeepSort algorithm according to claim 5 is characterized in that: The process of cascading matching the current frame of the suspicious target and the target detected in the previous frame of the current frame specifically includes the following steps: First, the features of the suspicious target are extracted through the re-identification network; Then, motion association matching is performed on the current frame of the suspicious target and the target detected in the previous frame of the current frame. The current frame is set as the target frame, and the previous frame of the current frame is set as the detection frame. The Mahalanobis distance d between the detection frame and the target frame is calculated. (1) (i, j), Mahalanobis distance d (1) The calculation formula for (i, j) is: In the above formula (7), d j Represents the coordinates of the jth detection box, y i represents the coordinates of the i-th target box, Represents the covariance matrix between the target box and the average detection box; Next, determine whether the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame have motion correlation matching. (1) If (i, j) is greater than the motion association threshold, the motion association matching of the target detected in the current frame of the suspicious target and the previous frame of the current frame is successful and proceeds to the next step, otherwise the motion association matching fails, and the target detected in the current frame of the suspicious target and the previous frame of the current frame continues to be matched by IoU; Next, the appearance correlation matching is performed on the target detected in the current frame and the previous frame of the suspicious target, and the cosine distance d between the current frame and the previous frame of the current frame is calculated. (2) (i, j), cosine distance d (2) The calculation formula for (i, j) is: In the above formula (8), r j Represents each detection box d j The extracted feature vector, and ||r j ||=1, Represents r j The transpose of Represents each target box r in the Kth frame of the video j The extracted feature vector, K represents the trajectory within the video frame, R k represents a vector library, and The vector library retains the most recent L k The feature vector of each track k in the frame; Next, determine whether the appearance of the target detected in the current frame and the previous frame of the current frame is associated with each other. If d (2) (i, j) is less than the appearance association threshold, then the appearance association matching of the target detected in the current frame of the suspicious target and the previous frame of the current frame is successful and enters S76, otherwise the appearance association matching fails, and the target detected in the current frame of the suspicious target and the previous frame of the current frame continue to perform IoU matching; Next, the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and matched to calculate the Mahalanobis distance d (1) (i, j) and the cosine distance d (2) (i, j) is the correlation matrix C(i, j), and the calculation formula of the correlation matrix C(i, j) is: C(i,j) = λd (1) (,j) + (1 - λ)d (2) (i,j) formula (9) In the above formula (9), d (1) (i, j) represents the Mahalanobis distance between i and j, which is calculated by formula (7), d (2) (i, j) represents the cosine distance between i and j, calculated by formula (8), and λ is a hyperparameter for adjusting the association effect; Finally, determine whether the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and associated to match. If C(i, j) is less than the combined association threshold, the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame are combined and associated to match successfully, that is, the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame are cascade matched successfully, update the Kalman filter parameters, retain and store the coordinates of the particle of the suspicious target, and enter S8. Otherwise, the combined association matching fails, and the target detected in the current frame of the suspicious target and the target detected in the previous frame of the current frame continue to be matched by IoU.

7. The method for detecting a vehicle that must stop when turning right based on the DeepSort algorithm according to claim 6 is characterized in that: The process of performing IoU matching between the current frame of the suspicious target and the target detected in the previous frame of the current frame specifically includes the following steps: First, let the current frame be the target frame, let the previous frame of the current frame be the detection frame, and calculate the position coordinates of the intersection of the target frame and the detection frame. The calculation formula is: In the above formula (10), max( ) is the maximum value function, min( ) is the minimum value function, (x a1 ,y a1 ) is the coordinate value of the upper left corner of the target frame a, (x a2 ,y a2 ) is the coordinate value of the lower right corner of the target frame a, (x b1 ,y b1 ) is the coordinate value of the upper left corner of the detection box b, (x b2 ,y b2 ) is the coordinate value of the lower right corner of the detection box b; The (x1, y1) obtained by the above formula (10) is the coordinate value of the upper left corner of the intersection of the upper target frame and the detection frame, and the (x2, y2) is the coordinate value of the lower right corner of the intersection of the target frame and the detection frame; Then, the area of ​​the intersection between the target frame and the detection frame is calculated. The formula for the area of ​​the intersection is: intersection=max(x2-x1+1.0,0)×max(y2-y1+1.0,0) Formula (11) In the above formula (11), intersection is the area of ​​the intersection, max() is the maximum value function, (x1, y1) and (x2, y2) are both calculated by the above formula (10), x1 is the x-axis coordinate value of the upper left corner coordinate of the intersection of the target frame and the detection frame, x2 is the x-axis coordinate value of the lower right corner coordinate of the intersection of the target frame and the detection frame, y1 is the y-axis coordinate value of the upper left corner coordinate of the intersection of the target frame and the detection frame, and y2 is the y-axis coordinate value of the lower right corner coordinate of the intersection of the target frame and the detection frame; Next, calculate the union area of ​​the target box and the detection box. The formula for the union area is: union=S A +S B -intersection formula(12) S A = (x a2 -x a1 +1.0)×(y a2 -y a1 +1.0) Formula(13) S B = (x b2 -x b1 +1.0)×(y b2 -y b1 +1.0) Formula(14) In the above formula (12), intersection is calculated by formula (11), S A Calculated by formula (13), S A Represents the area of ​​the target box, S B Calculated by formula (14), S B Indicates the area of ​​the detection box; Next, calculate the IOU value of the target box and the detection box. The IOU calculation formula is: In the above formula (15), intersection is calculated by formula (11), and union is calculated by formula (12); Finally, a threshold judgment is performed on the IOU of the target frame and the detection frame. If the IOU of the target frame and the detection frame is greater than the threshold, the target frame matches the detection frame, that is, the current frame matches the previous frame of the current frame, and the Kalman filter parameters are updated, and the coordinates of the particle of the suspicious target are retained and stored and enter S8, otherwise return to S6.

8. The method for detecting a vehicle that must stop when turning right based on the DeepSort algorithm according to claim 7 is characterized in that: S8 determines the right turn of the tracked suspicious target. The following steps are involved: First, let the rectangular coordinate system XOY, the forward direction of the optimal suspicious target is the y-axis, and the left direction of the optimal suspicious target is the x-axis direction; Then, let the coordinates of the center of mass of the best suspicious target in the current frame be (x, y), and the coordinates of the center of mass of the previous frame of the current frame be (x′, y′). The moving direction of the center of mass of the best suspicious target is used to determine whether the best suspicious target moves to the right front. The calculation formula is: In the above formula (16), dx is the movement of the optimal suspicious target mass center on the x-axis, and dy is the movement of the optimal suspicious target mass center on the y-axis; if dx>0, dy>0, the optimal suspicious target moves to the right front; Finally, when dx>0 and dy>0, determine whether the value of dx / dy is greater than the set counter threshold. If so, start counting. When the number of consecutive counts is greater than the set counter threshold, the optimal suspicious target is judged to be a right-turning vehicle, and the optimal suspicious target is used as the detected optimal target information; otherwise, the optimal suspicious target is not judged to be a right-turning vehicle and returns to S4.

9. The method for detecting a vehicle that must stop when turning right based on the DeepSort algorithm according to claim 8, characterized in that: The S9 performs frame supplementation on the target position of the current frame by using the Kalman filter method, which specifically includes the following steps: First, assume the state equation and covariance equation, the calculation formula of the state equation is: In the above formula (17), A is the state transfer matrix, is the prior prediction of the current frame, X k-1 is the optimal prediction of the previous frame, B is the control variable matrix, u k-1 represents the observation noise, w k-1 represents process noise; The calculation formula of the covariance equation is: In the above formula (18), is the covariance of the prior prediction of the current frame, A is the state transfer matrix, P k-1 is the covariance of the optimal prediction at the previous moment, A T is the transposed matrix of the state transfer matrix A, Q is the system noise covariance matrix; Then, set the system state X as follows: X = [x, y, w, h, dx, dy] T Formula (19) In the above formula (19), x is the x-axis coordinate value of the center point of the video frame, y is the y-axis coordinate value of the center point of the video frame, dx is the change in the x-axis coordinate value of the center point of the current frame compared to the x-axis coordinate value of the center point of the previous frame, dy is the change in the y-axis coordinate value of the center point of the current frame compared to the y-axis coordinate value of the center point of the previous frame, w is the width of the video frame, h is the height of the video frame, [ ] T is the transpose of the matrix; Set the control variable matrix B to a zero matrix, set the process noise p(w) to N(0, Q), and set the state transfer matrix A to: Next, the state equation (17) and the covariance equation (18) are updated, and the calculation formula of the updated state equation (17) is: In the above formula (20), Calculated by formula (17), K k is the Kalman gain at time k, Z k is the observation value of the box selection information predicted by the improved YOLOV7 network for the target in the previous frame, and H is the conversion matrix from the state vector to the measurement vector; The updated calculation formula of the covariance equation (18) is: In the above formula (21), I is the unit matrix, K k is the Kalman gain at time k, H is the transformation matrix from state vector to measurement vector, Calculated by formula (18); The above K k is the Kalman gain at time k, and the calculation formula is: In the above formula (22), Calculated by formula (15), H T is the transposed matrix of H, H is the transformation matrix from state vector to measurement vector, H T is the transposed matrix of H, R is the covariance matrix of the measurement noise; Finally, the X obtained through the above steps is k As the optimal prediction state, the optimal prediction state is the frame selection information of the system's optimal detection, and enters S10; if the previous frame target of the current frame is not detected, return to S2 to continue extracting the video.

10. The method for detecting a vehicle that must stop when turning right based on the DeepSort algorithm according to claim 9, characterized in that: The S10 determines whether there is a traffic violation based on the change in the grayscale value of the image pixel of the optimal target information. If the same pixel value is maintained for a time longer than a retention threshold after the grayscale value changes, there is no traffic violation; otherwise, it indicates a traffic violation.

Images