A real-time vehicle detection method

Abstract

The present invention relates to the technical field of computer vision and deep learning, and provides a real-time vehicle detection method, comprising the following steps: obtaining video data from an on-board camera, and cutting it into images frame by frame; adjusting the image size to a unified format and inputting it into a trained improved Swin‑Transformer network model, outputting the position coordinates and category confidence of a vehicle identification frame; intercepting the vehicle in the image according to the position coordinates of the vehicle identification frame, and obtaining a single vehicle image in the vehicle identification frame; graying the single vehicle image, and obtaining a grayscale image of the single vehicle image; inputting the grayscale image of the single vehicle image into a trained incremental width learning system, and obtaining a specific category of the single vehicle image; mapping the vehicle identification frame and the specific category of the single vehicle image back to the video data frame by frame, and obtaining a real-time vehicle detection result. The technical solution of the present invention is suitable for being directly deployed on an on-board device, and has good accuracy and less training time.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and deep learning technology, and in particular to a real-time vehicle detection method. Background Art

[0002] As an important part of the transportation system, highways play a vital role in promoting economic growth and social progress. In the past few years, the continuous increase in vehicles on the road has led to a high degree of lane occupancy. As a result, the surge in traffic density has become one of the main concerns of the society. The introduction of intelligent transportation systems has improved highway efficiency and reduced traffic accidents. The system aims to ease traffic pressure through close cooperation between individuals, vehicles and highway infrastructure. The Internet of Vehicles is a key application in the intelligent transportation system. This interconnected system is essential for optimizing traffic management and improving overall traffic efficiency. Internet of Vehicles tasks, including optimal path planning, intelligent traffic light coordination, and autonomous driving, require first capturing real-time traffic flow information and then using this information to make the next decision. Therefore, accurately capturing traffic flow and vehicle information is crucial to ensure the correctness of the final decision and the reliability of the intelligent system.

[0003] At present, convolutional neural networks are still the main method for vehicle detection in mainstream applications. Convolutional neural networks can obtain spatial feature information from images and perform inductive reasoning, which produces positive results for visual tasks. However, the static and local characteristics of convolution operations limit the network performance. Faced with dynamic road conditions, the adaptability of traditional detection algorithms is limited, resulting in unsatisfactory performance under complex road conditions. In addition, convolutional neural networks usually require a large number of training parameters, resources, and samples. Vehicle detection algorithms based on convolutional neural networks usually need to be trained and verified in a cloud environment with sufficient resources and computing power. However, data transmission between the cloud and on-board edge devices is susceptible to bandwidth congestion and response delays. These factors reduce the efficiency of vehicle detection algorithms and increase the probability of traffic accidents.

[0004] To meet these challenges, a lightweight vehicle detection network is needed with the advantages of simple network structure, higher performance, less computing resource consumption, short training time, and fast update speed. This network can be directly deployed on edge devices with limited resources to achieve effective and real-time vehicle detection in various dynamic traffic. Summary of the invention

[0005] The purpose of the present invention is to provide a real-time vehicle detection method to solve the problem that the real-time vehicle detection method in the prior art requires large storage resources and is not suitable for vehicle-mounted equipment.

[0006] In order to achieve the above-mentioned object, the present invention provides a real-time vehicle detection method, comprising the following steps: acquiring video data from a vehicle-mounted camera and cutting it into images frame by frame; adjusting the image size to a unified format and inputting the image into a trained improved Swin-Transformer network model, performing feature extraction and target detection on the image using the trained improved Swin-Transformer network model, generating a vehicle identification frame and a category confidence, and outputting the position coordinates and category confidence of the vehicle identification frame; intercepting the vehicle in the image according to the XY position coordinates of the upper left corner and the lower right corner of the vehicle identification frame, A single vehicle image in a vehicle identification frame is obtained; the single vehicle image is grayed, and each RBG three-channel pixel in the image is converted into a grayscale image by the formula Gray = 0.299 × R + 0.587 × G + 0.114 × B to obtain a grayscale image of the single vehicle image; the grayscale image of the single vehicle image is input into a trained incremental width learning system, and the trained incremental width learning system is used to perform feature extraction and classification prediction on the grayscale image of the single vehicle image to obtain a specific category of the single vehicle image; the vehicle identification frame and the specific category of the single vehicle image are mapped back to the video data frame by frame to obtain real-time vehicle detection results.

[0007] Among them, the category confidence includes the foreground category confidence and the background category confidence. The category confidence threshold is set. When the category confidence exceeds the threshold, it is the foreground, and when the category confidence is lower than the threshold, it is the background.

[0008] Furthermore, the method for obtaining a trained improved Swin-Transformer network model includes: based on the existing Swin-Transformer network model, simplifying the number of stacking sub-network blocks, using a 2-4-2 stacking strategy, and changing the Softmax activation function in the multi-head attention mechanism to a Relu function to obtain an improved Swin-Transformer network model; establishing a training set and a test set, wherein the images in the training set and the test set include images taken by an on-board camera at different times and on different road types; using the training set to train the improved Swin-Transformer network model, using the test set to test the improved Swin-Transformer network model, and using the loss function to adjust the parameters of the network structure of the improved Swin-Transformer network model to obtain a trained improved Swin-Transformer network model.

[0009] Furthermore, the method of adjusting the parameters of the network structure of the improved Swin-Transformer network model by using the loss function specifically includes: the loss function includes a rectangular box regression loss function and a confidence loss function; the rectangular box regression loss function is:

[0010]

[0011] Among them, L reg is the rectangular box regression loss value; R is:

[0012]

[0013] Among them, t i Contains x ,t y ,t h ,t w Four parameters, which are calculated as follows:

[0014]

[0015]

[0016]

[0017]

[0018] Among them, x, y, w, h are the center coordinates and width and height of the prediction box, x a ,y a ,h a ,w a are the center coordinates and width and height of the anchor box; where, Include Four parameters, which are calculated as follows:

[0019]

[0020]

[0021]

[0022]

[0023] Among them, x * ,y * ,w * ,h * is the center coordinate and width and height of the real annotation box, x a ,y a ,h a ,w a are the center coordinates and width and height of the anchor box; the category confidence loss function is:

[0024]

[0025] L d is the category confidence loss value, m is the number of samples, yi is the true value of the i-th sample, and y′ is the predicted value of the i-th sample;

[0026] The predicted rectangular box regression loss value and category confidence loss value are calculated and added using the rectangular box regression loss function and the category confidence loss function to obtain the final loss value. After obtaining the final loss value, the improved Swin-Transformer network model parameters are updated through the back gradient propagation algorithm. When the final loss value tends to infinitesimal or meets the specified training rounds, the predicted rectangular box and the real box coordinates and the predicted category confidence tend to be the same, and the model parameter adjustment is completed, thereby obtaining the trained improved Swin-Transformer network model.

[0027] Furthermore, the image is input into the trained improved Swin-Transformer network model, and the trained improved Swin-Transformer network model is used to extract features and detect targets on the image, generate a vehicle identification frame and a category confidence, and output the position coordinates of the vehicle identification frame and the category confidence. The method specifically includes: adjusting and scaling the input image through a preprocessing process, and uniformly adjusting the size of the image to N i ×M i ×J, and get the scaled image; where N i Represents the length of the image, M i Represents the width of the image, J represents the number of RGB channels of the image; i ×M i The image of size ×J is input into the convolutional layer of the improved Swin-Transformer network, and is extracted into a size of N through the convolutional layer. p ×M p ×J p The feature map of size N p ×M p ×J p Flatten the feature map to get a size of N p M p ×J p The feature vector of size N is used by using the multi-head attention mechanism in Swin-Transformer p M p ×J p The feature vector of size N is obtained by feature fusion c M c ×J c The feature vector of c M c ×J cThe feature vector of is passed through two fully connected layers in series, and then passed through two fully connected layers in parallel to obtain the coordinates of the predicted box and the predicted category confidence respectively.

[0028] Furthermore, the input image is scaled and the image size is uniformly adjusted to N i ×M i ×J, the method for obtaining the scaled image specifically includes: the original size of the input image is N×M×J, where N represents the original length size of the image, M represents the original width size of the image, and J represents the number of channels of the image; calculating the adaptive scaling ratio, the original size of the image is N×M, and the original target size of the adaptive scaling is N 1 ×M 1 , scaling factor Z 1 , Z 2 Calculated by the following formula;

[0029]

[0030]

[0031] When Z 1 <Z 2 When selecting Z 1 As the final scaling factor Z; when Z 1 ≥Z 2 When selecting Z 2 As the final scaling factor Z; calculate the size after adaptive scaling, multiply the length and width of the original image by the scaling factor Z to get the length and width N of the image to be filled 2 and M 2 ; When N 2 =N 1 When selecting M 2 As the position to be filled; when N 2 ≠N 1 When N 2 As the position to be filled; calculate the black edge filling value, with M 2 Taking the position to be filled as an example, calculate the height that needs to be filled, then calculate the array a that needs to be filled at both ends of the picture, and fill the array a to both ends of the picture to get the final scaled image; the size of the image after scaling N i and M i The calculation method is as follows:

[0032] N i =N 1 =N 2

[0033]

[0034] Furthermore, the method for obtaining a trained incremental width learning system specifically includes: establishing a training set and a test set, the images in the training set and the test set include grayscale images of multiple single vehicles of different categories; using the training set to train the width learning system, using the test set to test the width learning, and using the incremental learning strategy to optimize the width learning system to obtain a trained incremental width learning system.

[0035] Furthermore, the method for training the width learning system using the training set specifically includes: taking the grayscale image X of a single vehicle image in the training set as the input of the width learning system; wherein X is an R*E matrix; R represents the number of samples of the input data, and E represents the dimension of each sample number; performing feature mapping on X in the width learning system, and obtaining all mapping features Z s ; Among them, the nonlinear feature mapping Z of the input data X is performed using the following formula:

[0036]

[0037] in, and are all randomly generated matrices; s different Zs are combined to obtain s groups of feature nodes, Z s =[Z 1 ,Z 2 ,Z 3 ,…,Z s ], represents all mapping features extracted from input X through network mapping;

[0038] According to Z s , a randomly generated matrix and Calculate the corresponding enhanced node H l ; Use the following formula to calculate the enhanced node H;

[0039]

[0040] and It is still the randomly generated matrix. We can also choose different ξ here. The H we get from the above formula is j Also in matrix form, by combining different H together we can get the final set of enhanced nodes H l =[H 1 ,H 2 ,H 3 ,…,H l ], which represents all the enhanced nodes we get from the mapping feature; the mapping feature Z s and enhanced node H lMerge them together into a matrix, and send the merged matrix and to the width learning system to calculate the output Y; Y is an R*P matrix, where P represents the number of sample categories; map feature Z s and enhanced node H l The final output Y can be obtained by multiplying the combined matrix with the weights of the network connection;

[0041] The expression of output Y is:

[0042] Y=[Z s |H l ]W=PW (9)

[0043] Where P is Z s and H l The combined matrix P = [Z s |H l ], W is the weight of the connection network, which can be expressed as:

[0044]

[0045] Among them, γ→0, I is the unit matrix, lim γ→0 (γI+P T P) -1 P T Calculate the pseudo-inverse of the matrix P, multiply the pseudo-inverse of P by the output matrix Y to get the network connection weight W, and substitute the calculated W into (9) to get the final output Y of the network. According to the category confidence of all vehicle categories contained in Y, select the vehicle category with the highest category confidence as the final specific vehicle category.

[0046] Furthermore, the method of optimizing the width learning system using the incremental learning strategy specifically includes: adding new enhancement nodes, randomly generating the matrix again and Calculate the corresponding newly added enhanced node H j+k , where H j+k The calculation formula is:

[0047]

[0048] Among them, j is the number of original enhancement nodes of the width system, and new enhancement nodes are continuously added to finally obtain H l+1 =[H j+1 ,H j+2 ,H j+3 ,…,H j+k ], k = 1, 2, 3, ..., which represents a group of newly added enhanced nodes; the newly added enhanced nodes H l+1 and mapping feature Z s and enhanced node Hl Merge to get a new matrix, which is expressed as [Z s |H l ,H l+1 ], where the newly added set of enhanced nodes is merged into the original merge matrix; the newly merged matrix is ​​sent to and updated in the width learning system, and the output Y′ is calculated, and the expression of Y′ is as follows:

[0049] Y′=[Z s |H l ,H l+1 ]W′=P′W′(12)

[0050] Where P′ is the mapping feature Z s , Enhanced node H l And the newly added enhanced node H l+1 The combined matrix P = [Z s |H l ,H l+1 ], W′ is the updated weight of the connection network; since the model adds new nodes, the weight W needs to be re-optimized, and the weight of the newly added enhanced node is trained to obtain the updated weight W′. The expression of weight W′ is as follows:

[0051]

[0052] Where D is the newly added enhanced node and [Z s |H l ] by multiplying the pseudo-inverse of B T It is expressed as:

[0053]

[0054] Among them A m For [Z s |H l ], C is expressed as:

[0055] C=H l+1 -A m D(15)

[0056] The updated weight W′ can be calculated by equations (13), (14) and (15). Substituting the weight into equation (12) can obtain the updated final output Y′. Then, the vehicle category with the highest category confidence in Y′ is selected as the specific vehicle category of the final output.

[0057] Furthermore, the method of mapping the vehicle identification frame and the specific category of the single vehicle image back to the video data frame by frame to obtain the real-time vehicle detection result specifically includes: marking the image using the vehicle identification frame and the specific category of the single vehicle image to obtain the marked image; merging the marked images frame by frame, synthesizing the video and returning it to the display system.

[0058] By applying the technical solution of the present invention, the existing Swin-Transformer network model is improved, and the Swin-Transformer network model is combined with a width learning system optimized by an incremental learning strategy, which can realize real-time vehicle detection and is suitable for direct deployment on vehicle-mounted equipment, with good accuracy and less training time.

[0059] Specifically, in this application, the improved Swin-Transformer network shortens the depth of the network and changes the nonlinear function in attention based on the existing Swin-Transformer network. The advantages of the improved Swin-Transformer network are: increasing the reasoning speed of the model, so that it can be applied to vehicle-mounted edge devices with limited resources. In addition, replacing the Softmax activation function in the attention mechanism with the nonlinear function Relu can improve the network's feature extraction ability and budget speed, maximizing the reasoning speed of the model while maintaining high accuracy.

[0060] In this application, the advantages of combining the Swin-Transformer network and BLS are: the Swin-Transformer network can extract the area containing vehicles in the image better and faster, while the BLS algorithm can update the weights at one time, with faster training and reasoning speeds. The combination of the two can not only improve the detection effect and quality of the overall algorithm, but also greatly improve the training and reasoning speed of the network model. It can not only meet the resource load of the vehicle-mounted edge device, but also accurately predict road vehicles.

[0061] Based on the implementations provided in the above aspects, this application can also be further combined to provide more implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0063] Figure 1A flow chart of a real-time vehicle detection method according to an optional embodiment of the present invention;

[0064] Figure 2 The overall framework diagram of the real-time vehicle detection method of an optional embodiment of the present invention, (1) is the data collection and preprocessing process, (2) is the model training process after the data is processed, and (3) is the verification process on the Huawei Atlas 200 edge device after the model training is completed;

[0065] Figure 3 The Swin-Transformer network model diagram of the real-time vehicle detection method of the optional embodiment of the present invention. Among them, (a) shows the Swin-Transformer convolution of the scaled input image to obtain a size N p ×M p ×J p Feature diagram, (b) is the Swin-Transformer network model structure, (c) is the structure of the Swin-Transformer Block in the Swin-Transformer network model;

[0066] Figure 4 A network model diagram of a width learning system of a real-time vehicle detection method in an optional embodiment of the present invention;

[0067] Figure 5 A result diagram of cutting vehicle-mounted videos of different road scenes into images in an optional embodiment of the present invention;

[0068] Figure 6 A diagram showing the result of cutting the vehicle-mounted video at different time points into images in an optional embodiment of the present invention;

[0069] Figure 7 A result diagram of cutting out a vehicle in an image by using coordinates in an optional embodiment of the present invention;

[0070] Figure 8 This is a diagram of vehicle detection results in an optional embodiment of the present invention. DETAILED DESCRIPTION

[0071] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided in order to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art. If not specifically stated, the technical means used in the embodiments are conventional means well known to those skilled in the art.

[0072] The present invention relates to a hybrid real-time vehicle detection method based on Transformer and BLS, the steps are: obtaining video data through vehicle-mounted equipment, cutting it into images frame by frame; using Swin-Transformer network to extract features of the image, identifying the vehicle in the image and generating an identification frame; intercepting the vehicle in the identification frame, graying the intercepted vehicle image, and obtaining a vehicle grayscale image; using a width learning system to classify the vehicle grayscale image and using an incremental learning strategy to optimize the model, and obtaining the specific category of the vehicle; sending the identification frame data and the classification result back to the video detection system to display real-time video data. The present invention aims at the problem of the difficulty of deploying vehicle detection algorithms on vehicle-mounted equipment with limited resources.

[0073] The present invention belongs to the field of computer vision and deep learning, and relates to vehicle detection, vehicle classification and data preprocessing technology. The present invention provides a real-time vehicle detection method, specifically a hybrid real-time vehicle detection method based on Transformer and BLS, including vehicle detection, vehicle classification and data preprocessing, introducing a Swin-Transformer network to extract features from images, obtain accurate vehicle anchor boxes, and then use a width learning system to classify vehicle images in the anchor box and optimize the width learning system through an incremental learning strategy. The present invention can solve the problem that the data transmission between the cloud and the vehicle-mounted edge device of the vehicle detection algorithm is easily affected by bandwidth congestion and response delay, and the problem caused by directly deploying the detection algorithm on the vehicle-mounted edge device with limited resources.

[0074] The present invention proposes a real-time vehicle detection method based on a hybrid of Transformer and BLS, the steps are as follows:

[0075] Step 1: Get the video data from the vehicle camera and cut it into images frame by frame;

[0076] Step 2: Use the improved Swin-Transformer network structure to extract and detect features of the image obtained in step 1. The improved Swin-Transformer network structure extracts features of the vehicle in the image, calculates the position coordinates of the vehicle based on the extracted features, adjusts the Swin-Transformer network structure through the loss function, and finally outputs the vehicle coordinate results;

[0077] Step 3: Process the vehicle position coordinates detected in step 2, exclude the detection frame where the vehicle is not detected according to the confidence level, cut the vehicles in the image by the coordinates and grayscale them to obtain a grayscale image of a single vehicle image;

[0078] Step 4: Use the width learning system to classify the grayscale image of the vehicle image cut out in step 3 to obtain the specific category of the vehicle;

[0079] Step 5: Use the incremental learning strategy to optimize the width learning system, improve the classification accuracy of the network, and obtain more accurate vehicle categories, such as sedans, trucks, SUVs, etc.

[0080] Step 6: Map the vehicle position coordinates and vehicle recognition results obtained in steps 2 and 5 back to the video data frame by frame to obtain real-time video detection results;

[0081] Furthermore, the steps of the improved Swin-Transformer network structure detection in step 2 are:

[0082] Step 2-1: Get the vehicle video in step 1 and cut it, use the cut image as the input image, scale the input image, and adjust the size of the input image to N i ×M i ×J; where N i Represents the length of the image, M i Represents the width of the image;

[0083] Step 2-2: Image segmentation, input image is N i ×M i ×J, after convolution, the size is N p ×M p ×J p The feature map is flattened to a size of N p M p ×J p The eigenvector of

[0084] Step 2-3: Feature fusion, input feature vector is N p M p ×J p , the improved Swin-Transformer network obtains a size of N c M c ×J c The eigenvector of

[0085] Step 2-4: Set the size obtained in step 2-3 to N c M c ×J c The feature vector of is passed through two linear networks in parallel to obtain the predicted box and predicted confidence. The loss is calculated based on the rectangular box loss function between the real box and the confidence loss function.

[0086] Step 2-5: Adjust the improved Swin-Transformer network structure through the loss function, and finally output the position coordinates of the vehicle in the image;

[0087] Furthermore, in steps 2-4,

[0088] The rectangular box regression loss function is:

[0089]

[0090] Among them, L reg is the rectangular box regression loss value; R is:

[0091]

[0092] Among them, t i Contains x ,t y ,t h ,t w Four parameters, which are calculated as follows:

[0093]

[0094]

[0095]

[0096]

[0097] Among them, x, y, w, h are the center coordinates and width and height of the prediction box, x a ,y a ,h a ,w a are the center coordinates, width and height of the anchor box;

[0098] in, Include Four parameters, which are calculated as follows:

[0099]

[0100]

[0101]

[0102]

[0103] Among them, x * ,y * ,w * ,h * is the center coordinate and width and height of the real annotation box, x a ,y a ,h a ,w a are the center coordinates and width and height of the anchor box.

[0104] The confidence loss function is:

[0105]

[0106] In the formula, m is the number of samples, y is i is the true value of the ith sample, and y′ is the predicted value of the ith sample.

[0107] Furthermore, the classification method of the width learning system in step 4 is:

[0108] Step 4-1: Get the vehicle grayscale image obtained in step 3 as the input graphic data X, perform feature mapping on X in the width learning system, and obtain all mapping features Z s ;

[0109] Step 4-2: According to Z s , a randomly generated matrix and Calculate the corresponding enhanced node H l ;

[0110] Step 4-3: Map feature Z s and enhanced node H l Merge them together into a matrix, and send the merged matrix and to the width learning system to calculate the output Y.

[0111] Furthermore, the expression of output Y in step 4-3 is:

[0112] Y=[Z s |H l ]W=PW

[0113] Where P is the mapping feature Z s and enhanced node H l The combined matrix P = [Z s |H l ], W is the weight of the connection network, expressed as:

[0114]

[0115] Where γ→0, I is the unit matrix, and lim γ→0 (γI+P T P) -1 P T Compute the pseudoinverse of the matrix P.

[0116] Furthermore, the incremental learning strategy in step 5 is:

[0117] Step 5-1: Add enhancement nodes and randomly generate matrices and Calculate the corresponding enhanced node Hl+1 ;

[0118] Step 5-2: Add the newly added enhanced node H l+1 and mapping feature Z s and enhanced node H l Merge to get a new matrix;

[0119] Step 5-3: Send the newly merged matrix to the updated width learning system and calculate the output Y′;

[0120] Furthermore, the expression of output Y′ in step 5-3 is:

[0121] Y′=[Z s |H l ,H l+1 ]W′=P′W′

[0122] Where P′ is the mapping feature Z s , Enhanced node H l And the newly added enhanced node H l+1 The combined matrix P = [Z s |H l ,H l+1 ], W′ is the updated weight of the connection network, expressed as:

[0123]

[0124] Where D is the newly added enhanced node and [Z s |H l ] by multiplying the pseudo-inverse of B T It is expressed as:

[0125]

[0126] Among them A m For [Z s |H l ], C is expressed as:

[0127] C=H l+1 -A m D.

[0128] The present invention is a real-time vehicle detection method based on the hybrid of Transformer and BLS. The overall framework is as follows Figure 2 As shown in the figure. First, (a) describes the image acquisition and processing process. Image data is acquired through the vehicle camera, and the image is preprocessed and scaled to a fixed size. As shown in (b), the preprocessed image is used to train Swin-Transformer and BLS. Finally, in (3), the trained model is verified on the Huawei Alats 200 edge device.

[0129] First, the improved Swin-Transformer is used to extract and detect features of road images, then the vehicles in the images are segmented and grayscale processed, and finally the width learning system (BLS) is used to classify the processed vehicle images and the incremental learning strategy is used to optimize the model. The whole process is as follows: Figure 1 shown.

[0130] like Figure 5 and Figure 6 As shown, the road videos collected at different locations and at different time points in the public places in this embodiment are taken as an example for explanation, and the same is applicable to other road conditions.

[0131] In an optional embodiment of the present invention, a real-time vehicle detection method is provided, which is based on a hybrid of Transformer and BLS, and the main steps are as follows:

[0132] Step 1: Obtain video data from the vehicle camera and cut it into images frame by frame;

[0133] The acquisition target is to take the road conditions of daily travel captured by the vehicle camera as an example and cut them into images; the road conditions captured by the camera are rich, such as provincial roads, national roads, expressways and urban roads, etc., and include different time points such as morning, noon and dusk. The acquired vehicle video is cut into images at a fixed frame rate, as shown in the attached figure. Figure 5 and Figure 6 As shown in the figure, the video taken by the vehicle camera on different roads and at different time points is cut into images.

[0134] Step 2: Use the Swin-Transformer network structure to extract and detect features of the image obtained in step 1. The Swin-Transformer network structure extracts features of the vehicle in the image, calculates the position coordinates of the vehicle based on the extracted features, adjusts the Swin-Transformer network structure through the loss function, and finally outputs the result of the vehicle image;

[0135] Using the improved Swin-Transformer network structure Figure 3 As shown in the figure, the road image cut out in step 1 is sent to the improved Swin-Transformer network to extract features from the image. The position coordinates and confidence of the vehicle in the image are calculated based on the extracted features. The network is adjusted through the loss function, and the vehicle position coordinates are finally output. The improved Swin-Transformer network determines the category of the prediction box based on the confidence of the prediction box. The categories mainly include two parts: vehicle and non-vehicle. The prediction box of non-vehicle is excluded, and the position coordinates of the prediction box including the vehicle are output.

[0136] Cut the detected vehicle from the entire monitoring image and convert it to grayscale to obtain Figure 7 In the images shown, one car is one image.

[0137] The improved Swin-Transformer network structure detects pedestrians, and the detection process is as follows:

[0138] Step 2-1: Get the surveillance image in step 1 as the input image, perform adaptive image scaling on the input image, and adjust the size of the input image to N i ×M i ×J; where N i Represents the length of the image, M i The width dimension of the image;

[0139] The initial input image information includes an image size of N×M×J, where N represents the length of the image, M represents the width of the image, and J represents the number of channels of the image.

[0140] The adaptive image scaling calculation method:

[0141] In order to unify the size of the input image, the image is adaptively scaled. The specific steps are as follows:

[0142] A: First calculate the adaptive scaling ratio. The original size is N×M, and the original target size of the adaptive scaling is N 1 ×M 1 , scaling factor Z 1 , Z 2 Calculated by the following formula;

[0143]

[0144]

[0145] B: If Z 1 <Z 2 We choose Z 1 As the final scaling factor Z, otherwise choose Z 2 As the final scaling factor Z;

[0146] C: Calculate the size after adaptive scaling, multiply the length and width of the original image by the scaling factor Z to get the length and width N of the image to be filled 2 and M 2 ;

[0147] D: If N 2 =N 1 , select M 2 As the position to be filled, otherwise select N 2As a position to be filled;

[0148] E: Calculate the black edge filling value, in M 2 As an example, the position to be filled is used to calculate the height that needs to be filled, and the array a that needs to be filled at both ends of the image is calculated. Finally, the size N of the image after scaling is calculated. i and M i ;

[0149] N i =N 1 =N 2 (3)

[0150]

[0151] Step 2-2: Image segmentation, input image is N i ×M i ×J, after convolution, the size is N p ×M p ×J p The feature map is flattened to a size of N p M p ×J p The eigenvector of

[0152] Step 2-3: Feature fusion, input feature vector is N p M p ×J p , obtained through the Swin-Transformer network with a size of N c M c ×J c The eigenvector of

[0153] Step 2-4: Calculate the loss and the overlap area between the predicted box and the true box;

[0154] The loss of the present invention in the vehicle detection stage includes two parts, rectangular box loss and confidence loss. The rectangular box loss is defined by the following loss function, where L c is the loss value, k represents the category, k+1 represents the background category, p ij is a false negative class, p ji is a false positive class, p ii For the real class;

[0155] The rectangular box regression loss function is:

[0156]

[0157] Among them, L reg is the rectangular box regression loss value; R is:

[0158]

[0159] Among them, t i Contains x ,t y ,t h ,t w Four parameters, which are calculated as follows:

[0160]

[0161]

[0162]

[0163]

[0164] Among them, x, y, w, h are the center coordinates and width and height of the prediction box, x a ,y a ,h a ,w a are the center coordinates and width and height of the anchor box; where, Include Four parameters, which are calculated as follows:

[0165]

[0166]

[0167]

[0168]

[0169] Among them, x * ,y * ,w * ,h * is the center coordinate and width and height of the real annotation box, x a ,y a ,h a ,w a are the center coordinates and width and height of the anchor box.

[0170] The category confidence loss function is:

[0171]

[0172] Step 2-5: Adjust the improved Swin-Transformer network structure through the loss function, and finally output the position coordinates of the vehicle in the image;

[0173] Step 3: Process the vehicle position coordinates detected in step 2, cut the vehicles in the image by the coordinates and grayscale them to obtain a grayscale image of a single vehicle image;

[0174] The steps to process the image by coordinates are as follows:

[0175] Step 3-1: Get the coordinates and confidence of the image predicted by the network in step 2;

[0176] Step 3-2: Filter out the coordinates of the area where there is no vehicle by confidence, obtain the coordinates of the area with higher confidence, and segment the area;

[0177] Step 3-3: Grayscale the segmented area and convert the RGB color image into a grayscale image;

[0178] Step 4: Use the width learning system to classify the grayscale image of the vehicle image cut out in step 3 to obtain the specific category of the vehicle. The network structure of the width learning system (BLS) is as follows: Figure 4 As shown;

[0179] The classification method of width learning system is as follows:

[0180] Step 4-1: Get the vehicle image result obtained in step 2 as the input graphic data X, where X is an R*E matrix, R represents the number of samples of the input data, and E represents the dimension of each sample. Perform feature mapping on X in the width learning system and obtain all mapping features Z s ;

[0181] Use the following formula to perform nonlinear feature mapping Z on the input data X;

[0182]

[0183] in and are all randomly generated matrices. We combine s different Z to get s groups of feature nodes, Z s =[Z 1 ,Z 2 ,Z 3 ,…,Z s ], which represents all the mapping features we extract from the input X through network mapping.

[0184] Step 4-2: Then according to Z s , a randomly generated matrix and Calculate the corresponding enhanced node H l ;

[0185] The enhanced node H is calculated using the following formula;

[0186]

[0187] and It is still our randomly generated matrix. Through the above formula, we get H j Also in matrix form, by combining different H together we can get the final set of enhanced nodes H l =[H 1 ,H 2 ,H 3 ,…,H l ], which represents all the enhanced nodes we get from mapping features.

[0188] Step 4-3: Map feature Z s and enhanced node H l Merge them together into a matrix, and send the merged matrix and to the width learning system to calculate the output Y;

[0189] Y is an R*P matrix, where P represents the number of categories of samples.

[0190] Map feature Z s and enhanced node H l The final output Y can be obtained by multiplying the combined matrix with the weights of the network connection;

[0191] Y=[Z s |H l ]W=PW(9)

[0192] Where P is Z s and H l The combined matrix P = [Z s |H l ], W is the weight of the connection network, which can be expressed as:

[0193] W=lim γ→0 (γI+P T P) -1 P T Y(10)

[0194] In this case, we usually need to let γ→0, I be the identity matrix, and lim γ→0 (γI+P T P) -1 P T Calculate the pseudo-inverse of the matrix P, multiply the pseudo-inverse of P by the output matrix Y to get the network connection weight W. Substitute the calculated W into (9) to get the final output Y of the network.

[0195] Step 5: Use the incremental learning strategy to optimize the width learning system, improve the classification accuracy of the network, and obtain more accurate vehicle categories, including sedans, trucks, and SUVs.

[0196] The incremental learning strategy method is as follows:

[0197] Step 5-1: Add a new enhancement node and randomly generate the matrix again and Calculate the corresponding newly added enhanced node H j+k , where H j+k The calculation formula is:

[0198]

[0199] Among them, j is the number of original enhancement nodes of the width system, and new enhancement nodes are continuously added to finally obtain H l+1 =[H j+1 ,H j+2 ,H j+3 ,…,H j+k ], k = 1, 2, 3, ..., which represents a group of newly added enhanced nodes.

[0200] Step 5-2: Add the newly added enhanced node H l+1 and mapping feature Z s and enhanced node H l Merge to get a new matrix, which is expressed as [Z s |H l ,H l+1 ], where the newly added set of enhanced nodes are merged into the original merge matrix.

[0201] Step 5-3: Send the newly merged matrix to the updated width learning system and calculate the output Y′. The expression of Y′ is as follows:

[0202] Y′=[Z s |H l ,H l+1 ]W′=P′W′(12)

[0203] Where P′ is the mapping feature Z s , Enhanced node H l And the newly added enhanced node H l+1 The combined matrix P = [Z s |H l ,H l+1], W′ is the updated weight of the connection network. Since the model adds new nodes, the weight W needs to be re-optimized. The incremental learning strategy does not need to retrain all weight parameters. It only needs to train the weights of the newly added enhanced nodes to obtain the updated weight W′. The expression of weight W′ is as follows:

[0204]

[0205] Where D is the newly added enhanced node and [Z s |H l ] by multiplying the pseudo-inverse of B T It is expressed as:

[0206]

[0207] Among them A m For [Z s |H l ], C is expressed as:

[0208] C=H l+1 -A m D(15)

[0209] The updated weight W′ can be calculated by using equations (13), (14) and (15), and Y′ can be obtained by substituting the weight into equation (12).

[0210] Step 6: Map the vehicle position coordinates and vehicle category obtained in steps 2 and 4 back to the video data frame by frame to obtain real-time video detection results. During operation, the real-time video is segmented into images frame by frame and sent to the network to obtain the vehicle position coordinates and category. Based on the information obtained, the vehicle and category are marked in the image to obtain a marked image. Finally, the image is returned to the original video frame by frame to achieve the effect of real-time detection;

[0211] The implementation steps are as follows:

[0212] Step 6-1: Obtain the coordinates and classification results obtained in step 2 and step 4;

[0213] Step 6-2: Use the obtained data to mark the image and obtain the marked image, such as Figure 8 As shown;

[0214] Step 6-3: Merge the marked images frame by frame, synthesize the video and return it to the display system;

[0215] Experimental Results

[0216] The present invention conducted the following comparative experiments: (1) Comparison after replacing the Faster R-CNN backbone network with the improved Swin-Transformer network model (2) Comparison between the improved Swin-Transformer network model and other detection models after combining incremental width learning. The experimental results are shown in Table 1 below:

[0217] Table 1 Comparative experimental results

[0218]

[0219]

[0220] It can be seen from Table 1 that the present invention is better than most models in terms of training speed, and is only slower than the training speed of the SSD model, but the detection effect of the present invention is better than the SSD model; the present invention is better than most models in terms of detection effect, and is only worse than the YOLOV5 model, but the present invention is better than the YOLOV5 model in terms of training speed, and the storage resources of the present invention are less than those of the comparison models, and it is more suitable for deployment on vehicle-mounted equipment.

[0221] Therefore, in comparison, the overall performance of the present invention is better, balancing the detection accuracy, training speed and storage resources, and overall has a faster training speed and less computing and storage resources, and has a better detection effect. The experimental results show that the present invention is more suitable for deployment on edge devices, and better solves the problems caused by directly deploying detection algorithms on vehicle-mounted devices.

[0222] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

[0223] Unless otherwise defined, the technical terms or scientific terms used in the present disclosure should be understood by people with ordinary skills in the field to which the present disclosure belongs. The "first", "second" and similar words used in the present disclosure do not indicate any order, quantity or importance, but are only used to distinguish different components. Similarly, similar words such as "one", "one" or "the" do not indicate quantitative restrictions, but indicate that there is at least one. "Including" or "comprising" and similar words mean that the elements or objects appearing before the word cover the elements or objects listed after the word and their equivalents, without excluding other elements or objects.

Claims

1. A real-time vehicle detection method, characterized in that: The steps include: Obtain video data from the vehicle-mounted camera and cut it into images frame by frame; The image size is adjusted to a unified format and input into a trained improved Swin-Transformer network model, the trained improved Swin-Transformer network model is used to perform feature extraction and target detection on the image, a vehicle identification frame and a category confidence are generated, and the position coordinates and category confidence of the vehicle identification frame are output; the vehicle in the image is intercepted according to the XY position coordinates of the upper left corner and the lower right corner of the vehicle identification frame to obtain a single vehicle image in the vehicle identification frame; grayscale the single vehicle image to obtain a grayscale image of the single vehicle image; input the grayscale image of the single vehicle image into a trained incremental width learning system, and use the trained incremental width learning system to perform feature extraction and classification prediction on the grayscale image of the single vehicle image to obtain a specific category of the single vehicle image; The method for obtaining the trained improved Swin-Transformer network model includes: Based on the existing Swin-Transformer network model, the number of stacked sub-network blocks is simplified, a 2-4-2 stacking strategy is used, and the Softmax activation function in the multi-head attention mechanism is changed to the Relu function to obtain an improved Swin-Transformer network model. Establishing a training set and a test set, wherein the images in the training set and the test set include images taken by a vehicle-mounted camera at different times and on different road types; Using the training set to train the improved Swin-Transformer network model, using the test set to test the improved Swin-Transformer network model, and using a loss function to adjust the parameters of the network structure of the improved Swin-Transformer network model to obtain a trained improved Swin-Transformer network model; The vehicle identification frame and the specific category of the single vehicle image are mapped back to the video data frame by frame to obtain a real-time vehicle detection result.

2. The real-time vehicle detection method according to claim 1, characterized in that: The method of adjusting the parameters of the network structure of the improved Swin-Transformer network model by using the loss function specifically includes: The loss function includes rectangular box regression loss function and confidence loss function; The rectangular box regression loss function is: Among them, L reg is the rectangular box regression loss value; R is: Among them, t i Contains x , t y , t h , t w Four parameters, which are calculated as follows: Among them, x, y, w, h are the center coordinates and width and height of the prediction box, x a ,y a ,h a , w a are the center coordinates, width and height of the anchor box; in, Include Four parameters, which are calculated as follows: Among them, x * ,y * , w * ,h * is the center coordinate and width and height of the real annotation box, x a ,y a ,h a , w a are the center coordinates, width and height of the anchor box; The category confidence loss function is: L d is the category confidence loss value, m is the number of samples, y i is the true value of the i-th sample, and y′ is the predicted value of the i-th sample; The predicted rectangular box regression loss value and the category confidence loss value are calculated and added using the rectangular box regression loss function and the category confidence loss function to obtain a final loss value; after obtaining the final loss value, the improved Swin-Transformer network model parameters are updated by the reverse gradient propagation algorithm; when the final loss value tends to infinitesimal or meets the specified training rounds, the predicted rectangular box and the real box coordinates and the predicted category confidence tend to be the same, and the model parameter adjustment is completed, thereby obtaining a trained improved Swin-Transformer network model.

3. The real-time vehicle detection method according to claim 1, characterized in that: Inputting the image into a trained improved Swin-Transformer network model, performing feature extraction and target detection on the image using the trained improved Swin-Transformer network model, generating a vehicle identification frame and a category confidence, and outputting the position coordinates of the vehicle identification frame and the category confidence specifically includes: The input image is adjusted and scaled through the preprocessing process, and the size of the image is uniformly adjusted to N i ×M i ×J, and get the scaled image; where N i Represents the length of the image, M i Represents the width of the image, and J represents the number of RGB channels of the image; After scaling N i ×M i The image of size ×J is input into the convolutional layer of the improved Swin-Transformer network, and is extracted into a size of N through the convolutional layer. p ×M p ×J p The feature map of size N p ×M p ×J p Flatten the feature map to get a size of N p M p ×J p The eigenvector of The multi-head attention mechanism in Swin-Transformer is used to p M p ×J p The feature vector of size N is obtained by feature fusion c M c ×J c The eigenvector of The resulting size is N c M c ×J c The feature vector of is passed through two fully connected layers in series, and then passed through two fully connected layers in parallel to obtain the coordinates of the predicted box and the predicted category confidence respectively.

4. The real-time vehicle detection method according to claim 3, characterized in that: The method of scaling the input image and uniformly adjusting the size of the image to Ni×Mi×J to obtain the scaled image specifically includes: The original size of the input image is N×M×J, where N represents the original length of the image, M represents the original width of the image, and J represents the number of channels of the image; Calculate the ratio of adaptive scaling, the original size of the image is N×M, the original target size of adaptive scaling is N1×M1, and the scaling coefficients Z1 and Z2 are calculated by the following formula; When Z1<Z2, select Z1 as the final scaling factor Z; when Z1≥Z2, select Z2 as the final scaling factor Z; Calculate the size after adaptive scaling, and multiply the length and width of the original image by the scaling factor Z to obtain the length and width N2 and M2 of the image to be filled; When N2=N1, select M2 as the position to be filled; when N2≠N1, select N2 as the position to be filled; Calculate the black edge filling value, take M2 as the position to be filled, calculate the original height to be filled, and then calculate the array a to be filled at both ends of the image, fill the array a to both ends of the image to get the final scaled image; the size of the image after scaling N i and M i The calculation method is as follows: N i =N1=N2 5. The real-time vehicle detection method according to claim 1, characterized in that: The method for obtaining the trained incremental width learning system specifically includes: Establishing a training set and a test set, wherein the images in the training set and the test set include grayscale images of multiple images of single vehicles of different categories; The width learning system is trained using the training set, the width learning is tested using the test set, and the width learning system is optimized using an incremental learning strategy to obtain a trained incremental width learning system.

6. The real-time vehicle detection method according to claim 5, characterized in that: The method of training the width learning system using the training set specifically includes: The grayscale image X of a single vehicle image in the training set is used as the input of the width learning system; wherein X is an R*E matrix; R represents the number of samples of the input data, and E represents the dimension of each sample number; Map X in the width learning system and get all the mapping features Z s ; Among them, the nonlinear feature mapping Z of the input data X is performed using the following formula: in, and are all randomly generated matrices; s different Zs are combined to obtain s groups of feature nodes, Z s =[Z1, Z2, Z3, ..., Z s ], represents all mapping features extracted from input X through network mapping; According to Zs, the randomly generated matrix and Calculate the corresponding enhanced node H l ; The enhanced node H is calculated using the following formula; and It is still the randomly generated matrix. The Hj we get from the above formula is also in matrix form. By combining different H together, we can get the final set of enhanced nodes H 1 =[H1, H2, H3, ..., H l ], which represents all the enhanced nodes we get from mapping features; Map feature Z s and enhanced node H l Merge them together into a matrix, and send the merged matrix and to the width learning system to calculate the output Y; Y is an R*P matrix, where P represents the number of sample categories; Map feature Z s and enhanced node H l The final output Y can be obtained by multiplying the combined matrix with the weights of the network connection; The expression of output Y is: Y=[Z s |H l ]W=PW (9) Where P is Z s and H l The combined matrix P = [Z s |H 1 ], W is the weight of the connection network, which can be expressed as: Among them, γ→0, I is the unit matrix, lim γ→0 (γI+P T P) -1 PT calculates the pseudo-inverse of the matrix P. The pseudo-inverse of P is multiplied by the output matrix Y to obtain the connection weight W of the network. Substituting the calculated value into (9) can obtain the final output Y of the network. The category confidence of all vehicle categories contained in Y is used to select the vehicle category with the highest category confidence as the final specific vehicle category.

7. The real-time vehicle detection method according to claim 6, characterized in that: The method for optimizing the width learning system using the incremental learning strategy specifically includes: Add a new enhancement node and randomly generate the matrix again and Calculate the corresponding newly added enhanced node H j+k , where H j+k The calculation formula is: Among them, j is the number of original enhancement nodes of the width system, and new enhancement nodes are continuously added to finally obtain H l+1 =[H j+1 , H j+2 , H j+3 , ..., H j+k ], k = 1, 2, 3, ..., which represents a group of newly added enhanced nodes; The newly added enhanced node H l+1 and mapping feature Z s and enhanced node H 1 Merge to get a new matrix, which is expressed as [Z s |H 1 , H l+1 ], where the newly added set of enhanced nodes are merged into the original merge matrix; The newly merged matrix is ​​fed into and updated into the width learning system to calculate the output Y′, whose expression of Y′ is as follows: Y′=[Z s |H l ,H l+1 ]W′=P′W′ (12) Where P′ is the mapping feature Z s , enhance node H l And the newly added enhanced node H l+1 The combined matrix P = [Z s |H l , H l +1 ], W′ is the updated weight of the connection network; Since the model adds new nodes, the weight W needs to be re-optimized and the weights of the newly added enhanced nodes are trained to obtain the updated weight W′. The expression of weight W′ is as follows: Where D is the newly added enhanced node and [Z s |H l ] by multiplying the pseudo-inverse of B T It is expressed as: Among them A m For [Z s |H l ], C is represented by: C=H l+1 -A m D (15) The updated weight W′ can be calculated by equations (13), (14) and (15). Substituting the weight into equation (12) can obtain the updated final output Y′. Then, the vehicle category with the highest category confidence in Y′ is selected as the specific vehicle category of the final output.

8. The real-time vehicle detection method according to claim 1, characterized in that: The method of mapping the vehicle identification frame and the specific category of the single vehicle image back to the video data frame by frame to obtain a real-time vehicle detection result specifically includes: Marking the image using the vehicle identification frame and the specific category of the single vehicle image to obtain a marked image; The marked images are merged frame by frame to synthesize the video and return it to the display system.

Images