A method and device for identifying a vehicle double flash

Abstract

The application discloses a kind of vehicle double flash lamp identification method and device, propose a kind of vehicle double flash lamp identification method combining spatial attention mechanism and contrast learning, by constructing the convolutional neural network of attention mechanism as light state feature extraction module, the frame sequence image block pair of the same light of the same vehicle is used as training sample, and the feature extraction network model is trained using the way of contrast learning.The application uses convolutional neural network to extract features to reduce the impact of unstable light state features caused by environmental changes, the network attention is focused on the light area through the spatial attention mechanism, solving the problem of unstable light area segmentation in image processing method, and the contrast learning method is used to improve the discrimination of features.The application effectively improves the robustness and accuracy of vehicle double flash lamp identification through the above method.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning, in particular to a vehicle double flash light recognition method and device. BACKGROUND

[0002] In the prior art, vehicle double flash recognition methods mainly include the following:

[0003] 1. Color space grid clustering method: This method uses a grid clustering method in the color space model YCbCr to identify vehicle double flash light flickering, clusters the Cr component according to the Cr component variation of the divided grid, and determines whether it is double flashing according to the clustering result.

[0004] 2. Brightness difference judgment method: This method extracts the accurate position of the vehicle light through contour detection, and takes at least three images of the same vehicle along the time interval, and the brightness value of the vehicle light in the three images is at least one greater than the brightness value difference threshold of the other two, to determine whether the vehicle is a vehicle double flash.

[0005] 3. HSV color model judgment method: This method first locates the tail light, then divides the target color in the HSV space, and sets the dynamic threshold of the left and right two side lights, respectively, extracts the pixel mean value of the tail light area for three seconds, compares the difference between the pixel mean values of the adjacent two seconds, and if the difference is greater than the threshold, the double flash light is turned on.

[0006] The present application inventors found that the prior art method has at least the following technical problems in the implementation of the present application:

[0007] 1. Color space grid clustering method: This method uses color features, and the application environment is an indoor parking lot. It also needs to rely on experience to divide the grid, which is easily affected by the environment. Too much experience design and limited use environment make it difficult to promote the algorithm on a large scale.

[0008] 2. Brightness difference judgment method: This method compares the accurate light brightness difference to determine whether the vehicle is turned on double flash. Under natural conditions, due to vehicle movement and surrounding environment, the image captured by the camera is blurred, and the vehicle light contour area cannot be extracted, resulting in the inability to obtain the accurate position of the vehicle light. In addition, the vehicle light brightness difference needs to be compared with the experience threshold, which limits the use of this method.

[0009] 3. HSV color model judgment method: Color and brightness are used as the main features of the vehicle light state, which is easily affected by distance, surrounding environment (vehicle body color, light), and has poor robustness. The use of color and contour segmentation to locate the vehicle light is easily affected by the vehicle body dirt, and the accuracy is low. At the same time, the double flash recognition uses an experience threshold to determine the experience threshold, which is difficult to define.

[0010] In summary, the above methods all use artificial experience features, which are greatly affected by environmental factors. Motion blur causes the vehicle light area to be unable to be accurately segmented, and the addition of different backgrounds can even change the distribution of the features, thereby causing low double flash recognition accuracy. SUMMARY

[0011] The application provides a vehicle double flash light recognition method and device to solve or at least partially solve the technical problem of low recognition accuracy in the prior art.

[0012] To solve the above technical problems, the application adopts the following technical solutions:

[0013] The first aspect provides a vehicle double flash light recognition method, comprising:

[0014] Collecting a large number of vehicle videos with double flash or left and right turning behaviors;

[0015] According to the collected videos, training sets and test sets are made, wherein the training sets and test sets are sample pairs composed of two time sequence images in the same vehicle light instance sequence;

[0016] A vehicle light state feature extraction network is constructed, which focuses attention on the vehicle light area through a spatial attention mechanism and extracts vehicle light state features using a convolutional neural network. The vehicle light state feature extraction network adopts a twin network, and the two networks share weights;

[0017] Based on the made training sets, the constructed vehicle light state feature extraction network is trained in a contrast learning manner. During the training process, after the vehicle light state feature extraction network extracts the feature vectors of the sample pairs in the training sets, the Euclidean distances between the feature vectors of the sample pairs are calculated as feature distances. Then, the calculated feature distances and sample labels are input into a loss function to calculate the loss. The network parameters are optimized through gradient descent and back propagation. The network with the optimal network parameters is saved as the trained vehicle light state feature extraction network. The vehicle double flash light recognition network with the highest vehicle light state discrimination degree on the test set and the corresponding discrimination threshold are obtained. The discrimination threshold is used as the feature distance threshold;

[0018] According to the obtained real-time vehicle video data, vehicle light samples are extracted;

[0019] The trained vehicle light state feature extraction network is used to extract vehicle light state features from the extracted vehicle light samples;

[0020] The Euclidean distance between the features of the current frame and the features of the previous frame in each vehicle light instance of each vehicle is calculated, which is denoted as the feature distance;

[0021] According to the relationship between the obtained feature distance and the feature distance threshold, it is determined whether the state of the vehicle light changes, if the state changes, the number of state changes of all vehicle lights of the same vehicle within a preset time T is counted, and it is determined whether the number of state changes exceeds a change threshold, if the number of state changes exceeds the change threshold, it is determined that the vehicle is turned on double flash, otherwise the vehicle is not turned on double flash.

[0022] In an embodiment, the training sample and the test sample are made according to the collected video, including:

[0023] The time sequence image block of each vehicle light instance is cropped in combination with the vehicle detector, the vehicle light detector and the multi-target tracker;

[0024] The time sequence image of each vehicle light instance is classified according to the state of the vehicle light, and two samples are randomly selected in the same vehicle light instance sequence to form a sample pair, if the states of the two vehicle light samples in the sample pair are consistent, the sample pair label is 0, otherwise 1;

[0025] The obtained sample pair is divided into a training set and a test set according to a preset proportion.

[0026] In an embodiment, the cropped time sequence image block of the vehicle light instance is a square image block with the vehicle light as the center and the long side of the vehicle light as the side length, and the formula is:

[0027]

[0028] image represents the original image, width and height represent the width and height of the original image respectively, sx1 and sx2 represent the minimum and maximum values of the horizontal coordinates of the vehicle light in the original image respectively, sy1 and sy2 represent the minimum and maximum values of the vertical coordinates of the vehicle light in the original image respectively, extend represents the expansion ratio, cx and cy represent the horizontal and vertical coordinates of the center of the vehicle light, size represents the long side of the vehicle light, sample represents the vehicle light sample cropped from the original image image, dx1 and dx2 represent the minimum and maximum values of the horizontal coordinates of the cropped vehicle light sample respectively, dy1 and dy2 represent the minimum and maximum values of the vertical coordinates of the cropped vehicle light sample respectively.

[0029] In an embodiment, the processing process of the constructed vehicle light state feature extraction network includes:

[0030] The input image data is subjected to data enhancement;

[0031] The first 4 layers of a preset lightweight separable convolution neural network are used to form a backbone network to extract an initial feature map from the data-enhanced input image data;

[0032] The initial feature map is convolved by a basic convolution operation BaseConv to obtain a spatial attention mask map;

[0033] The initial feature map and the spatial attention mask map are bilinearly attention-pooled, the feature matrix of the initial feature map is weighted by the spatial attention mask map, and a processed attention mask map is obtained.

[0034] The processed attention mask map is normalized by linear normalization to obtain a normalized feature vector.

[0035] In an embodiment, the loss function has the form:

[0036]

[0037] wherein y represents the value of the sample label, d represents the feature distance between samples, M represents the maximum interval, and B represents the number of samples per iteration, wherein when y = 0, the states of the two samples are the same, and the loss at this time is the distance between the two samples, the greater the distance, the greater the penalty, and when y = 1, the states of the two samples are different, and the loss at this time is related to the difference between the feature distance and the maximum interval.

[0038] In an embodiment, the vehicle light samples are extracted from the acquired real-time vehicle video data, comprising:

[0039] Each vehicle instance in the video image is acquired by using a vehicle detector, a vehicle light detector, and a multi-target tracker;

[0040] All vehicle lights in the vehicle are detected every preset number of frames, and for each vehicle instance sequence, the vehicle light number is obtained by matching the vehicle light with the identified vehicle light in the vehicle sequence, specifically comprising: first, normalizing the vehicle light coordinates, the formula is:

[0041]

[0042] wherein vx1 and vx2 represent the minimum and maximum values of the horizontal coordinates of the vehicle bounding rectangle, vy1 and vy2 represent the minimum and maximum values of the vertical coordinates of the vehicle bounding rectangle, src_lx1 and src_lx2 represent the minimum and maximum values of the horizontal coordinates of the original vehicle light rectangle, src_ly1 and src_ly2 represent the minimum and maximum values of the vertical coordinates of the original vehicle light rectangle, vcx and vcy represent the horizontal and vertical coordinates of the vehicle center, dst_lx1 and dst_lx2 represent the minimum and maximum values of the horizontal coordinates of the normalized vehicle light rectangle, dst_ly1 and dst_ly2 represent the minimum and maximum values of the vertical coordinates of the normalized vehicle light rectangle, and normal represents the normalization coefficient; then the overlap between the vehicle light and the identified vehicle light is calculated:

[0043]

[0044] wherein box1, box2 represent the normalized vehicle light bounding boxes, and overlap represents the overlap degree of box1 and box2; the vehicle light matching is performed through the overlap degree, and the matching formula is as follows:

[0045]

[0046] wherein overlap_box represents the overlap degree of the vehicle light, i, j, and k represent the vehicle number, the vehicle light number, and the frame sequence number respectively, and thresh represents the matching threshold; the formula indicates that if the overlap degree of the current vehicle light and the vehicle light identified in the buffer sequence is greater than the matching threshold, the current vehicle light cur_box is matched with the identified vehicle light , wherein cur_box represents the current vehicle light bounding box, , and the identified vehicle light bounding box is represented by

[0047] A square image block centered at the vehicle light and having the length of the long side of the vehicle light is intercepted as a vehicle light sample, and is denoted as

[0048] In an embodiment, whether the state of the vehicle light of the vehicle changes is determined by the following formula:

[0049]

[0050] wherein i, j, and k represent the vehicle number, the vehicle light number, and the frame sequence number respectively, dist represents the distance between the features, thresh represents the feature distance threshold, and flag represents the flag of whether the state of the vehicle light changes.

[0051] Based on the same inventive concept, the second aspect of the present application provides a vehicle double flash light recognition device, comprising:

[0052] A video collection module is configured to collect a large number of videos of vehicles with double flash or left turning or right turning behaviors.

[0053] A data set construction module is configured to make a training set and a test set according to the collected videos, wherein the training set and the test set are sample pairs composed of two time sequence images in the same vehicle light instance sequence.

[0054] ​The feature extraction network construction module is configured to construct a car light state feature extraction network, the car light state feature extraction network focuses attention on the car light region through a spatial attention mechanism, and extracts car light state features by using a convolutional neural network, the car light state feature extraction network adopts a twin network, and two networks share weights;

[0055] The training module is configured to train the constructed car light state feature extraction network in a contrast learning manner based on the prepared training set, in the training process, after the feature vectors of a sample pair in the training set are extracted by the car light state feature extraction network, the Euclidean distance between the feature vectors of the sample pair is calculated as a feature distance, the calculated feature distance and a sample label are input into a loss function to calculate a loss, the network parameters are optimized in a gradient descent and back propagation manner, the network with the optimal network parameters is saved as the trained car light state feature extraction network, the vehicle double flash identification network with the highest car light state distinguishing degree on the test set and the corresponding distinguishing threshold are obtained, and the distinguishing threshold is taken as a feature distance threshold;

[0056] The car light sample extraction module is configured to extract car light samples from the obtained real-time vehicle video data;

[0057] The car light state feature extraction module is configured to extract car light state features of the extracted car light samples by using the trained car light state feature extraction network.

[0058] The distance calculation module is configured to calculate the Euclidean distance between the features of the current frame and the features of the previous frame in each car light instance of each vehicle, and the Euclidean distance is recorded as a feature distance.

[0059] The double flash identification module is configured to determine whether the state of the car light of the vehicle changes according to the relationship between the obtained feature distance and the feature distance threshold, if the state changes, count the number of state changes of all car lights of the same vehicle within a preset time T, and determine whether the number of state changes exceeds a change threshold, if the number of state changes exceeds the change threshold, it is determined that the vehicle starts double flashing, otherwise, the vehicle does not start double flashing.

[0060] Based on the same inventive concept, the third aspect of the present application provides a computer readable storage medium having a computer program stored thereon, the program being executed by a processor to implement the method of the first aspect.

[0061] Based on the same inventive concept, the fourth aspect of the present application provides a computer device, comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor executes the program to implement the method of the first aspect.

[0062] Compared with the prior art, the present application has the following advantages and beneficial technical effects:

[0063] The application provides a vehicle double flash lamp recognition method, first, a large amount of video data is collected, and a training set and a test set are constructed based on the collected video data, then a vehicle lamp state feature extraction network is constructed, which adopts a convolutional neural network based on an attention mechanism, and a frame sequence image block pair of the same vehicle lamp of the same vehicle is taken as a training sample, since the convolutional neural network is adopted to extract the vehicle lamp state feature, the influence of the unstable vehicle lamp state feature caused by environmental changes can be reduced, the network attention is focused on the vehicle lamp area through the spatial attention mechanism, the problem of unstable vehicle lamp area segmentation in the image processing method is solved, and the feature discrimination is improved through the contrast learning method in the training process, finally, the vehicle lamp state extraction in the vehicle lamp sample in the real-time vehicle video data is performed through the trained vehicle lamp state feature extraction network, distance calculation and state change judgment are performed, so that the double flash lamp recognition is realized, and the robustness and precision of the vehicle double flash lamp recognition are effectively improved. BRIEF DESCRIPTION OF DRAWINGS

[0064] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0065] Figure 1 The vehicle double flash lamp recognition overall flowchart in the embodiment of the present application;

[0066] Figure 2 The time sequence diagram of the same vehicle lamp of the same vehicle in the embodiment of the present application;

[0067] Figure 3 The schematic diagram of classifying the same vehicle lamp by state in the embodiment of the present application;

[0068] Figure 4 The schematic diagram of the sample pair constructed in the embodiment of the present application;

[0069] Figure 5 The training framework diagram of the vehicle double flash lamp recognition model in the embodiment of the present application;

[0070] Figure 6 The structure diagram of the vehicle lamp state feature extraction network in the embodiment of the present application;

[0071] Figure 7 The attention mechanism extraction vehicle lamp area weight mask schematic diagram in the embodiment of the present application. DETAILED DESCRIPTION

[0072] The present application inventors find through a large amount of research and practice that the main reasons for low recognition accuracy in the prior art include:

[0073] 1. The vehicle light state feature robustness is not strong. The current mainstream method for judging the vehicle double flash adopts experience features such as color, brightness, etc. In a real scene, different light, different distance, different vehicle light, different vehicle body color, vehicle movement and other factors will cause the features extracted by the method to be unstable and weakly distinguishable.

[0074] 2. The vehicle light region segmentation is difficult. The current vehicle light region segmentation mainly adopts an image processing method, and the motion blur caused by the vehicle body color and vehicle movement makes the vehicle light region segmentation difficult.

[0075] Based on this, the present application provides a vehicle double flash light recognition method combining spatial attention mechanism and contrast learning, which can effectively solve the above problems and improve the robustness and accuracy of the algorithm.

[0076] That is, the current mainstream vehicle double flash light recognition method is to first locate the vehicle light position information, segment the vehicle light region using an image processing method, then extract experience features such as color, brightness, etc., and finally determine whether the vehicle is double flashing by comparing the degree of change of the vehicle light feature vector between the front and rear frames. The present application provides a vehicle double flash light recognition method combining spatial attention mechanism and contrast learning, which mainly solves the following problems and innovations:

[0077] The present application provides a vehicle double flash light recognition method, which constructs an attention mechanism convolutional neural network as a vehicle light state feature extraction module, takes the frame sequence image block pair of the same vehicle light of the same vehicle as a training sample, and trains the feature extraction network model using a contrast learning method. The present application uses a convolutional neural network to extract features to reduce the influence of unstable vehicle light state features caused by environmental changes, focuses the network attention on the vehicle light region through a spatial attention mechanism, solves the problem of unstable vehicle light region segmentation using an image processing method, and improves the distinguishability of the features using a contrast learning method. The present application effectively improves the robustness and accuracy of vehicle double flash light recognition through the above methods.

[0078] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0079] Embodiment one

[0080] The embodiment of the present application provides a video of a large number of vehicles with double flashing or left turning or right turning behaviors;

[0081] S1: training set and test set are made according to collected videos, wherein the training set and the test set are sample pairs composed of two time sequence images in the same car light instance sequence;

[0082] S2: a car light state feature extraction network is constructed, the car light state feature extraction network focuses attention on the car light region through a spatial attention mechanism, and extracts car light state features by using a convolutional neural network, the car light state feature extraction network adopts a twin network, and the two networks share weights;

[0083] S3: based on the made training set, a car light state feature extraction network is trained in a contrast learning manner, in the training process, after feature vectors of sample pairs in the training set are extracted by the car light state feature extraction network, the Euclidean distances between the feature vectors of the sample pairs are calculated as feature distances, then the calculated feature distances and sample labels are input into a loss function to calculate a loss, network parameters are optimized in a gradient descent and back propagation manner, a network with optimal network parameters is saved as the trained car light state feature extraction network, a vehicle double flashing light recognition network with the highest car light state distinguishing degree on the test set and a corresponding distinguishing threshold are obtained, and the distinguishing threshold is taken as a feature distance threshold;

[0084] S4: a car light sample is extracted according to real-time vehicle video data;

[0085] S5: the trained car light state feature extraction network is used for car light state feature extraction on the extracted car light sample;

[0086] S6: the Euclidean distance between the feature of a current frame and the feature of a previous frame in each car light instance of each vehicle is calculated and recorded as a feature distance;

[0087] S7: whether the state of the car light of the vehicle changes or not is judged according to the relationship between the obtained feature distance and the feature distance threshold, if the state changes, the number of state changes of all car lights of the same vehicle within a preset time T is counted, and whether the number of state changes exceeds a change number threshold is judged, if the number of state changes exceeds the change number threshold, it is determined that the vehicle turns on double flashing, otherwise, the vehicle does not turn on double flashing.

[0088] The present application provides a vehicle double flashing light recognition method, and the application scenarios are as follows: in a video stream, a vehicle and a car light are visible to the naked eye, the area of the car light is greater than a specified area threshold (by default, 50 pixels), and the duration of the vehicle in the field of view is greater than a specified time threshold (by default, 5 seconds). Figure 1This is a flowchart illustrating the overall process of vehicle hazard light recognition in this embodiment of the invention. The recognition method mainly includes two stages: the training stage of the offline vehicle light state feature extraction network (S1-S3) and the online vehicle hazard light recognition stage (S4-S7).

[0089] S3 is the training process for the vehicle headlight state feature extraction network. The specific training framework is as follows: Figure 5 As shown, the system includes a headlight state feature extraction module, a feature distance calculation module, and a loss calculation module. The feature extraction module employs a Siamese network, with the two sub-networks sharing weights. The benefit of this step is the construction of a headlight state feature extraction network training framework, used to train the headlight state feature extraction network model.

[0090] In the specific implementation process, the feature distance calculation module is used to calculate the Euclidean distance between the feature vectors of sample pairs, and the formula is: x i ,y i These represent two feature vectors respectively. The value of the i-th component is given, where n represents the dimension of the feature vector (n = 512 in this invention), and dist represents the distance between two feature vectors. The benefit of this implementation is that it calculates the distance between headlight pairs, providing a basis for loss calculation.

[0091] In one implementation, training samples and test samples are created based on the collected videos, including:

[0092] By combining the vehicle detector, the headlight detector, and the multi-target tracker, time-series image patches of each headlight instance are cropped out;

[0093] The time series images of each vehicle light instance are classified according to the state of the vehicle light. Two samples are randomly selected from the same vehicle light instance sequence to form a sample pair. If the states of the two vehicle light samples in the sample pair are the same, the sample pair label is 0; otherwise, it is 1.

[0094] The obtained sample pairs are divided into training set and test set according to a preset ratio.

[0095] In the specific implementation process, such as Figure 2 The image shown is a timing diagram of the same headlight on the same vehicle. Each headlight instance sequence image is categorized by state, as follows: Figure 3 As shown (where, Figure 3 The three images on the left show the headlights off, and the four images on the right show the headlights on. Then, two samples are randomly selected from the same headlight instance sequence to form a sample pair. If the two headlight samples in the sample pair have the same state, the sample pair is labeled 0; otherwise, it is labeled 1. Figure 4The left group of states is inconsistent, and the middle and right groups of samples are consistent in state. The benefit of the embodiment is to provide data for training the vehicle lamp state feature extraction network.

[0096] In an embodiment, the time sequence image block of the cropped vehicle lamp instance is a square image block centered on the vehicle lamp with the long side of the vehicle lamp as the side length, and the formula is:

[0097]

[0098] image represents the original image, width and height represent the width and height of the original image respectively, sx1 and sx2 represent the minimum and maximum values of the horizontal coordinates of the vehicle lamp in the original image respectively, sy1 and sy2 represent the minimum and maximum values of the vertical coordinates of the vehicle lamp in the original image respectively, extend represents the expansion ratio, cx and cy represent the horizontal and vertical coordinates of the center of the vehicle lamp, size represents the long side of the vehicle lamp, sample represents the vehicle lamp sample cropped from the original image image, dx1 and dx2 represent the minimum and maximum values of the horizontal coordinates of the cropped vehicle lamp sample respectively, and dy1 and dy2 represent the minimum and maximum values of the vertical coordinates of the cropped vehicle lamp sample respectively.

[0099] In the implementation process, vehicle detection and vehicle lamp detection are realized by yolov5, and multi-target tracking is realized by sort. The default value of extend is 0.5.

[0100] In an embodiment, the processing procedure of the vehicle double flash lamp recognition model includes the following steps:

[0101] Data augmentation is performed on the input image data;

[0102] A backbone network composed of the first 4 layers of a preset lightweight separable convolution neural network is used to extract an initial feature map from the input data-augmented image data;

[0103] A spatial attention mask map is obtained by performing convolution on the initial feature map through a basic convolution operation BaseConv;

[0104] Bilinear attention pooling is performed on the initial feature map and the spatial attention mask map, the feature matrix of the initial feature map is weighted by the spatial attention mask map, and a processed attention mask map is obtained;

[0105] The processed attention mask map is normalized through linear normalization, and a normalized feature vector is obtained.

[0106] In the implementation process, the structure diagram of the vehicle lamp state feature extraction network is as shown in Figure 6The shown (wherein augument represents data augmentation, augument image represents the image after data augmentation, backbone represents the backbone network, feature maps are initial feature maps, Attention maps represent attention mask maps obtained after basic convolution, BAP represents bilinear attention pooling operation, normalize represents normalization, and features represent the final output features of the car light state feature extraction network).

[0107] First, the input 112x112 color image is data augmented, including random cropping, random noise addition, random brightness, and random contrast change, and is sent to the backbone (backbone network) composed of the first 4 layers of mobilenetv2 (a lightweight separable convolution neural network) to extract the initial feature map FM=(fm d1,d2,d3 ) C×H×W , wherein FM represents a feature map matrix, C, H, and W represent the size of each dimension of the three-dimensional matrix, and in the present application, C, H, and W are 32, 14, and 14 respectively, fm d1,d2,d3 represents the element value at the dimension coordinates d1, d2, and d3 in the three-dimensional matrix.

[0108] Then, FM is obtained through a set of BaseConv operations to obtain a spatial attention mask map AM=(am d1,d2,d3 ) M×H×W , wherein in the present application, M, H, and W are 16, 14, and 14 respectively, am d1,d2,d3 represents the element value at the dimension coordinates d1, d2, and d3 in the three-dimensional matrix. After average visualization, as shown in Figure 7 , the BaseConv operation is composed of 32x16 3x3 convolutions combined with BN and RELU, the input is a 32x14x14 tensor, and the output is a 16x14x14 tensor. FM and AM are subjected to BAP operation to obtain a feature matrix F=(f d1,d2 ) C×M , wherein in the present application, C and M are 32 and 16 respectively, the BAP operation is to weight the feature matrix with the attention mask, increase the weight of the car light area, and reduce the weight of the non-car light area, as shown in Figure 7 , the formula is:

[0109] Finally, the feature matrix F=(f d1,d2 ) C×M is expanded into a one-dimensional feature vector In the present application, the dimension of the vector is 512, and normalization is performed, and the normalization formula is: , wherein the vector is obtained from data x i, i = 1, 2,... n, vector is normalized to is normalized to. The benefit of this step is to build a car light state feature extraction network model, use a convolutional neural network as a car light state feature extractor, and add a spatial attention mechanism to effectively increase the weight of the car light area and reduce the weight of the non-car light area, thereby excluding the influence of the non-car light area on the features.

[0110] In one embodiment, the loss function has the form:

[0111]

[0112] where y represents the value of the sample label, d represents the feature distance between samples, M represents the maximum interval, and B represents the number of samples per iteration, wherein when y = 0, the states of the two samples are the same, and the loss at this time is the distance between the two samples, the greater the distance, the greater the penalty, and when y = 1, the states of the two samples are different, and the loss at this time is related to the difference between the feature distance and the maximum interval.

[0113] In the specific implementation process, the feature distance dist and the sample label label are input into the loss function to calculate loss. M represents the maximum interval, and the default is 2.0. The benefit of this embodiment is to calculate the loss of the training samples, to maximize the spatial distance between different state samples and to minimize the spatial distance between the same state samples, and to provide power for the optimization of the network model parameters through loss calculation.

[0114] In the specific training process, the training and testing sample sets prepared in the foregoing steps are used in this embodiment, the number ratio is 8:2, and a network model training framework is constructed. The network training learning rate is 0.01, and the gradient descent method is used to train 100 epochs (all samples are iterated once). At the 30th, 50th, and 80th epochs, the learning rate is multiplied by 0.1. After each epoch of training, the accuracy of the model is verified on the test set, and the feature extraction network and the feature distance threshold at the highest accuracy are saved. The benefit of this embodiment is to train the feature extraction network model, optimize the network parameters through gradient descent and back propagation, obtain a feature extraction network model with the highest car light state discrimination on the test set, and provide a feature distance threshold for the online vehicle double flash light recognition model.

[0115] In one embodiment, the car light samples are extracted from the obtained real-time vehicle video data, including:

[0116] Using a vehicle detector, a car light detector, and a multi-target tracker to obtain each vehicle instance in the video image;

[0117] Detecting all vehicle lights in the vehicle once every preset number of frames, and obtaining the light number through light matching with the identified light in the vehicle sequence for each vehicle instance sequence, specifically comprising: first normalizing the light coordinates, the formula is:

[0118]

[0119] Wherein, vx1, vx2 respectively represent the minimum and maximum values of the horizontal coordinates of the vehicle bounding rectangle, vy1, vy2 respectively represent the minimum and maximum values of the vertical coordinates of the vehicle bounding rectangle, src_lx1, src_lx2 respectively represent the minimum and maximum values of the horizontal coordinates of the original light rectangle, src_ly1, src_ly2 respectively represent the minimum and maximum values of the vertical coordinates of the original light rectangle, vcx, vcy respectively represent the horizontal and vertical coordinates of the vehicle center, dst_lx1, dst_lx2 represent the minimum and maximum values of the horizontal coordinates of the normalized light rectangle, dst_ly1, dst_ly2 represent the minimum and maximum values of the vertical coordinates of the normalized light rectangle, and normal represents the normalization coefficient; then the overlap between the light and the identified light is calculated:

[0120]

[0121] Wherein, box1, box2 respectively represent the normalized light bounding rectangle, and overlap represents the overlap of box1 and box2; then the light is matched through the overlap, and the matching formula is:

[0122]

[0123] Wherein, overlap_box represents the overlap of the light, i, j, k respectively represent the vehicle number, the light number, and the frame sequence number, and thresh represents the matching threshold, and the formula represents that if the current light and the identified light in the buffer sequence have an overlap greater than the matching threshold, the current light cur_box is matched with the identified light Wherein, cur_box represents the light bounding rectangle, represents the light bounding rectangle of the vehicle number i, the light number j, and the frame sequence number k in the buffer zone;

[0124] A square image block centered on the light and having the length of the long side of the light is intercepted as a light sample, denoted as

[0125] In the specific implementation process, the preset number of frames can be set according to the actual situation, for example, 5 frames, 6 frames, etc. Then the light feature extraction is performed: the extracted light sample is sent to the trained light state feature extraction network for feature extraction, denoted as where i, j, k represent vehicle number, vehicle light number, frame number respectively, and represents feature extraction network function.

[0126] In an embodiment, whether the state of the vehicle light changes is determined by the following formula:

[0127]

[0128] where i, j, k represent vehicle number, vehicle light number, frame number respectively, and dist, thresh, represent distance between features, feature distance threshold, and flag of whether the state of the vehicle light changes respectively.

[0129] In a specific implementation process, the vehicle double flashing behavior is defined as: whether the number of state changes of the vehicle light on both sides of the head or tail of a vehicle within a time T (for example, 2 seconds) is greater than N (for example, 3), if greater than N, the vehicle is considered to have double flashing light. Whether the state of the vehicle light changes, for a vehicle light instance, as shown in the left group, the state of the vehicle light changes, and the state of the vehicle light in the middle and right groups does not change. The benefit of this embodiment is to define the problem to be solved, which is conducive to analyzing the advantages and disadvantages of existing methods, so as to propose a better solution. Figure 4

[0130] After determining whether the state of the vehicle light changes, double flashing recognition is performed by the following method:

[0131] The number N (default is 3) of state changes of all vehicle lights of each vehicle within a time T (default is 2 seconds, frame rate is 25, and the number of frames is 25x2=50) is counted, and if the number of state changes of two tail lights or head lights of a vehicle is greater than N at the same time, it is considered that the vehicle has double flashing light. The formula is:

[0132]

[0133] where i, j, T, C represent vehicle number, vehicle light number, counting time period, and counting frame number respectively, and N represents change threshold, and flick represents whether double flashing flag.

[0134] The beneficial effects of the present application are:

[0135] (1) A vehicle light state feature extraction network is constructed, which focuses the network attention on the vehicle light region through a spatial attention mechanism, and extracts vehicle light state features by using a convolutional neural network. The convolutional neural network is used to extract features, which can reduce the influence of unstable vehicle light state features caused by environmental changes. The spatial attention mechanism focuses the network attention on the vehicle light region, which can solve the problem of unstable segmentation of the vehicle light region by the image processing method.​

[0136] (2) The training framework of the vehicle lamp state feature extraction network is constructed, and the contrast learning method is used to improve the discrimination of the features.

[0137] Embodiment Two

[0138] Based on the same inventive concept, the embodiment provides a vehicle double flash lamp recognition device, which comprises:

[0139] A video collection module is configured to collect a large number of videos of vehicles with double flash or left turning or right turning behaviors.

[0140] A data set construction module is configured to make a training set and a test set according to the collected videos, wherein the training set and the test set are sample pairs composed of two time sequence images in the same vehicle lamp instance sequence.

[0141] A feature extraction network construction module is configured to construct a vehicle lamp state feature extraction network, the vehicle lamp state feature extraction network focuses the network attention on the vehicle lamp region through a spatial attention mechanism, and extracts vehicle lamp state features by using a convolutional neural network, the vehicle lamp state feature extraction network adopts a twin network, and the two networks share weights.

[0142] A training module is configured to train the constructed vehicle lamp state feature extraction network based on the made training set by using a contrast learning method, in the training process, after the vehicle lamp state feature extraction network extracts the feature vectors of the sample pairs in the training set, the Euclidean distances between the feature vectors of the sample pairs are calculated as feature distances, then the calculated feature distances and sample labels are input into a loss function to calculate a loss, the network parameters are optimized by gradient descent and back propagation, the network with the optimal network parameters is saved as the trained vehicle lamp state feature extraction network, the vehicle double flash lamp recognition network with the highest vehicle lamp state discrimination on the test set and the corresponding discrimination threshold are obtained, and the discrimination threshold is taken as a feature distance threshold.

[0143] A vehicle lamp sample extraction module is configured to extract vehicle lamp samples according to the obtained real-time vehicle video data.

[0144] A vehicle lamp state feature extraction module is configured to extract vehicle lamp state features of the extracted vehicle lamp samples by using the trained vehicle lamp state feature extraction network.

[0145] The distance calculation module is configured to calculate the Euclidean distance between the features of the current frame and the features of the previous frame in each light instance of each vehicle, and the distance is denoted as a feature distance; the double flash identification module is configured to determine whether the state of the light of the vehicle changes according to the relationship between the obtained feature distance and a feature distance threshold, if the state changes, count the number of state changes of all the lights of the same vehicle within a preset time T, and determine whether the number of state changes exceeds a change threshold, if the number of state changes exceeds the change threshold, it is determined that the vehicle turns on the double flash, otherwise, the double flash is not turned on.

[0146] Since the device introduced in the embodiment two of the present application is the device used in the implementation of the method for identifying the double flash of the vehicle in the embodiment one of the present application, the specific structure and the deformation of the device can be understood by the person skilled in the art based on the method introduced in the embodiment one of the present application, and thus will not be described here again. Any device used in the method in the embodiment one of the present application belongs to the scope of the present application.

[0147] Embodiment three

[0148] Based on the same inventive concept, the present application further provides a computer readable storage medium having a computer program stored thereon, the program being executed to implement the method as described in the embodiment one.

[0149] Since the computer readable storage medium introduced in the embodiment three of the present application is the computer readable storage medium used in the implementation of the method for identifying the double flash of the vehicle in the embodiment one of the present application, the specific structure and the deformation of the computer readable storage medium can be understood by the person skilled in the art based on the method introduced in the embodiment one of the present application, and thus will not be described here again. Any computer readable storage medium used in the method in the embodiment one of the present application belongs to the scope of the present application.

[0150] Embodiment four

[0151] Based on the same inventive concept, the present application further provides a computer device, including a storage, a processor and a computer program stored in the storage and executable on the processor, the processor executes the above-mentioned program to implement the method in the embodiment one.

[0152] Since the computer device introduced in the embodiment four of the present application is the computer device used in the implementation of the method for identifying the double flash of the vehicle in the embodiment one of the present application, the specific structure and the deformation of the computer device can be understood by the person skilled in the art based on the method introduced in the embodiment one of the present application, and thus will not be described here again. Any computer device used in the method in the embodiment one of the present application belongs to the scope of the present application.

[0153] Those skilled in the art will appreciate that embodiments of the present application can be devised for a variety of other systems which are currently developed or later developed. Thus, the present application is intended to include alternative steps as can be possessed now or developed in the future for performing the purposes of the application. Examples of the application can be provided as a method, a system or a computer program product. Accordingly, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present application can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage devices, and the like) embodying computer readable program code thereon for use by or in connection with an instruction execution system. For the purposes of this description, a computer-usable or computer readable storage medium can be any apparatus that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.

[0154] The present application is described in reference to the drawings, which are as follows: Figure 1 Each flow and / or block in the flowchart and / or block diagram illustrations, and combinations of flows and / or blocks in the flowchart and / or block diagram illustrations, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks. Figure 1 Each flow and / or block in the flowchart and / or block diagram illustrations, and combinations of flows and / or blocks in the flowchart and / or block diagram illustrations, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart and / or block diagram block or blocks.

[0155] While the preferred embodiments of the application have been described, additional variations and modifications can be employed, as will be appreciated by those of ordinary skill in the art, once armed with the

[0156] It will be apparent to those skilled in the art that various modifications and variations can be made to the present application without departing from the spirit or scope of the application. Thus, it is intended that the present application cover the modifications and variations of this application provided they come within the scope of the appended claims and their equivalents.

Claims

1. A method of identifying a vehicle double flasher, characterized by, The method comprises the following steps: Collect a large number of videos of vehicles with double flashing or left / right turning behaviors; According to the collected videos, training sets and test sets are made, wherein the training sets and test sets are sample pairs composed of two time sequence images within the same vehicle light instance sequence; A vehicle light state feature extraction network is constructed, which focuses attention on the vehicle light area through a spatial attention mechanism and extracts vehicle light state features using a convolutional neural network. The vehicle light state feature extraction network adopts a twin network, and the two networks share weights; Based on the made training sets, the constructed vehicle light state feature extraction network is trained in a contrast learning manner. During the training process, after the feature vectors of the sample pairs in the training sets are extracted by the vehicle light state feature extraction network, the Euclidean distances between the feature vectors of the sample pairs are calculated as feature distances. Then, the calculated feature distances and sample labels are input into a loss function to calculate the loss. The network parameters are optimized through gradient descent and back propagation. The network with the optimal network parameters is saved as the trained vehicle light state feature extraction network. The vehicle double flashing light recognition network with the highest vehicle light state discrimination degree on the test set and the corresponding discrimination threshold are obtained. The discrimination threshold is used as the feature distance threshold; According to the obtained real-time vehicle video data, vehicle light samples are extracted; The trained vehicle light state feature extraction network is used to extract vehicle light state features from the extracted vehicle light samples; The Euclidean distance between the features of the current frame and the features of the previous frame in each vehicle light instance of each vehicle is calculated, which is recorded as a feature distance; According to the relationship between the obtained feature distance and the feature distance threshold, it is determined whether the state of the vehicle light of the vehicle changes. If the state changes, the number of state changes of all vehicle lights of the same vehicle within a preset time T is counted, and it is determined whether the number of state changes exceeds a change threshold. If the number of state changes exceeds the change threshold, it is determined that the vehicle turns on the double flashing light, otherwise, the vehicle does not turn on the double flashing light.

2. The method of claim 1, wherein, According to the collected videos, training samples and test samples are made, including: Each vehicle light instance time sequence image block is cropped by combining a vehicle detector, a vehicle light detector and a multi-target tracker; Each vehicle light instance time sequence image is classified according to the state of the vehicle light. Two samples are randomly selected within the same vehicle light instance sequence to form a sample pair. If the states of the two vehicle light samples in the sample pair are consistent, the sample pair label is 0, otherwise, the sample pair label is 1; The obtained sample pairs are divided into training sets and test sets according to a preset proportion.

3. The method of claim 2, wherein the vehicle double flasher is identified by, The cropped vehicle light instance time sequence image block is a square image block with the vehicle light as the center and the long side of the vehicle light as the side length, and the formula is: The image represents an original image, width and height represent the width and height of the original image respectively, sx1 and sx2 represent the minimum and maximum values of the horizontal coordinates of the vehicle lamp in the original image respectively, sy1 and sy2 represent the minimum and maximum values of the vertical coordinates of the vehicle lamp in the original image respectively, extend represents an expansion ratio, cx and cy represent the horizontal and vertical coordinates of the center of the vehicle lamp, size represents the long side of the vehicle lamp, sample represents a vehicle lamp sample cropped from the original image image, dx1 and dx2 represent the minimum and maximum values of the horizontal coordinates of the cropped vehicle lamp sample respectively, and dy1 and dy2 represent the minimum and maximum values of the vertical coordinates of the cropped vehicle lamp sample respectively.

4. The method of claim 1, wherein, The processing process of the constructed vehicle lamp state feature extraction network includes: performing data enhancement on the input image data; using a backbone network composed of the first 4 layers of a preset lightweight separable convolution neural network to extract an initial feature map from the input data-enhanced image data; performing convolution on the initial feature map through a basic convolution operation BaseConv to obtain a spatial attention mask map; performing bilinear attention pooling on the initial feature map and the spatial attention mask map, weighting the feature matrix of the initial feature map using the spatial attention mask map, and obtaining a processed attention mask map; performing linear normalization on the processed attention mask map to obtain a normalized feature vector.

5. The method of claim 1, wherein, The form of the loss function is: where y represents the value of the sample label, d represents the feature distance between samples, M represents the maximum interval, and B represents the number of samples per iteration, wherein when y=0, the two samples are the same, and the loss at this time is the distance between the two samples, the greater the distance, the greater the penalty, and when y=1, the two samples are different, and the loss at this time is related to the difference between the feature distance and the maximum interval.

6. The method of claim 1, wherein, According to the obtained real-time vehicle video data, a vehicle lamp sample is extracted, including: using a vehicle detector, a vehicle lamp detector and a multi-target tracker to obtain each vehicle instance in a video image; detecting all vehicle lamps in a vehicle every preset number of frames, for each vehicle instance sequence, matching the vehicle lamp with the identified vehicle lamp in the sequence to obtain a vehicle lamp number, specifically including: first normalizing the vehicle lamp coordinates, the formula is: Wherein, vx1, vx2 represent minimum and maximum values of horizontal coordinates of the vehicle's bounding rectangle respectively, vy1, vy2 represent minimum and maximum values of vertical coordinates of the vehicle's bounding rectangle respectively, src_lx1, src_lx2 represent minimum and maximum values of horizontal coordinates of the original vehicle lamp rectangle respectively, src_ly1, src_ly2 represent minimum and maximum values of vertical coordinates of the original vehicle lamp rectangle respectively, vcx, vcy represent horizontal and vertical coordinates of the vehicle center respectively, dst_lx1, dst_lx2 represent minimum and maximum values of horizontal coordinates of the normalized vehicle lamp rectangle, dst_ly1, dst_ly2 represent minimum and maximum values of vertical coordinates of the normalized vehicle lamp rectangle, and normal represents a normalization coefficient; then, the overlap between the vehicle lamp and the identified vehicle lamp is calculated: Wherein, box1, box2 represent the normalized vehicle lamp bounding rectangle respectively, and overlap represents the overlap between box1 and box2; then, the vehicle lamp is matched through the overlap, and the matching formula is: wherein overlap_box represents the overlap degree of the vehicle light, i, j, k represent the vehicle number, the vehicle light number, and the frame sequence number respectively, and thresh represents the matching threshold, and the formula indicates that if the overlap degree of the current vehicle light and the vehicle light identified in the buffer sequence is greater than the matching threshold, the current vehicle light cur_box is matched with the identified vehicle light wherein cur_box represents the current vehicle light circumscribed rectangle, represents the circumscribed rectangle of the vehicle light with the vehicle number i, the vehicle light number j, and the frame sequence number k in the buffer sequence. A square image block centered on the car light with the length of the long side of the car light is intercepted as a car light sample, denoted as 7. The method of claim 1, wherein, Whether the state of the vehicle lamp changes is determined by the following formula: where i, j, k represent vehicle number, light number and frame number respectively, dist,thresh, represent the distance between features, the feature distance threshold, and a flag indicating whether the light state has changed, respectively.

8. An identification device of a vehicle double flasher, characterized by, The method comprises the following steps: A video collection module is configured to collect a large number of videos of vehicles with double flashing or left or right turning behaviors; A data set construction module is configured to generate a training set and a test set according to the collected videos, wherein the training set and the test set are sample pairs composed of two time sequence images in the same vehicle lamp instance sequence; A feature extraction network construction module is configured to construct a vehicle lamp state feature extraction network, the vehicle lamp state feature extraction network focuses attention on the vehicle lamp region through a spatial attention mechanism, and extracts vehicle lamp state features by using a convolutional neural network, the vehicle lamp state feature extraction network adopts a twin network, and the two networks share weights; A training module is configured to train the constructed vehicle lamp state feature extraction network in a contrast learning manner based on the generated training set, in the training process, after the vehicle lamp state feature extraction network extracts feature vectors of the sample pairs in the training set, the Euclidean distances between the feature vectors of the sample pairs are calculated as feature distances, the calculated feature distances and sample labels are input into a loss function to calculate a loss, the network parameters are optimized in a gradient descent and back propagation manner, the network with the optimal network parameters is saved as the trained vehicle lamp state feature extraction network, the vehicle double flashing lamp recognition network with the highest vehicle lamp state distinguishing degree on the test set and the corresponding distinguishing threshold are obtained, and the distinguishing threshold is taken as a feature distance threshold; A vehicle lamp sample extraction module is configured to extract vehicle lamp samples according to real-time vehicle video data; A vehicle lamp state feature extraction module is configured to extract vehicle lamp state features of the extracted vehicle lamp samples by using the trained vehicle lamp state feature extraction network. The distance calculation module is configured to calculate an Euclidean distance between the feature of the current frame and the feature of the previous frame in each light instance of each vehicle, denoted as a feature distance; the double flashing identification module is configured to determine whether the state of the light of the vehicle changes according to the relationship between the obtained feature distance and a feature distance threshold value, if the state changes, count the number of state changes of all the lights of the same vehicle within a preset time T, and determine whether the number of state changes exceeds a change number threshold value, if the number of state changes exceeds the change number threshold value, it is determined that the vehicle turns on double flashing, otherwise, the vehicle does not turn on double flashing.

9. A computer readable storage medium having stored thereon a computer program, characterized in that, The program, when executed by the processor, implements the method of any one of claims 1 to 7.

10. A computer device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor implements the method of any one of claims 1 to 7 when executing the program.

Images