A method and system for detecting vehicle violations

Abstract

The present invention discloses a detection method and system for vehicle violation behavior, and the system includes a binocular camera module, a central processing unit, a GPS positioning module, a steering gear control module, a cloud computing server and a cloud management platform. The present invention uses the violation evidence chain formed by the video information shot by the main vehicle and the slave vehicle, and can collect violation certificates from various angles to perform vehicle re-identification to comprehensively judge whether the vehicle is in violation, so that the evidence chain is more complete and the evidence is clearer, which greatly reduces the false detection rate; and the evidence chain accurately detects violations through the vehicle re-identification algorithm, and the algorithm carried by the hardware is the Yolo-DAP algorithm that can resist adversarial sample attacks, and performs better in adversarial sample attacks and detects more accurately.

Description

Technical Field

[0001] The present invention relates to the technical field of driving safety, and in particular to a detection method and a system for detecting vehicle violations that can be used to collect evidence of violations while driving. Background Art

[0002] Intelligent transportation promotes harmony among people, vehicles and roads, and is an important part of urban development. Intelligent detection of vehicle violations is an important part of traffic management, which is related to the safety of people's lives and property. Vehicle violation detection usually uses videos shot by cameras in fixed positions, and managers use the videos to determine which vehicles have violated the rules, and record the violation process as a basis for punishing the driver. In this process, first of all, a lot of manpower and material resources are required. Manpower is needed to detect many videos, and there will be false detections and missed detections; secondly, due to the limited number of fixed-position cameras, only violations within a certain range can be captured; finally, due to the limited coverage of existing cameras, some drivers will have a fluke mentality, act recklessly outside the camera shooting area, ignore traffic regulations, and cause major accidents.

[0003] Some research has been done in the prior art. For example, a Chinese patent with publication number CN104952122A discloses a driving recorder that can automatically collect evidence of violations, including a three-dimensional visual system and a processing module. The three-dimensional visual system collects video data in the driving environment and identifies traffic signs and surrounding vehicles in the driving environment to generate traffic sign information and surrounding vehicle information, and determines whether the surrounding vehicles trigger the violation trigger event corresponding to the traffic sign information. If so, a trigger detection signal is generated and transmitted to the processing module. After receiving the trigger detection signal, the processing module determines whether a violation has occurred based on the corresponding traffic sign information, surrounding vehicle information and video data. If so, the three-dimensional visual system is controlled to collect corresponding violation photos to generate violation evidence information, and the violation evidence information is locked and saved. The present invention can detect violation trigger events and automatically generate violation evidence during driving, simplifying the reporting process and eliminating the psychology of illegal driving. The above scheme can only save the violation evidence information in a single line, and cannot guide the surrounding vehicles to collect the violation information of the violating vehicle. This will cause missed detection or false detection due to the angle problem, resulting in the inability to form an evidence chain. In addition, the saved violation evidence is a photo, which may be blurred or incomplete violation information, lack of evidence chain and other problems. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for detecting vehicle violations, which can improve the coverage area of ​​detection and effectively reduce the missed detection rate of vehicle violations.

[0005] In order to solve the above technical problems, the present invention also discloses a method for detecting vehicle violations, comprising the following steps:

[0006] (1) The main vehicle collects video streams in the driving environment and uses the target detection algorithm to detect targets in the video streams to determine whether there are any illegal vehicles in the video streams. When it is determined that there are illegal behaviors, it also determines whether the current road is in a special state. If the road is not in a special state, it reminds the user that the vehicle ahead may have violated the law, identifies the vehicle's license plate number, and records the time period of the violation. The relevant video of the detected illegal vehicle is compressed and uploaded to the cloud computing server together with the GPS information and time information.

[0007] (2) The cloud computing server reconstructs the image through super-resolution in the cloud and uploads the license plate number of the illegal vehicle, the time corresponding to the video frame, and the current GPS information to the cloud management platform;

[0008] (3) The cloud computing server sends messages to the slave vehicles around the violating vehicle based on the GPS information;

[0009] (4) After receiving the message from the vehicle, the video information shot during the violation time is compressed and uploaded to the cloud computing server;

[0010] (5) The cloud computing server reconstructs the video stream information with super-resolution, and uses the video information of the master vehicle and the slave vehicle to form a chain of evidence for vehicle violation to re-identify the vehicle and determine the position of the vehicle in the video image, and comprehensively judge whether the vehicle has violated the law from multiple angles; if the vehicle has violated the law and is not in the special vehicle database, the illegal vehicle information and the chain of evidence are saved in the cloud management platform database; if the vehicle violation still cannot be determined and is not in the special vehicle database, it is uploaded to the manual channel for judgment; if the vehicle has not violated the law, the chain of evidence is deleted.

[0011] Preferably, when the slave vehicle receives the image information sent by the master vehicle, the image information is first parsed to obtain the timestamp information transmitted from the cloud management server; then the video stream in the timestamp is compressed, and then the camera servo is moved to collect information about surrounding vehicles to determine whether there is a target vehicle in the current field of view. If the target vehicle is still recognized in the picture, the camera servo is moved to a suitable shooting angle and shoots for another 5 seconds, and the video data shot again is compressed and uploaded to the cloud computing server.

[0012] Preferably, in step (1), if the road is in a special state, it is determined whether the vehicle violates the rules when the road is in the special state. If so, the license plate number of the vehicle is identified and the violation time period is recorded; and the relevant video frames of the detected violating vehicle are compressed and uploaded to the cloud computing server.

[0013] Preferably, the target detection algorithm used in step (1) is the YOLO-DAP algorithm, comprising the following steps:

[0014] ① Train the basic YOLO recognition network: freeze the parameters of the FPRN layer and the PRN layer to train a neural network that can normally recognize objects;

[0015] ②Training the SVM classifier: First, organize the adversarial sample image dataset, which contains normal images and images after interference. The normal image corresponds to the output label 0, and the image after interference is the adversarial image. The model input is the interference data, that is, the interference pixel after the adversarial image and the normal image are subtracted according to the vector operation. If the output label is 0, it means it is not an interference pixel, and if the output label is 1, it means it is an interference pixel. After multiple iterations, when the SVM classifier can perform well in the test dataset, stop training;

[0016] ③Training the PRN layer and the FPRN layer: Freeze the parameters of the Yolo backbone network except the FPRN layer, that is, only update the parameters of the PRN layer and the FPRN layer in the following training process. The training data set is normal images and interference images generated by normal images. In one iteration, the normal image is directly input into the target network to obtain the predicted label, that is, whether the image is a normal image or an image that will affect the detection network after malicious disturbance. Then, the interference image is input into the PRN layer and then into the target detection network to obtain the location information, confidence information and category information of objects such as vehicles and pedestrians in the image. Finally, the network weights are updated according to the calculation results of the loss function.

[0017] ④ Classify the attack image: Use the trained SVM classifier to classify the input image to determine whether the image is an attack image;

[0018] ⑤ Repair the attack image: If the input image is judged to be an attack image in step ④, the image is repaired through the PRN layer. The repair process is to pass the input adversarial sample image through a convolution layer, then through 5 residual modules and then through two convolution layers, and output the image after removing interference;

[0019] ⑥ Image partitioning: After steps ④ and ⑤, the image will be divided into S x S grids. If the center of an object falls in this grid, then this grid is responsible for predicting the object. Each network needs to predict the location information and confidence information of B BBoxes. One location information corresponds to four location information and one confidence information. The confidence represents the confidence of the object contained in the predicted object box and the accuracy of the prediction of this object.

[0020] ⑦Add the Brelu activation function to the YOLO-DAP network: All activation functions in the network are replaced with bounded leaky relu activation functions;

[0021] ⑧ Repair the feature image: When the image feature information reaches the FPRN layer, the image feature layer is repaired adversarially. The feature map of the input adversarial sample image first passes through three residual modules and then a convolution layer, and the output is the feature map after removing interference;

[0022] ⑨ Detect object information: For each grid, two bounding boxes are predicted. After predicting S x S x 2 target windows, the target windows with lower probability are removed according to the threshold. Finally, the redundant windows are removed by maximum suppression to obtain the vehicle’s location information, category information, and confidence information.

[0023] Preferably, the vehicle re-identification method used is a re-id method with geographic location information and a characterization filter, which comprises the following steps:

[0024] ① Extract high-level representation: extract high-level representation features from the image;

[0025] ② Characteristic feature filtering: After the image passes through the characteristic feature filter, the samples that are judged as non-target vehicles in the high-level representation are filtered out, leaving the samples with the same characteristic features as the target vehicle;

[0026] ③ Sample filtering layer with geographic location information: When the sample is input into the subsequent processing process, it will carry the location, angle and other information when the sample was collected. These information are input at the same time, and the corresponding network parameters will calculate these parameters to strengthen or weaken the confidence of the sample. The sample filtering layer will integrate the location, angle and other information into the image itself and serve as the input of the feature extraction network;

[0027] ④ Fusion of global feature extraction and local feature extraction: Next, the image enters the feature extraction stage. Feature extraction consists of two parts, namely global feature extraction and evenly aligned local feature extraction. The result of global extraction will be used to solve the ID loss of the classifier; the result of local extraction will be spliced ​​with the result of global extraction, and then input into the difficult sample mining module to solve the loss. This loss is used to measure the accuracy of the REID algorithm, and the model network will be updated according to this loss when training the REID algorithm model.

[0028] The invention discloses a detection system for vehicle violation behavior, comprising a binocular camera module, which is used for collecting video streams and visual ranging; a central processing unit, which is used for performing target detection on the video stream collected by the binocular camera module, and sending the collected video stream to a cloud computing server after compression, and processing the communication between various devices; a GPS positioning module, which is used for obtaining accurate latitude and longitude information of the vehicle, and sending it to the central processing unit; a steering gear control module, which is used for controlling the steering of the binocular camera; a voice interaction module, which is responsible for voice interaction with users; a cloud computing server, which is used for super-resolution reconstruction of received images and re-identification of vehicles after reconstruction to determine specific violation situations; and a cloud management platform, which is used for managing vehicle violation information and handing over unconfirmed illegal vehicles to manual confirmation processing.

[0029] Preferably, the above scheme also includes a pre-collision detection module, which is used to obtain ranging information from the binocular camera module and issue a pre-collision warning when the distance between the front vehicle and the vehicle is lower than a threshold.

[0030] Preferably, the above scheme also includes a stabilizer module, which is arranged on the binocular camera module and is responsible for stabilizing the camera to prevent the inability to collect video streams and visual ranging due to vehicle bumps.

[0031] Preferably, the above scheme also includes a solar power supply system for powering various devices of the driving recorder.

[0032] Compared with the prior art, the present invention has the following advantages:

[0033] 1. The present invention can be used as a vehicle violation detection device, which greatly improves the coverage area of ​​detection and effectively reduces the missed detection rate of vehicle violation behaviors;

[0034] 2. The present invention utilizes the video information shot by the main vehicle and the slave vehicle to form a violation evidence chain, which can collect violation documents from various angles for vehicle re-identification to comprehensively determine whether the vehicle is in violation, making the evidence chain more complete and the evidence clearer, greatly reducing the false detection rate.

[0035] 3. The present invention adopts the method of evidence chain to accurately detect violations through vehicle re-identification algorithm. The algorithm carried by the hardware is the Yolo-DAP algorithm that can resist adversarial sample attacks. It performs better in adversarial sample attacks and has more accurate detection.

[0036] 4. Super-resolution reconstruction can process the reduced resolution caused by compression to increase video pixels and increase recognition accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] Figure 1 A flow chart of a method for detecting vehicle violations according to the present invention;

[0038] Figure 2 It is a structural diagram of Yolo-DAP in the present invention;

[0039] Figure 3 It is a network structure diagram of PRN in the present invention;

[0040] Figure 4 It is a network structure diagram of FPRN in the present invention;

[0041] Figure 5 Flow chart of the cloud computing server in the present invention;

[0042] Figure 6 This is an application flow chart of the vehicle re-identification algorithm in the present invention;

[0043] Figure 7 Schematic diagram of the structure of the re-id algorithm used in the present invention;

[0044] Figure 8 The present invention is a module block diagram of a system for detecting vehicle violations. DETAILED DESCRIPTION

[0045] In order to make the purpose and advantages of the present invention more clear, the present invention is described in detail below with reference to the accompanying drawings and embodiments.

[0046] like Figure 1 As shown, the present invention also discloses a method for detecting vehicle violations, comprising the following steps:

[0047] In the main vehicle mode: (1) The binocular camera module collects the video stream in the driving environment and transmits it to the central processor. The central processor continuously obtains the video stream from the binocular camera module and uses the Yolo-DAP (yolodefense adversarial perturbations) algorithm to detect targets in the video stream to determine whether there are illegal vehicles in the video stream; when it is determined that there are illegal behaviors, it also determines whether the current road is in a special state (such as road construction). If the road is not in a special state, it reminds the user that the vehicle ahead may have violated the law, and identifies the vehicle's license plate number and records the time period of violation; and compresses the relevant video of the detected illegal vehicle and uploads it to the cloud computing server together with the GPS information and time information. If the road is in a special state, it determines whether the vehicle violates the law when the road is in a special state. If it violates the law, it identifies the vehicle's license plate number and records the time period of violation; and compresses the relevant video frames of the detected illegal vehicle and uploads them to the cloud computing server.

[0048] The Yolo-DAP (yolo defense adversarial perturbations) algorithm is a target detection algorithm that can resist the interference of adversarial samples. The algorithm is improved based on the Yolov3 algorithm so that the algorithm has the ability to resist adversarial samples. The improvements of Yolo-DAP to Yolov3 are as follows: First, the activation function Relu function in Yolov3 is replaced with the bounded leaky Relu function (Bounded leakyRelu) which performs better in adversarial sample attacks. 2. Add a PRNF network (Pertrubation Rectifying Network Combine Feature) to process the interference image data. The main structure of the PRNF network has two parts: a) Network head: SVM binary classifier and perturbation correction network PRN (Pertrubation Rectifying Network), SVM is responsible for determining whether the image is an adversarial image sample. PRN is responsible for eliminating the perturbation of the adversarial image sample and outputting a normal image, and modifying the loss function of PRN. b) Feature perturbation correction structure FPRN, which performs secondary adversarial repair on the feature map obtained by multiple sampling of the neural network. c) Optimize the loss function of the PRN algorithm and add the difference of the generated image to the original loss function as the loss function of the PRNF network.

[0049] like Figure 2 As shown in the figure, it is the structure diagram of Yolo-DAP. After the image is input, the SVM detector first determines whether the image is an adversarial sample. If it is not an adversarial sample, it will not pass through the PRN (perturbation repair) layer and directly input the image into the subsequent network. When the image feature information reaches the FPRN layer, the image feature layer is adversarially repaired. Figure 2 DBB is an important network module, which consists of convolutional layers, BN (batch normalization) layers and brelu activation functions. The res_unit module is a residual neural network module, which is composed of two DBB modules stacked together. During the calculation process, the parameters of the two layers are added to form a residual module. In the Resn module, n is the number of res_unit modules in the module. It first fills the feature map with zeros (adds 0 pixels around the feature map to adjust the size of the feature map) and then inputs it into the DBB module and then passes through n res_unit modules. Concat: tensor concatenation. Concatenate two tensors, which will expand the dimension of the tensor.

[0050] The specific steps of the Yolo-DAP algorithm are as follows:

[0051] ① Train the basic YOLO recognition network: Freeze the parameters of the FPRN layer and the PRN layer to train a neural network that can normally recognize objects.

[0052] ② Training SVM classifier: In the present invention, SVM classifier is used to classify ordinary images and adversarial sample images. The process is as follows: first, the adversarial sample image data set is sorted, and the data contains normal images and images after interference. The normal image corresponds to the output label 0, and the image after interference is the adversarial image. The model input is the interference data, that is, the interference pixel after the adversarial image and the normal image are subtracted according to the vector operation. If the output label is 0, it means that it is not an interference pixel, and if the output label is 1, it means that it is an interference pixel. After multiple iterations, when the SVM classifier can perform well in the test data set, stop training.

[0053] ③Training the PRN layer and the FPRN layer: Freeze the parameters of the Yolo backbone network except the FPRN layer, that is, only update the parameters of the PRN layer and the FPRN layer in the following training process. The training data set is a normal image and an interference image generated by a normal image. In one iteration, the normal image is directly input into the target network to obtain a predicted label, and then the interference image is input into the PRN layer and then into the target detection network to obtain a predicted label, and finally the network weight is updated according to the calculation result of the loss function. The present invention uses the dataset cityscapes, which contains a dataset of objects such as cars, people, and bicycles. The image is used as input data and the category label is used as output data. Finally, a neural network model that can normally identify the positions of objects such as vehicles and people in the image is trained. The residual structure added at the FPRN network can ensure that the Yolo backbone network will not be affected by the newly added network FPRN during training.

[0054] ④ Classify the attack image: Use the trained SVM classifier to classify the input image to determine whether the image is an attack image;

[0055] ⑤ Repair the attack image: If the input image is judged to be an attack image in step ④, the image is repaired through the PRN layer. The repair process is to pass the input adversarial sample image through a convolution layer, then through 5 residual modules and then through two convolution layers, and output the image after removing interference;

[0056] ⑥ Image partitioning: After steps ④ and ⑤, the image will be divided into S x S grids. If the center of an object falls in this grid, then this grid is responsible for predicting the object. Each network needs to predict the location information and confidence information of B BBoxes. One location information corresponds to four location information and one confidence information. The confidence represents the confidence of the object contained in the predicted object box and the accuracy of the prediction of this object.

[0057] ⑦Brelu activation function: All activation functions in the network are replaced by bounded leaky relu activation function;

[0058] ⑧ Repair the feature image: When the image feature information reaches the FPRN layer, the image feature layer is repaired adversarially. The feature map of the input adversarial sample image first passes through three residual modules and then a convolution layer, and the output is the feature map after removing interference;

[0059] ⑨ Detect object information: For each grid, two bounding boxes are predicted. After predicting S x S x 2 target windows, the target windows with lower probability are removed according to the threshold. Finally, the redundant windows are removed by maximum suppression to obtain the vehicle’s location information, category information, and confidence information.

[0060] PRNF (Pertrubation Rectifying Network Combine Feature) disturbance detection network:

[0061] Adversarial perturbation: Let I c ∈R d represents the distribution of (unperturbed) natural images in d-dimensional space, so that a class label is associated with each of its samples I c ~I c Let C(.) be a classifier (a deep network) that maps an image to its class label, i.e., C(I c ):I c →l∈R. If the vector ρ∈R d It is a universal adversarial perturbation for the classifier if the following constraints are satisfied:

[0062]

[0063] Where P(.) represents the probability. ||.|| p l represents the vector p norm, such that p∈[1,∞), δ∈(0,1] represents the fooling ratio, and ξ is a predefined constant.

[0064] PRNF loss function: The PRN layer is located at the beginning of the entire network. The joint network is trained to minimize the following cost, i.e., the loss function value of the PRNF network:

[0065]

[0066] in, and l i are the labels predicted by the joint network and the target network respectively, l i The labels must be computed for the distractor-free images. For N training examples, Calculate the loss, and θ p and b p Represents the weight and bias parameters of the PRN network layer. fp and b fp Represents the weights and bias parameters of the FPRN network layer. D(.) is responsible for calculating the distance between the output image and the uninterrupted image. i represents the output of the PRN layer, Indicates a distraction-free image. represents the output of the FPRN layer, Represents the feature map of the unperturbed image after passing through the network layer.

[0067] In this patent, we use the R-square method to calculate and D(.), in order to distinguish these two items, different symbols are used to represent them. The mean square error is part of the R-square calculation method. The mean square error formula is as follows:

[0068]

[0069]

[0070] In the expression of D(.), Represents the pixel in the image / feature map output by the perturbation correction network. i Represents the pixels on the original image / feature map obtained based on the original image.

[0071] Same reason The calculation formula is:

[0072]

[0073] The molecule is responsible for the calculation and l i The mean square error is The numerical value of .

[0074] like Figure 4The network structure of FPRN is shown in Figure 1. Res is a convolutional residual module, which consists of two convolutional layers. The input image first passes through three residual modules and then a convolutional layer. The input is the feature map of the adversarial sample image, and the output is the feature map after removing interference.

[0075] PRNF network training: During the training of the Yolo backbone network, the PRN layer is not included in the training process. When the Yolo backbone network reaches the training completion, the parameters of the Yolo backbone network are frozen (excluding the FPRN layer in the backbone network), that is, only the parameters of the PRN layer and the FPRN layer are updated in the following training process. The training data set is the normal image and the interference image generated by the normal image. In one iteration, the normal image is directly input into the target network (without passing through the PRN layer) to obtain the predicted label l i and Then the interference image is input into the PRN layer to obtain and Then update the network weight θ according to the calculation result of the loss function p and θ fp .

[0076] The complete prediction process of the Yolo-DAP network: When the SVM classifier, Yolo backbone network and PRNF network are all trained, Yolo-DAP can predict the image normally. When a picture is input, the SVM classifier first classifies the image. If the image is determined to be an adversarial sample, the image is input into the PRN layer. If the image is not determined to be an adversarial sample, the original image is directly input into the network. The network detects the objects in the image. Output the category and location information of the object in the image.

[0077] (2) The cloud computing server performs super-resolution reconstruction of the image in the cloud and uploads the license plate number, video frame corresponding time and current GPS information of the illegal vehicle to the cloud management platform.

[0078] (3) The cloud computing server sends messages to the slave vehicles around the violating vehicle based on the GPS information.

[0079] (4) In slave vehicle mode: When the slave vehicle receives the image information sent by the master vehicle, it first parses the image information to obtain the timestamp information transmitted from the cloud management server; then it compresses the video stream within the timestamp, and then moves the camera servo to collect information about surrounding vehicles to determine whether there is a target vehicle in the current field of view. If the target vehicle is still recognized in the picture, the camera servo is moved to a suitable shooting angle and shoots for another 5 seconds. The video data shot again is compressed and uploaded to the cloud computing server.

[0080] (5) The cloud computing server reconstructs the video stream information with super-resolution, and uses the video information of the master vehicle and the slave vehicle to form a chain of evidence for vehicle violation to re-identify the vehicle and determine the vehicle's position in the video image, and comprehensively judge whether the vehicle has violated the law from multiple angles; if the vehicle has violated the law and is not in the special vehicle database, the illegal vehicle information and the chain of evidence are saved in the cloud management platform database; if the vehicle's violation still cannot be determined and is not in the special vehicle database, it is uploaded to the manual channel for judgment; if the vehicle has not violated the law, the chain of evidence is deleted.

[0081] The cloud computing server is responsible for receiving information packets uploaded by each smart driving recorder. The information packets are mainly composed of two types of information and are sent twice. The first type is uncompressed information and the second type is compressed information. The first type of information is the first uncompressed information, which mainly includes the following information: the timestamp obtained by GPS when the driving recorder first identifies the vehicle, the identified license plate number and the GPS information of the vehicle. The second type of information is the compressed video clip from the time when the system determines that the vehicle is a violation to the time when it leaves the monitoring range or the violation ends, as well as the time information when it leaves the monitoring range or the violation ends.

[0082] The information packets are sent by two types of vehicles: a master vehicle (the vehicle that first discovers the violation) and a slave vehicle (the vehicle that completes the chain of evidence), wherein the master vehicle sends a complete information packet and the slave vehicle sends an information packet containing only the second type of information.

[0083] like Figure 5 The figure shows a flow chart of the cloud computing server. For the first type of information sent by the main vehicle, the cloud computing server will store the image information into a newly created evidence chain folder after receiving it, and will also send the parsed GPS information to the surrounding online slave vehicles at the same geographical location. After receiving the information, the slave vehicle will immediately start to identify the corresponding illegal vehicle and record the video evidence chain, and these videos will all have real-time timestamps.

[0084] For the second type of information, the video clips recorded by the device will first be compressed and then uploaded to the evidence chain folder corresponding to the illegal vehicle. The time information of leaving the monitoring range in the first type of information of the main vehicle will be synchronously sent to the slave vehicle in the above-mentioned geographical location. After receiving it, the slave vehicle will stop recording and collecting evidence, and compress the video information and send it to the corresponding folder (evidence chain folder).

[0085] After evidence collection is completed, the cloud computing server will unzip the compressed videos in the evidence chain folder and then reconstruct them with super-resolution. The video in the evidence chain will be re-identified to determine the specific vehicle, and the multi-angle video will be combined to make a second violation determination for the vehicle that was determined to be in violation by the vehicle system. If it is determined to be a violation, the evidence will be stored in the cloud management platform, otherwise the corresponding folder will be deleted, and the cases that cannot be determined will be submitted to manual review.

[0086] The present invention also discloses a method for evidence chain splicing, which is based on the evidence chain splicing of the vehicle re-identification algorithm, that is, a re-id method with geographic location information and characterization filter. The difference between this method and the existing vehicle re-identification algorithm is that: 1. Characterization feature filter is added to the Re-id algorithm; 2. Sample filtering with geographic location information is proposed and integrated into the new evaluation index DistAP; 3. The loss function of the Re-id algorithm is improved, and a new loss function BHTLoss is proposed. Figure 6 The figure shows the application flow chart of the vehicle re-identification algorithm of the present invention. When the evidence-collecting vehicle (main vehicle) finds that the vehicle in the field of view may have violated the rules, it will send information to the cloud computing server. The cloud computing server sends information to other vehicles (slave vehicles) around the evidence-collecting vehicle based on the GPS information of the evidence-collecting vehicle. After receiving the message, other vehicles (slave vehicles) around the evidence-collecting vehicle will take pictures of the illegal vehicle for evidence collection and compress the relevant video stream and upload it to the cloud computing server. After the cloud computing server collects the evidence information of all vehicles, it uses the re-id algorithm to process the evidence chain, finds the target vehicle from different angles in multiple videos, and comprehensively judges whether it violates the rules.

[0087] The specific steps include:

[0088] ① Extract high-level representation: extract high-level representation features from the image;

[0089] ② Characteristic feature filtering: After the image passes through the characteristic feature filter, the samples that are judged as non-target vehicles in the high-level representation are filtered out, leaving the samples with the same characteristic features as the target vehicle;

[0090] ③ Sample filtering layer with geographic location information: When the sample is input into the subsequent processing process, it will carry the location, angle and other information when the sample was collected. These information are input at the same time, and the corresponding network parameters will calculate these parameters to strengthen or weaken the confidence of the sample. The sample filtering layer will integrate the location, angle and other information into the image itself and serve as the input of the feature extraction network;

[0091] ④ Fusion of global feature extraction and local feature extraction: Next, the image enters the feature extraction stage. Feature extraction consists of two parts, namely global feature extraction and evenly aligned local feature extraction. The result of global extraction will be used to solve the ID loss of the classifier; the result of local extraction will be spliced ​​with the result of global extraction, and then input into the difficult sample mining module to solve the loss, which is used to measure the accuracy of the REID algorithm. When training the REID algorithm model, the model network will be updated according to the loss.

[0092] like Figure 7The figure shows a schematic diagram of the structure of the re-id algorithm used in the present invention. After the input image passes through the characterization feature filter, a sample with the same characterization feature as the target vehicle is left (for example, the car is red). Next, the image is subjected to feature extraction and pooling to obtain a feature map, which will be processed by two different pipelines. The first one will obtain the predicted ID of the target after multi-scale feature extraction and passing through a classifier. The predicted ID will participate in the calculation of the ID loss value; the second one will obtain the global loss after upsampling through a fully connected layer. The local loss is calculated from multiple feature maps stacked after the image is averaged. Among them, NUP is a general network unit, which consists of a convolution with a residual and a reshape module. Its function is to output the input to a specified size after the residual convolution, and the function of the Reshape module is to adjust the size of the vector.

[0093] Representational feature filters: Attention mechanisms for local representational features

[0094] The role of the feature filter is to screen the image before it enters the network and remove unnecessary images, thereby saving computing power. For example, if the target vehicle in the original image has a characteristic red color, in the detection image, the feature filter will remove the blue and orange parts of the detection image and retain the possible target red vehicle.

[0095] The characterization feature filter in the Re-id algorithm of the present application shares a feature extraction network with the backbone network.

[0096] The purpose of multi-scale feature extraction in solving the ID loss value is to extract target features from multiple scales to prevent prediction failures due to different distances and angles of the same target. Multi-scale feature extraction and Yolo-DAP share the same feature extraction network.

[0097] Equal alignment: Partial alignment pooling (PAP) solves the problem of different angles and relative positions of objects under different viewing angles. Through partial alignment pooling, the Re-ID algorithm can identify vehicles at different angles in the image.

[0098] In the local feature learning of the network, dynamic programming is used to calculate the shortest path to achieve feature alignment in different regions. The state transfer equation for dynamic programming to calculate the shortest path is:

[0099]

[0100] In the above formula, d ij is the local feature distance, S ij is the shortest distance from (1,1) to (i,j) in the distance matrix.

[0101] In Re-id,

[0102] TP: true possitive, the predicted truth value is true and the actual truth value is true;

[0103] TN: true negative, the predicted true value is false and the actual true value is false;

[0104] FP: false possitive, the predicted truth value is true but the actual truth value is false;

[0105] FN: false negative, the predicted true value is false but the actual true value is true;

[0106] Accuracy:

[0107] Accuracy = number of samples predicted correctly / total number of samples, that is, the proportion of samples predicted correctly (including positive samples predicted correctly and negative samples predicted correctly. The calculation formula for accuracy is:

[0108]

[0109] Among them: AllSamples represents all samples.

[0110] Precision: Precision rate: The precision rate indicates the accuracy of the prediction of a certain type of sample and can be used to indicate the accuracy of the detection. Its calculation formula is as follows:

[0111]

[0112] Among them, AllDetrction represents the number of samples of a certain category in the Ground Truth (actual true value).

[0113] Recall: Recall rate

[0114] The recall rate represents the ratio of the number of correctly predicted samples of a certain category to the number of actual true values ​​(sample labels) of all samples in that category.

[0115] AP: Average Precision

[0116] With Recall as the horizontal axis and Precision as the vertical axis, we can draw a PR (Precision-Recall) curve, and the area under the PR curve is defined as AP. By introducing the interpolation method, the formula for calculating AP is as follows:

[0117]

[0118] Among the evaluation indicators of the vehicle re-identification model:

[0119] Rank-1 (first hit rate): indicates the probability that the image with the highest similarity to the retrieval target in the candidate library is the target vehicle.

[0120] mAP (mean average precision): mAP can be used to measure the re-id algorithm more comprehensively. Assuming there are n pictures, the Re-ID algorithm will identify these n pictures and give a confidence score for each picture whether there is a retrieved vehicle. We will i It is called the picture where the retrieved vehicle appears. i The mAP will be high only when the overall confidence is high.

[0121] Sample filtering with geolocation information:

[0122] Add geographic location information dimension to the sample and introduce function DistAP:

[0123]

[0124] Where threshold is the threshold. The higher the threshold, the lower the correlation between DistAP and the geographic location. Sample.dist represents the distance between the sample and the target. P(.) is used to calculate Precision, and r(.) is used to calculate Recall. This function selects samples that are within a certain distance range from the target location. Obviously, the closer the distance to the target, the easier it is to capture the target again, so this method is particularly useful in multi-angle capture and re-identification.

[0125] Loss function: Improved Batch Hard Triplet Loss—BHTLoss

[0126] Batch Hard Triplet Loss (BHTLoss) is an improved triplet loss based on Hard sample mining (difficult sample mining, so-called difficult samples refer to image samples whose IDs are difficult to determine). The purpose of BHTLoss is to solve a defect of triplet loss: randomly selecting three images from the training set may result in the selection of simple samples, such as very similar positive sample pairs and very dissimilar negative sample pairs, which will limit the generalization ability of the model.

[0127] If there are two input images I1 and I2, their normalized feature vectors fI1 and fI2. We define the Euclidean distance between the feature vectors of these two images as:

[0128] d I1，I2 =||f I1 -fI2 ||2

[0129] Suppose picture a and picture p are a positive sample pair, and picture a and picture n are a negative sample pair.

[0130] The ternary loss is expressed as Z

[0131] Z=(d a，p -d a，n +α) +

[0132] Where (c)+ means max(c,0), where d a，p and d a，n They represent the Euclidean distance between the anchor sample and the positive sample pair and the Euclidean distance between the anchor sample and the negative sample pair. α is a hyperparameter that can be adjusted.

[0133] The core idea of ​​BHTLoss is to randomly select P IDs for each training batch, and randomly select K different pictures for each ID, that is, a batch contains P×K pictures. Then, for each picture a in each batch, we can select a positive sample with the farthest distance from a and a negative sample with the closest distance to a to form a triple. We define the picture set with the same ID as a as A, and the remaining picture set with different IDs as B, then the BHTLoss formula can be expressed as:

[0134]

[0135] Where c is a threshold parameter set manually. BHTLoss calculates the Euclidean distance between a and each image in a batch in the feature space, and then selects the distance d of the positive sample p that is farthest (least similar) from a. α，p The distance d to the nearest (most similar) negative sample n α，n To calculate the ternary loss. Usually BHTLoss works better than the traditional ternary loss.

[0136] The present invention also encrypts the transmitted information. In order to ensure the uniqueness of the identity and prevent the transmitted information from being forged or altered, we set a dedicated ID number for each device. The ID is bound to the vehicle license plate number and stored in the cloud database. After compressing the image, the ID is used as the key to encrypt the image. After receiving the encrypted data, the server uses the ID to decrypt it.

[0137] The present invention discloses a detection system for vehicle violation behavior, such as Figure 8As shown, it includes a binocular camera module, a central processing unit, a GPS positioning module, a steering gear control module, a cloud computing server and a cloud management platform. The binocular camera module is connected to the input and output of the central processing unit, and is used to collect video streams and visual ranging, and transmit them to the central processing unit. The GPS positioning module is connected to the input of the central processing unit, and is used to obtain the precise latitude and longitude information of the vehicle, and send it to the central processing unit. The steering gear control module is connected to the input and output of the central processing unit, and is used to receive the instructions of the central processing unit and the steering of the binocular camera. The voice interaction module is connected to the input and output of the central processing unit, and is responsible for the voice interaction with the user. The switch of the driving recorder can be controlled through voice interaction, and the user is reminded through the voice function when the vehicle ahead is detected to have illegal behavior or the pre-collision module detects that a collision may occur. The cloud computing server is connected to the input and output of the central processing unit, and is used to perform super-resolution reconstruction of the received image and re-identify the vehicle after reconstruction to determine the specific violation. The cloud management platform is connected to the input and output ends of the cloud computing server, and is used to manage vehicle violation information and to manually confirm the unconfirmed illegal vehicles. The central processor is used to detect targets on the video stream collected by the binocular camera module, send the collected video stream to the cloud computing server, and process the communication between various devices.

[0138] As a preferred embodiment, the present solution also includes a pre-collision detection module, the input end of the pre-collision detection module is connected to the output end of the binocular camera module, the output end of the pre-collision detection module is connected to the input end of the central processing unit, the pre-collision detection module is used to obtain ranging information from the binocular camera module, when the distance between the front vehicle and the vehicle is lower than a threshold, the signal is sent to the central processing unit, and the central processing unit issues a pre-collision warning through the voice interaction module.

[0139] This solution also includes a stabilizer module, the output end of which is connected to the input end of the binocular camera module. The stabilizer module is arranged on the binocular camera module and is responsible for stabilizing the camera to prevent the inability to collect video streams and visual ranging due to vehicle bumps.

[0140] The present solution also includes a solar power supply system, the input end and the output end of the solar power supply system are respectively connected to the input end and the output end of the central processing unit, and are used to power various devices of the driving recorder.

[0141] The above embodiments are only specific examples for further describing the purpose, technical solutions and beneficial effects of the present invention, and the present invention is not limited thereto. Any modification, equivalent replacement, improvement, etc. made within the scope of the disclosure of the present invention are included in the protection scope of the present invention.

Claims

1. A method for detecting vehicle violations, characterized in that: The following steps are involved: (1) The main vehicle collects video streams in the driving environment and uses a target detection algorithm to detect targets in the video streams to determine whether there are any illegal vehicles in the video streams; When it is determined that there is a violation, it is also determined whether the current road is in a special state. If the road is not in a special state, the user is reminded that the vehicle ahead may have violated the law, and the license plate number of the vehicle is identified and the time period of the violation is recorded. The relevant video of the detected illegal vehicle is compressed and uploaded to the cloud computing server together with the GPS information and time information. (2) The cloud computing server reconstructs the image through super-resolution in the cloud and uploads the license plate number, time corresponding to the video frame, and current GPS information of the illegal vehicle to the cloud management platform; (3) The cloud computing server sends messages to the slave vehicles around the violating vehicle based on the GPS information; (4) After receiving the message from the vehicle, the video information shot during the violation time is compressed and uploaded to the cloud computing server; (5) The cloud computing server reconstructs the video stream information with super-resolution, and the violation evidence chain formed by the video information of the master vehicle and the slave vehicle is used to re-identify the vehicle and determine the position of the vehicle in the video screen, and comprehensively judge whether the vehicle has violated the law from multiple angles; if the vehicle has violated the law and is not in the special vehicle database, the illegal vehicle information and evidence chain are saved in the cloud management platform database; if the vehicle violation situation still cannot be determined and is not in the special vehicle database, it is uploaded to the manual channel for judgment; if the vehicle has not violated the law, the violation evidence chain is deleted.

2. A method for detecting vehicle violations according to claim 1, characterized in that: When the slave vehicle receives the image information sent by the master vehicle, it first parses the image information to obtain the timestamp information transmitted from the cloud management server; then it compresses the video stream in the timestamp, and then moves the camera servo to collect information about surrounding vehicles to determine whether there is a target vehicle in the current field of view. If the target vehicle is still recognized in the picture, the camera servo is moved to a suitable shooting angle and shoots for another 5 seconds. The video data shot again is compressed and uploaded to the cloud computing server.

3. A method for detecting vehicle violations according to claim 1 or 2, characterized in that: In step (1), if the road is in a special state, it is determined whether the vehicle violates the rules when the road is in a special state. If so, the license plate number of the vehicle is identified and the time period of the violation is recorded; and the relevant video frames of the detected violating vehicle are compressed and uploaded to the cloud computing server.

4. A method for detecting vehicle violations according to claim 1 or 2, characterized in that: The target detection algorithm used in step (1) is the YOLO-DAP algorithm, which includes the following steps: ① Train the basic YOLO recognition network: freeze the parameters of the FPRN layer and the PRN layer to train a neural network that can normally recognize objects; ②Training the SVM classifier: First, organize the adversarial sample image dataset, which contains normal images and images after interference. The normal image corresponds to the output label 0, and the image after interference is the adversarial image. The model input is the interference data, that is, the interference pixel after the adversarial image and the normal image are subtracted according to the vector operation. If the output label is 0, it means it is not an interference pixel, and if the output label is 1, it means it is an interference pixel. After multiple iterations, when the SVM classifier can perform well in the test dataset, stop training; ③Training the PRN layer and the FPRN layer: Freeze the parameters of the Yolo backbone network except the FPRN layer, that is, only update the parameters of the PRN layer and the FPRN layer in the following training process. The training data set is normal images and interference images generated by normal images. In one iteration, the normal image is directly input into the target network to obtain the predicted label, that is, whether the image is a normal image or an image that will affect the detection network after malicious disturbance. Then, the interference image is input into the PRN layer and then into the target detection network to obtain the location information, confidence information and category information of vehicles and pedestrians in the image. Finally, the network weights are updated according to the calculation results of the loss function. ④ Classify the attack image: Use the trained SVM classifier to classify the input image to determine whether the image is an attack image; ⑤ Repair the attack image: If the input image is judged to be an attack image in step ④, the image is repaired through the PRN layer. The repair process is to pass the input adversarial sample image through a convolution layer, then through 5 residual modules and then through two convolution layers, and output the image after removing interference; ⑥ Image partitioning: After steps ④ and ⑤, the image will be divided into S x S grids. If the center of an object falls in this grid, then this grid is responsible for predicting the object. Each network needs to predict the location information and confidence information of B BBoxes. One location information corresponds to four location information and one confidence information. The confidence represents the confidence of the object contained in the predicted object box and the accuracy of the prediction of this object. ⑦Add the Brelu activation function to the YOLO-DAP network: all activation functions in the network are replaced with bounded leakyrelu activation functions; ⑧ Repair the feature image: When the image feature information reaches the FPRN layer, the image feature layer is repaired adversarially. The feature map of the input adversarial sample image first passes through 3 residual modules and then a convolution layer, and the output is the feature map after removing interference; ⑨ Detect object information: For each grid, two bounding boxes are predicted. After predicting S x S x 2 target windows, the target windows with lower probability are removed according to the threshold. Finally, the redundant windows are removed by maximum suppression to obtain the vehicle’s location information, category information, and confidence information.

5. A method for detecting vehicle violations according to claim 1 or 2, characterized in that: The vehicle re-identification method used is a re-id method with geographic location information and characterization filter, which includes the following steps: ① Extract high-level representation: Extract high-level representation features from the image; ② Characteristic feature filtering: After the image passes through the characteristic feature filter, the samples that are judged as non-target vehicles in the high-level representation are filtered out, leaving the samples with the same characteristic features as the target vehicle; ③ Sample filtering layer with geographic location information: When the sample is input into the subsequent processing process, it will carry the location and angle information when the sample was collected. These information are input at the same time, and the corresponding network parameters will calculate these parameters to strengthen or weaken the confidence of the sample. The sample filtering layer will integrate the location and angle information into the image itself and serve as the input of the feature extraction network; ④ Fusion of global feature extraction and local feature extraction: Next, the image enters the feature extraction stage. Feature extraction consists of two parts: global feature extraction and evenly aligned local feature extraction. The result obtained from global extraction will be used to solve the ID loss of the classifier; the result obtained from local extraction will be spliced ​​with the result obtained from global extraction, and then input into the difficult sample mining module to solve the loss.

6. A vehicle violation detection system, the detection system executing the detection method according to any one of claims 1 to 5, characterized in that: The detection system comprises Binocular camera module, used to collect video streams and visual distance measurement; The central processing unit is used to detect targets on the video stream collected by the binocular camera module, compress the collected video stream and send it to the cloud computing server, and process the communication between various devices; GPS positioning module, used to obtain the precise latitude and longitude information of the vehicle and send it to the central processor; The steering gear control module is used to control the steering of the binocular camera; Voice interaction module, responsible for voice interaction with users; A cloud computing server is used to perform super-resolution reconstruction on the received images and to perform vehicle re-identification on the reconstructed images to determine specific violations; The cloud management platform is used to manage vehicle violation information and to manually confirm the illegal vehicles that cannot be confirmed.

7. A vehicle violation detection system according to claim 6, characterized in that: It also includes a pre-collision detection module, which is used to obtain ranging information from the binocular camera module and issue a pre-collision warning when the distance between the front vehicle and the vehicle is lower than a threshold.

8. A vehicle violation detection system according to claim 6, characterized in that: It also includes a stabilizer module, which is arranged on the binocular camera module and is responsible for stabilizing the camera to prevent the inability to collect video streams and visual ranging due to vehicle bumps.

9. A vehicle violation detection system according to any one of claims 6 to 8, characterized in that: It also includes a solar power supply system for supplying power to various devices of the driving recorder.

Images