Vehicle violation identification method, device, electronic device and storage medium

Abstract

The present application provides a vehicle violation identification method, device, electronic device and storage medium, including: performing multi-target detection on the to-be-processed images in the video stream sequence frames associated with the vehicle driving situation, the detection targets include a first target and N second targets, the first target is a vehicle object, and the second target is an object associated with the vehicle violation that triggers the first target; for the to-be-processed image, according to the position information of the detection frame corresponding to the detection target, establish a binding relationship between the second target and the first target to which it belongs; update the binding relationship according to the tracking situation of the detection target; for each first target, count the number of image frames corresponding to the different categories of the first target and the vehicle violation triggered by the matching second target; in response to the number of image frames corresponding to the vehicle violation meeting the preset conditions, output an alarm prompt. The present application can automatically and effectively identify vehicle violations and improve the operating efficiency of the recognition system.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, electronic device and storage medium for identifying vehicle violation behaviors. Background Art

[0002] As a low-carbon and environmentally friendly means of transportation, electric vehicles are increasingly used. With the continuous increase in the number of electric vehicles, the safety of electric vehicles has also attracted people's attention. In order to ensure the safety of pedestrians and vehicles, relevant departments are paying more and more attention to the rectification of electric vehicle violations, which mainly include electric vehicles on the road without licenses, unauthorized modifications, failure to wear safety helmets as required, overloading, etc. With the development of artificial intelligence technology, methods that rely on surveillance videos and use image recognition technology to automatically identify electric vehicle violations are also increasingly being used.

[0003] In the prior art, when identifying electric vehicle violations based on image recognition technology, it usually only supports the identification of not wearing a helmet and overloading. When identifying the behavior of not wearing a helmet, the person and the electric vehicle are first detected, and then the helmet ROI (Region of Interest) area is determined, and the helmet detection model is used for identification. The long process leads to low recognition efficiency.

[0004] At present, when identifying violations of electric vehicles, the detection results of a single image are usually used as the basis for alarms. Pedestrians passing by electric vehicles are mistakenly identified as belonging to the electric vehicle, and the ownership of people is misjudged when multiple electric vehicles approach. This can easily trigger false alarms, resulting in low recognition accuracy for not wearing a helmet and overloading. Summary of the invention

[0005] In view of the above problems, the embodiments of the present application provide a vehicle violation identification method, device, electronic device and storage medium that overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, an embodiment of the present application provides a method for identifying vehicle violation behavior, including:

[0007] In the case of acquiring a video stream sequence frame associated with a vehicle driving condition, performing multi-target detection on an image to be processed in the video stream sequence frame, the detected targets including a first target and N second targets, N being an integer greater than or equal to 1, the first target being a vehicle object, the second target being an object associated with a vehicle violation that triggers the first target, and the image to be processed corresponds to at least one first target;

[0008] For the image to be processed, establishing a binding relationship between the second target and the first target to which it belongs according to the position information of the detection frame corresponding to the detection target determined by multi-target detection;

[0009] Update the binding relationship according to the tracking status of the detection target in the video stream sequence frame;

[0010] For each first target, counting the number of image frames of the to-be-processed images corresponding to different categories of vehicle violations of the first target, wherein the vehicle violation of the first target is triggered based on a second target with a subordinate relationship;

[0011] In response to the number of image frames corresponding to at least one type of vehicle violation of the first target meeting a preset condition, an alarm prompt is output.

[0012] In a second aspect, an embodiment of the present application provides a vehicle violation identification device, comprising:

[0013] a detection module, configured to perform multi-target detection on an image to be processed in a video stream sequence frame associated with a vehicle driving condition, when acquiring the video stream sequence frame associated with the vehicle driving condition, wherein the detection targets include a first target and N second targets, where N is an integer greater than or equal to 1, the first target is a vehicle object, the second target is an object associated with a vehicle violation that triggers the first target, and the image to be processed corresponds to at least one first target;

[0014] An establishing module, used for establishing, for the image to be processed, a binding relationship between the second target and the first target to which it belongs according to the position information of the detection frame corresponding to the detection target determined by multi-target detection;

[0015] An updating module, used for updating the binding relationship according to the tracking status of the detection target in the video stream sequence frame;

[0016] A statistics module, for counting, for each first target, the number of image frames of the to-be-processed images corresponding to different categories of vehicle violations of the first target, wherein the vehicle violations of the first target are triggered based on a second target with a subordinate relationship;

[0017] The output module is used to output an alarm prompt in response to the number of image frames corresponding to at least one type of vehicle violation of the first target meeting a preset condition.

[0018] In a third aspect, an embodiment of the present application provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the vehicle violation identification method described in the first aspect.

[0019] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the vehicle violation identification method described in the first aspect are implemented.

[0020] The technical solution of the embodiment of the present application performs multi-target detection on the image to be processed in the sequence frames of the video stream, establishes a binding relationship between the second target and the first target according to the position information of the detection frame corresponding to the detection target, updates the binding relationship based on the tracking situation of the detection target, counts the number of image frames corresponding to different categories of vehicle violations triggered by the first target based on the second target with a subordinate relationship, and outputs an alarm prompt when the number of image frames corresponding to the vehicle violation meets the conditions. It can realize automatic and effective identification of multiple types of vehicle violations based on multi-target detection, detection target binding and tracking, and improve the operating efficiency of the recognition system; and by making full use of the time and space information of each detection target, it can effectively eliminate the situation of misbinding of the detection target, improve the binding accuracy of the detection target, and then ensure the accuracy of violation identification; at the same time, when the number of image frames corresponding to the vehicle violation meets the conditions, an alarm is issued, which can effectively reduce the situation of false alarms. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram showing a method for identifying vehicle violation behavior provided in an embodiment of the present application;

[0022] Figure 2 A flowchart showing an implementation of a vehicle violation identification method provided in an embodiment of the present application;

[0023] Figure 3 A flow chart showing a method for identifying vehicle violations in a specific scenario provided by an embodiment of the present application;

[0024] Figure 4 A schematic diagram showing a vehicle violation identification device provided in an embodiment of the present application;

[0025] Figure 5 A structural block diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0027] It should be understood that the references to "one embodiment" or "an embodiment" throughout the specification mean that the specific features, structures, or characteristics associated with the embodiment are included in at least one embodiment of the present application. Therefore, the references to "in one embodiment" or "in an embodiment" appearing throughout the specification do not necessarily refer to the same embodiment. In addition, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0028] In the various embodiments of the present application, it should be understood that the size of the serial numbers of the following processes does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0029] The vehicle violation identification method provided in the embodiment of the present application is applicable to the identification of traffic violations of two-wheeled electric vehicles, and can also be applied to the identification of violations of other types of vehicles, such as motorcycles, three-wheeled electric vehicles, four-wheeled electric vehicles and automobiles. For motorcycles, for example, license plate violation identification (such as failure to register), overload violation identification, failure to wear a helmet violation identification, and additional parts violation identification can be performed; for three-wheeled electric vehicles and four-wheeled electric vehicles, for example, license plate violation identification (such as failure to register), overload violation identification, and additional parts violation identification can be performed; for automobiles, for example, license plate violation identification (such as license plate obstruction), overload violation identification, and modification violation identification can be performed. The embodiment of the present application introduces the vehicle violation identification method using a two-wheeled electric vehicle as an example.

[0030] The vehicle violation identification method provided in the embodiment of the present application is applied to the identification system, such as Figure 1 As shown, the method includes:

[0031] Step 101: When obtaining a video stream sequence frame associated with a vehicle driving condition, multi-target detection is performed on an image to be processed in the video stream sequence frame, where the detection targets include a first target and N second targets, where N is an integer greater than or equal to 1, the first target is a vehicle object, the second target is an object associated with a vehicle violation that triggers the first target, and the image to be processed corresponds to at least one first target.

[0032] First, a video stream sequence frame associated with the vehicle driving condition is obtained, and the video stream sequence frame is composed of extracting frames from the video stream according to a certain frame extraction interval and arranging them in the order of the frame extraction. The obtained video stream sequence frames may include the driving conditions of two-wheeled electric vehicles, motorcycles, three-wheeled electric vehicles, four-wheeled electric vehicles, cars and other vehicles.

[0033] For the acquired video stream sequence frames, multi-target detection is performed on each frame of the image to be processed in the video stream sequence frames, and the corresponding detection targets include a first target and N second targets, where N is an integer greater than or equal to 1. Among them, multi-target detection is to detect and identify multiple objects in the image and give their location information. Unlike image recognition, target detection requires the positioning of objects, that is, giving the position and size of objects in the image. For example, multiple pedestrians, vehicles and other objects are detected and located in a street view image. Target detection usually includes two tasks, namely target positioning and target classification. Target positioning refers to accurately locating the position and size of the target in the image, while target classification is to classify the located targets.

[0034] In this embodiment, the first target is a vehicle object, and the second target is an object associated with triggering a vehicle violation. For example, in the scenario where the first target is a two-wheeled electric vehicle, the N second targets include: a human body object, a license plate object, an accessory object, a head object without a helmet, and a head object with a helmet. The accessory object is an illegal object installed on the electric vehicle object, such as a windshield or a parasol. Accordingly, the vehicle violations triggered by the second target include, for example: an electric vehicle on the road without a license plate, an electric vehicle equipped with a parasol or a windshield, a person on the electric vehicle not wearing a helmet, and an electric vehicle overloaded.

[0035] For each frame of the image to be processed, it corresponds to at least one first target. Usually, the second target corresponding to the image to be processed is one or more, and for each second target, the corresponding quantity can be one or more. In special cases, there is no second target in the image to be processed, such as the second target is a license plate, and the situation of driving without a license plate can be identified for the image to be processed.

[0036] Step 102: for the image to be processed, a binding relationship between the second target and the first target to which it belongs is established according to the position information of the detection frame corresponding to the detection target determined by the multi-target detection.

[0037] When performing multi-target detection, each detection target corresponds to a detection frame, and the position information of the detection frame corresponding to the detection target can be determined based on the multi-target detection. After performing multi-target detection on the image to be processed, for the image to be processed, a binding relationship between the second target and the first target to which it belongs is established based on the position information of the detection frame corresponding to the detection target. Among them, for multiple frames of images to be processed in the video stream sequence frame, multi-target detection can be performed sequentially based on the time order.

[0038] For example, in the two-wheeled electric vehicle scenario, the human body, license plate, windshield, and parasol are bound to the electric vehicle through the positional relationship between the detection frames. Since the human head has an inseparable binding relationship with the human body, the head is matched with the electric vehicle to establish a binding relationship between the human body, license plate, windshield, parasol, head (including heads wearing helmets and heads not wearing helmets) and the electric vehicle.

[0039] Step 103: Update the binding relationship according to the tracking status of the detection target in the video stream sequence frames.

[0040] During multi-target detection, the corresponding detection targets are tracked, that is, each detection target is tracked in the video stream sequence frame using the corresponding tracking ID (Identity document). Target tracking is one of the important research directions in the field of computer vision. Its goal is to detect, extract, identify and track objects of interest in continuous images, so as to obtain relevant parameters of the target objects, such as position, speed, scale, trajectory, etc., and further process and analyze them to understand the behavior of the target objects or complete higher-level tasks.

[0041] In this embodiment, the detection target is tracked based on the tracking ID in the video stream sequence frame, the motion trajectory of the tracking target (the tracked detection target) is obtained, and the binding relationship between the second target and the first target is updated according to the tracking situation of the detection target (the motion trajectory of the tracking target), so as to make full use of the space and time information based on the principle that components that do not belong to the same whole will eventually separate over time, and release the wrong binding according to the motion trajectory of the tracking target. By updating the binding relationship, the accuracy of binding the first target and the second target through the position relationship can be improved.

[0042] As a specific implementation plan, in the two-wheeled electric vehicle scenario, a tracking ID is assigned to each electric vehicle in the image to be processed, and a tracking ID is assigned to each human body, license plate, windshield, parasol, and human head, and the detection target is tracked in the video stream sequence frame using the tracking ID. Based on the tracking of the detection target (human body, electric vehicle, windshield, parasol, license plate, human head), the wrong binding is released using the movement trajectory of the tracking target, which can reduce the misbinding situations such as pedestrians passing by the electric vehicle being mistakenly identified as pedestrians belonging to the electric vehicle, pedestrians holding umbrellas being mistakenly bound to electric vehicle parasols, and multiple close-range electric vehicles being mixed and bound, greatly improving the accuracy of binding electric vehicles and other targets through positional relationships.

[0043] Step 104: for each first target, count the number of image frames of the to-be-processed images corresponding to different categories of vehicle violations of the first target, where the vehicle violations of the first target are triggered based on the second target with a subordinate relationship. Since the to-be-processed images correspond to at least one first target, for each first target, count the number of image frames of the to-be-processed images corresponding to each category of vehicle violations of the first target, so as to accumulate the number of image frames of each category of vehicle violations.

[0044] Among them, the vehicle violation of the first target is triggered based on the second target with a subordinate relationship. The second target with a subordinate relationship with the first target includes the second target bound to the first target, and the second target that should be subordinate to the first target that is not identified by the multi-target detection. The absence of the second target will trigger the vehicle violation. For example, the second target that should be subordinate to the first target that is not identified by the multi-target detection is a license plate object. If the license plate object is missing, the multi-target detection cannot recognize the license plate, but the license plate should be subordinate to the vehicle. The absence of the license plate will trigger the vehicle violation of driving without a license.

[0045] The following examples introduce the situations where the second target with a subordinate relationship to the first target triggers vehicle violations. The missing license plate as the second target will trigger the subordinate electric vehicle to generate a vehicle violation, the overloading of the human body as the second target will trigger the bound electric vehicle to generate a vehicle violation, and the installation of a parasol as the second target will trigger the bound electric vehicle to generate a vehicle violation.

[0046] Since the binding relationship between the first target and the second target can be updated and the wrong binding relationship can be released based on the motion trajectory of the detection target, the accuracy of the binding between the first target and the second target can be guaranteed. When accumulating the number of image frames of the image to be processed in which there is a vehicle violation, it is necessary to consider the previously counted misbinding situation, and subtract the number of image frames of the image to be processed corresponding to the misbinding situation from the accumulated number of image frames. As an example, for the image to be processed, based on the binding relationship between parasol B and electric vehicle A, it is identified that the electric vehicle A in the image to be processed has a vehicle violation of installing parasol B, then for the vehicle violation corresponding to the installation category of electric vehicle A, the corresponding number of image frames of the image to be processed will be accumulated. When parasol B is identified as a parasol held by a pedestrian during the tracking process of the detection target, the binding relationship is updated, the historical binding is released, and the number of image frames of the electric vehicle violation accumulated by the historical binding is subtracted accordingly.

[0047] Step 105 : In response to the number of image frames corresponding to at least one type of vehicle violation of the first target meeting a preset condition, outputting an alarm prompt.

[0048] For each first target, in response to the number of image frames corresponding to at least one type of vehicle violation of the first target reaching at least one matching frame number threshold, an alarm prompt including the location information of the alarm target and the violation category is output, so that an alarm is only issued when the accumulated number of image frames with vehicle violations reaches the preset frame number threshold, which can effectively reduce false alarms.

[0049] The warning targets include a first target and a second target that triggers the vehicle violation. After the position information corresponding to the first target and the second target that triggers the vehicle violation is output, the first target and the second target that triggers the vehicle violation can be marked on the image based on the position information of the warning target, such as by using a frame of a specific color to frame the first target and the second target, so that relevant personnel can quickly and accurately understand the violation targets based on the marking situation. And by outputting the violation category, relevant personnel can understand the specific category to which the violation of the first target belongs.

[0050] The above implementation scheme of the present application performs multi-target detection on the image to be processed in the sequence frames of the video stream, establishes a binding relationship between the second target and the first target according to the position information of the detection frame corresponding to the detection target, updates the binding relationship based on the tracking situation of the detection target, counts the number of image frames corresponding to different categories of vehicle violations triggered by the first target based on the second target with a subordinate relationship, and outputs an alarm prompt when the number of image frames corresponding to the vehicle violation meets the conditions. It can realize automatic and effective identification of multiple types of vehicle violations based on multi-target detection, detection target binding and tracking, and improve the operating efficiency of the recognition system; and by making full use of the time and space information of each detection target, it can effectively eliminate the situation of misbinding of the detection target, improve the binding accuracy of the detection target, and then ensure the accuracy of violation identification; at the same time, when the number of image frames corresponding to the vehicle violation meets the conditions, an alarm is issued, which can effectively reduce the situation of false alarms.

[0051] The following describes the case where multi-target detection is performed on the image to be processed. When a video stream sequence frame associated with the vehicle driving situation is obtained, multi-target detection is performed on the image to be processed in the video stream sequence frame, including:

[0052] In response to the acquired video stream sequence frames, performing image preprocessing on the initial image in the video stream sequence frames to determine an image to be processed;

[0053] The image to be processed in the video stream sequence frame is input into the target detection model, and multi-target detection is performed on the image to be processed based on the target detection model to determine the position information of the detection box corresponding to the detection target; wherein the video stream sequence frame is determined by extracting frames of video stream data associated with the vehicle driving condition.

[0054] After obtaining the video stream data associated with the vehicle's driving conditions (such as the road condition monitoring video stream), the video stream data is extracted to obtain a video stream sequence frame, and then each initial image in the video stream sequence frame is subjected to image preprocessing, including image resizing, normalization and other cleaning operations, to obtain the image to be processed. By extracting the video stream, some images can be extracted from the video stream for analysis, which can reduce the workload while ensuring the analysis effect; by preprocessing the initial image, a standard image that meets the requirements can be provided, so that the standard image can be processed as the image to be processed.

[0055] After obtaining the image to be processed, the image to be processed in the video stream sequence frame is input into the trained target detection model in chronological order, and multi-target detection is performed on the image to be processed based on the target detection model to determine the position information of the detection box corresponding to the detection target.

[0056] The target detection model in this embodiment adopts the YOLOv5 (You Only Look Once version5) model, which can also be replaced by other target detection models as needed. YOLOv5 is an end-to-end deep learning model that can detect and locate targets directly from the original image. It is used to identify and locate objects in videos or images. It is the latest version of the YOLO algorithm and is improved on the basis of YOLOv4. YOLOv5 uses convolutional neural networks to learn the features of objects in images, and uses multi-scale prediction and grid segmentation to detect and locate targets. Its advantage is that it can run at high speed and work well on different image resolutions. In general, YOLOv5 is an efficient target detection model that can be applied to many different scenarios, including autonomous driving, robot perception, image analysis, etc.

[0057] Since the video stream sequence frames include multiple frames of images to be processed, the images to be processed can be input into the target detection model in chronological order, and the target detection model processes each frame of the image to be processed to locate the detection target in the image to be processed.

[0058] As a specific implementation plan, in the two-wheeled electric vehicle scenario, the target detection model supports the detection of seven detection targets: electric vehicles, human bodies, heads without helmets, heads with helmets, electric vehicle license plates, windshields, and electric vehicle parasols. The target detection model performs multi-target detection on the image to be processed and determines the position information of the detection frame corresponding to each detection target. For example, if the number of electric vehicles is 5, the position information of the detection frame is determined for each electric vehicle.

[0059] The above implementation scheme inputs the image to be processed in the video stream sequence frame into the target detection model, performs multi-target detection based on the target detection model, and outputs the position information of the detection box corresponding to the detected target, so as to realize the use of the target detection model to provide the position of the detected target in the image.

[0060] The following is an introduction to a scheme for establishing a binding relationship. For an image to be processed, according to position information of a detection frame corresponding to a detection target determined by multi-target detection, a binding relationship between a second target and its first target is established, including:

[0061] In the image to be processed, for each first detection frame, traverse each second detection frame, and calculate the degree of overlap between the second detection frame and the first detection frame based on the position information of the first detection frame and the second detection frame, the first detection frame is the detection frame corresponding to the first target, and the second detection frame is the detection frame corresponding to the second target;

[0062] Identify a second target that directly matches the first target based on the degree of overlap, and determine a second target that indirectly matches the first target based on the subordinate relationship between the second targets;

[0063] A binding relationship between the matching first target and the second target is established.

[0064] After using the target detection model to perform multi-target detection on the image to be processed, the position information of the detection frame corresponding to each detection target in the image to be processed can be identified. Then, in the image to be processed, for each first detection frame corresponding to the first target, each second detection frame (the detection frame corresponding to the second target) is traversed, and the overlap degree between the second detection frame and the first detection frame is calculated based on the position information of the first detection frame and the second detection frame. The overlap degree of the detection frame is measured based on IoU (Intersection over Union). IoU is a concept used in target detection. It calculates the overlap rate of the "predicted border" and the "real border", that is, the ratio of their intersection and union. By calculating the IoU of two detection frames (such as rectangular frames), the overlap degree of the two detection frames can be evaluated. The IoU value is between 0-1. When IoU is 0, it means that the two detection frames have no intersection. When IoU is 1, it means that the two detection frames completely overlap.

[0065] Based on the calculated overlap degree, the second target that directly matches the first target is identified, wherein the first target and the second target whose overlap degree of the detection frame is greater than a preset threshold are directly matched detection targets. After determining the second target that directly matches the first target, the second target that indirectly matches the first target is determined based on the subordinate relationship between the second targets, thereby determining the second target that directly matches and indirectly matches the first target, and establishing a binding relationship between the matching first target and the second target. Among them, the second target that indirectly matches the first target (such as the second target A1) is subordinate to the second target that directly matches the first target (such as the second target A), that is, after determining the second target that directly matches the first target, it is determined that other second targets subordinate to the second target are indirectly matched with the first target.

[0066] As a specific implementation scheme, in the two-wheeled electric vehicle scenario, for each electric vehicle detection frame, each human body frame, electric vehicle license plate frame, windshield frame, and electric vehicle parasol frame are traversed. When the IoU between a certain detection frame and the electric vehicle detection frame is greater than the preset threshold, the detection target corresponding to the detection frame is bound to the electric vehicle, so as to realize the binding of detection targets such as human body, license plate, windshield, parasol, etc. with the electric vehicle based on the position relationship. It should be noted that when the number of license plates, windshields, and electric vehicle parasols bound to the same electric vehicle exceeds 1, only the detection target with the largest IoU with the electric vehicle detection frame is bound, and the preset thresholds corresponding to detection targets of different categories may be different (such as the license plate size is small, and the corresponding preset thresholds can be set based on the actual situation). Since the human head is subordinate to the human body, after the human body is bound to the electric vehicle, the head without a helmet and the head with a helmet are bound to the human body based on the subordinate relationship between the human head and the human body, and the binding relationship between the electric vehicle and the head is determined by the binding relationship between the electric vehicle and the human body.

[0067] In the above implementation scheme, after determining the position information of the detection frame based on multi-target detection, the overlap degree of the detection frame is calculated according to the position information of the detection frame, and the second target that directly or indirectly matches the first target is identified by using the overlap degree and the subordinate relationship between the second target, thereby establishing a binding relationship between the matching first target and the second target.

[0068] After the binding relationship is established, it is necessary to update the binding relationship. When the binding relationship is updated according to the tracking status of the detection target in the video stream sequence frame, it includes:

[0069] Track the detected target in the video stream sequence frames based on the tracking algorithm;

[0070] Identify whether the second target remains bound to the first target historically bound based on the tracking situation;

[0071] In response to the second target not maintaining a binding state with the historically bound first target, updating the binding relationship.

[0072] In this embodiment, when tracking the detected target, the DeepSORT (Deep learning based SORT) tracking algorithm is used, and other tracking algorithms can also be replaced as needed. Among them, DeepSORT is an improved version of the SORT (Simple Online and Realtime Tracking) multi-target tracking algorithm, which improves the accuracy of tracking objects that are occluded for a long time and reduces the phenomenon of frequent ID switching. On the basis of the traditional algorithm, the SORT algorithm uses Kalman filtering to process the correlation of each frame and uses the Hungarian algorithm for correlation measurement, which improves its detection performance by dozens of times. However, the problem of frequent ID switching of the SORT algorithm is more obvious, that is, the SORT algorithm is only suitable for objects with less occlusion and relatively stable motion. DeepSORT realizes correlation measurement by combining more accurate measurements of action and appearance information, and uses convolutional neural networks to extract features, which increases the robustness to missing and occlusion. At the same time, it is easy to implement and efficient, and is also suitable for online scenarios.

[0073] The DeepSORT algorithm is used to track the detection target using the tracking ID in the video stream sequence frame, and the position information of the detection box is recorded during tracking. Based on the tracking situation, it is identified whether the second target remains bound to the first target that is historically bound. That is, for each frame of the tracked image to be processed, it is determined whether the second target in the current frame image remains bound to the first target that is historically bound. If not, it means that the historical binding is likely to be a misbinding, and the historical binding is released to update the binding relationship. For the image to be processed corresponding to the misbinding situation, the identified vehicle violation associated with the misbinding situation needs to be updated.

[0074] As a specific implementation scheme, each detection target corresponds to a tracking ID. In the two-wheeled electric vehicle scenario, for each frame of the image to be processed, it is determined whether the human frame, electric vehicle license plate frame, windshield frame, electric vehicle parasol frame in the current frame image are still bound to the electric vehicle that has been historically bound. If they are not bound, it indicates that the historical binding is likely to be a mistake, and the historical binding needs to be released. The binding relationship between the head without a helmet, the head with a helmet, and the electric vehicle is updated based on the binding status of the human body and the electric vehicle. For the image to be processed whose binding relationship has been released, the corresponding vehicle violation status needs to be updated.

[0075] In the above implementation scheme, whether the historically bound second target and the first target remain in a bound state is identified based on the tracking of the detection target. If not, the established binding relationship is updated to correct the erroneous binding situation in time through the update of the binding relationship, thereby improving the accuracy of binding the first target and the second target through the position relationship.

[0076] The following is an introduction to a scheme for counting the number of image frames corresponding to each type of vehicle violation of the first target. When counting the number of image frames of the to-be-processed images corresponding to each type of vehicle violation of the first target, the method includes:

[0077] For each first target, identifying in the video stream sequence frames an image to be processed in which a vehicle violation behavior associated with the first target exists and a violation category corresponding to the image to be processed;

[0078] Based on the recognition situation, the number of image frames corresponding to each type of vehicle violation existing in the first target is counted.

[0079] For each first target in the detection target, an image to be processed in which a vehicle violation behavior associated with the current first target exists is identified in the video stream sequence frame, and the violation category corresponding to the image to be processed is identified to determine one or more violation categories corresponding to the image to be processed. After identifying the image to be processed associated with the current first target and the violation category corresponding to the image to be processed, the number of image frames corresponding to each type of vehicle violation behavior existing in the first target is counted based on the identification situation, so as to compare the number of image frames corresponding to each type of vehicle violation behavior with the matching frame number threshold. When it is determined that the number of image frames corresponding to a certain type of vehicle violation behavior reaches the matching frame number threshold, an alarm prompt including the location information of the alarm target and the violation category is output, so as to issue an alarm when the image associated with the vehicle violation behavior meets the frame number condition, thereby effectively reducing the situation of false alarms.

[0080] It should be noted that for a frame of the image to be processed, it corresponds to one or more first targets. In the case where the image to be processed corresponds to multiple first targets, the vehicle violations of each first target in the current image to be processed can be analyzed simultaneously, that is, for the current image to be processed, it is identified whether each first target has vehicle violations, and for the first targets with vehicle violations, the number of image frames is accumulated.

[0081] In a specific implementation scheme, the first target is an electric vehicle. When detecting whether the first target in the image to be processed has a vehicle violation triggered by a second target with a subordinate relationship, the image to be processed is detected whether there is a license plate violation, an overloading violation, a violation of not wearing a helmet, or a violation of installing a windshield or a parasol.

[0082] Among them, when detecting whether there is a license plate violation in an image to be processed, the position information of the electric vehicle detection frame can be compared with the position information recorded last time to determine whether the electric vehicle is close to or far away from the camera. If the electric vehicle is close to the camera, it means that the electric vehicle is driving head-on. Since the electric vehicle license plate is usually installed behind the vehicle, it is normal that the license plate is not detected in this state, and the number of frames of driving without a license plate violation is not accumulated. On the contrary, if the electric vehicle is far away from the camera, the license plate should be detected. If the current frame of the image to be processed does not detect the license plate bound to the electric vehicle, the number of frames of the electric vehicle driving without a license plate violation is accumulated.

[0083] When detecting whether there is overloading violation in the image to be processed, the number of human bodies bound to each electric vehicle in the current frame of the image to be processed is determined. For example, if it exceeds 2, the number of frames of overloading violation of the electric vehicle is accumulated.

[0084] When detecting whether there is any violation of not wearing a helmet in the image to be processed, the number of people not wearing helmets bound to each electric vehicle in the current frame of the image to be processed is determined. If it is greater than 0, the number of frames of violation of not wearing a helmet for the electric vehicle is accumulated.

[0085] When checking whether there is any violation of the regulations by adding windshields or parasols to the image to be processed, the number of windshields or parasols bound to each electric vehicle in the current frame of the image to be processed is determined. If it is greater than 0, the number of frames in which the electric vehicle has installed a parasol or windshield in violation of the regulations is accumulated.

[0086] It should be noted that due to the existence of misbinding, when counting the number of image frames corresponding to various types of vehicle violations of the first target for each first target, the vehicle violations related to the misbinding situation need to be considered. For the images to be processed with vehicle violations identified based on the misbinding situation, the number of image frames of the images to be processed with vehicle violations corresponding to the misbinding situation is deducted from the accumulated number of image frames to avoid accumulating images with vehicle violations identified based on the misbinding situation.

[0087] In the above implementation scheme, for the first target, the images to be processed and the corresponding violation categories of the vehicle violations associated with the first target are identified, and the number of image frames corresponding to each type of vehicle violation of the first target is counted based on the identification situation, so as to issue an alarm when the number of image frames meets the conditions. By accumulating the images to be processed with vehicle violations, false alarms can be reduced.

[0088] The following is an introduction to the vehicle violation identification method provided by this application through an overall implementation process. Figure 2 As shown, including:

[0089] Step 201: Acquire video stream data associated with vehicle driving conditions, extract frames from the video stream data to determine video stream sequence frames, and pre-process initial images in the video stream sequence frames to generate images to be processed.

[0090] Step 202: Input the image to be processed in the video stream sequence frame into the target detection model, perform multi-target detection on the image to be processed based on the target detection model, and determine the position information of the detection box corresponding to the detection target.

[0091] Step 203: Establish a binding relationship between the second target and the first target according to the position information of the detection frame.

[0092] Step 204: Track the detected target in the video stream sequence frames based on the tracking algorithm, and update the binding relationship between the second target and the historically bound first target according to the tracking situation.

[0093] Step 205: for each first target, the number of image frames corresponding to each type of vehicle violation existing in the first target is accumulated, and when the number of image frames corresponding to a certain type of vehicle violation reaches a matching frame number threshold, an alarm prompt is output.

[0094] In the above implementation process, based on multi-target detection, detection target binding and tracking, multiple types of vehicle violations can be automatically and effectively identified, thereby improving the operating efficiency of the recognition system; and the situation where the detection target is misbound can be effectively eliminated, thereby improving the binding accuracy of the detection target; and an alarm can be issued when the number of image frames corresponding to the vehicle violation meets the conditions, thereby effectively reducing the situation of false alarms.

[0095] The following is an introduction to the vehicle violation identification method in the two-wheeled electric vehicle scenario through a specific example. Figure 3 As shown, including:

[0096] Step 301: Obtain road monitoring video stream data.

[0097] Step 302: extract frames from the video stream data to obtain video stream sequence frames, and perform cleaning operations such as resizing and normalization on the initial images of the video stream sequence frames to obtain images to be processed.

[0098] Step 303: input the image to be processed into the target detection model, and the target detection model detects seven detection targets: electric vehicles, human bodies, electric vehicle license plates, heads without helmets, heads with helmets, windshields, and electric vehicle parasols, and outputs the position information of the detection boxes corresponding to the detection targets.

[0099] Step 304, based on the location information, the detection target is bound in the following manner: for each electric vehicle detection frame, each human body frame, electric vehicle license plate frame, windshield frame, and electric vehicle parasol frame are traversed to identify the target detection frame whose IoU with the electric vehicle detection frame is greater than a preset threshold, and the detection target corresponding to the target detection frame is bound to the electric vehicle; and the head without a helmet and the head with a helmet are bound to the human body to determine the binding relationship between the electric vehicle and the head.

[0100] Step 305: Use a tracking algorithm to track the human body, electric vehicle, windshield, parasol, electric vehicle license plate, head without helmet, and head with helmet in the video stream sequence frames, and record the position information of the detection frame.

[0101] Step 306: Identify illegal behaviors of electric vehicles in the images to be processed in the video stream sequence frames.

[0102] Step 307: Based on the tracking situation, identify whether the human body, electric vehicle license plate, windshield, electric vehicle parasol, head without a helmet, head with a helmet and the electric vehicle with historical binding in the image to be processed are still in a binding state; if not, release the historical binding to update the binding relationship, and reduce the number of image frames of the electric vehicle violation determined based on the historical binding accordingly.

[0103] Step 308: for the tracked images to be processed, identify whether the number of image frames corresponding to various types of violations of electric vehicles has reached a matching frame number threshold.

[0104] If the number of image frames corresponding to all categories of vehicle violations has not reached the matching frame number threshold, return to step 303 to continue detecting the next frame of image to be processed; if the number of image frames corresponding to one or more categories of vehicle violations has reached the matching frame number threshold, enter step 309 to trigger the corresponding alarm, and clear the number of image frames corresponding to the corresponding category of vehicle violations of the electric vehicle, and return to step 303 to continue detecting the next frame.

[0105] Step 309: Output the location information and violation category of the alarm target.

[0106] The above implementation process introduces a method for identifying electric vehicle violations that combines target detection and tracking technology. It supports the identification of many common violations, such as electric vehicles on the road without license plates, electric vehicles equipped with parasols or windshields, not wearing helmets, and electric vehicle overloading. It realizes automatic and effective identification of electric vehicle violations and improves the operating efficiency of the identification system.

[0107] The present application also provides a vehicle violation identification device, such as Figure 4 As shown, including:

[0108] The detection module 401 is used to perform multi-target detection on the image to be processed in the video stream sequence frame when acquiring the video stream sequence frame associated with the vehicle driving situation, where the detection targets include a first target and N second targets, where N is an integer greater than or equal to 1, the first target is a vehicle object, the second target is an object associated with the vehicle violation that triggers the first target, and the image to be processed corresponds to at least one first target;

[0109] Establishing module 402, for establishing a binding relationship between the second target and the first target to which it belongs according to the position information of the detection frame corresponding to the detection target determined by multi-target detection for the image to be processed;

[0110] An updating module 403, configured to update the binding relationship according to the tracking status of the detection target in the video stream sequence frames;

[0111] A statistics module 404 is used to count, for each first target, the number of image frames of the to-be-processed images corresponding to different categories of vehicle violations of the first target, wherein the vehicle violation of the first target is triggered based on a second target with a subordinate relationship;

[0112] The output module 405 is configured to output an alarm prompt in response to the number of image frames corresponding to at least one type of vehicle violation of the first target meeting a preset condition.

[0113] Optionally, the detection module includes:

[0114] A processing and determining submodule, configured to perform image preprocessing on an initial image in the video stream sequence frames in response to the acquired video stream sequence frames, and determine the image to be processed;

[0115] A detection determination submodule, used to input the image to be processed in the video stream sequence frame into a target detection model, perform multi-target detection on the image to be processed based on the target detection model, and determine the position information of the detection frame corresponding to the detection target;

[0116] The video stream sequence frames are determined by extracting frames from video stream data associated with the vehicle driving condition.

[0117] Optionally, the establishing module includes:

[0118] A first processing submodule is used to traverse each second detection frame in the image to be processed for each first detection frame, and calculate the overlap degree of the second detection frame with the first detection frame based on the position information of the first detection frame and the second detection frame, where the first detection frame is the detection frame corresponding to the first target, and the second detection frame is the detection frame corresponding to the second target;

[0119] A second processing submodule, configured to identify the second target directly matching the first target based on the overlap degree, and determine the second target indirectly matching the first target based on the subordinate relationship between the second targets;

[0120] An establishment submodule is used to establish a binding relationship between the first target and the second target that match each other.

[0121] Optionally, the update module includes:

[0122] A tracking submodule, used for tracking the detected target in the video stream sequence frames based on a tracking algorithm;

[0123] An identification submodule, configured to identify whether the second target and the first target historically bound remain in a binding state based on the tracking situation;

[0124] The updating submodule is used to update the binding relationship in response to the second target not maintaining the binding state with the first target historically bound.

[0125] Optionally, the statistics module is further used for:

[0126] For each of the first targets, identifying in the video stream sequence frames an image to be processed in which a vehicle violation behavior associated with the first target exists and a violation category corresponding to the image to be processed;

[0127] The number of image frames corresponding to each type of vehicle violation existing in the first target is counted based on the recognition situation.

[0128] Optionally, the output module is further used for:

[0129] In response to the number of image frames corresponding to at least one type of vehicle violation of the first target reaching at least one matching frame number threshold, outputting an alarm prompt including location information of the alarm target and the violation category;

[0130] The warning target includes the first target and a second target that triggers the vehicle's illegal behavior.

[0131] Optionally, the first target is an electric vehicle object, the second target includes a human body object, a license plate object, an attachment object, a head object without a helmet, and a head object with a helmet, and the attachment object is an illegal object attached to the electric vehicle object;

[0132] The binding relationship includes: the binding relationship between the human body object, the license plate object, the accessory object, the head object not wearing a helmet, the head object wearing a helmet and the electric vehicle object respectively; the binding relationship between the head object not wearing a helmet, the head object wearing a helmet and the electric vehicle object is established based on the human body object as an intermediate medium;

[0133] The vehicle violations of the electric vehicle objects include: overloading violations, not wearing helmet violations, license plate violations and additional installation violations.

[0134] As for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0135] On the other hand, an embodiment of the present application also provides an electronic device, including a memory, a processor, a bus, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps in the above-mentioned vehicle violation identification method when executing the program.

[0136] For example, Figure 5 A schematic diagram of the physical structure of an electronic device is shown.

[0137] like Figure 5 As shown, the electronic device may include: a processor 510, a communication interface 520, a memory 530 and a communication bus 540, wherein the processor 510, the communication interface 520 and the memory 530 communicate with each other via the communication bus 540. The processor 510 may call the logic instructions in the memory 530, and the processor 510 is used to execute each process of the vehicle violation identification method of the embodiment of the present application, which will not be described one by one here.

[0138] In addition, the logic instructions in the above-mentioned memory 530 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art, and the computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0139] On the other hand, an embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps in the vehicle violation identification method provided in the above embodiments.

[0140] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0141] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for identifying vehicle violation behavior, characterized in that: include: In the case of acquiring a video stream sequence frame associated with a vehicle driving condition, performing multi-target detection on an image to be processed in the video stream sequence frame, the detected targets including a first target and N second targets, N being an integer greater than or equal to 1, the first target being a vehicle object, the second target being an object associated with a vehicle violation that triggers the first target, and the image to be processed corresponds to at least one first target; For the image to be processed, establishing a binding relationship between the second target and the first target to which it belongs according to the position information of the detection frame corresponding to the detection target determined by multi-target detection; Update the binding relationship according to the tracking status of the detection target in the video stream sequence frame; For each first target, counting the number of image frames of the to-be-processed images corresponding to different categories of vehicle violations of the first target, wherein the vehicle violation of the first target is triggered based on a second target with a subordinate relationship; In response to the number of image frames corresponding to at least one type of vehicle violation of the first target meeting a preset condition, an alarm prompt is output.

2. The method according to claim 1, characterized in that In the case of acquiring a video stream sequence frame associated with the vehicle driving condition, performing multi-target detection on the image to be processed in the video stream sequence frame, comprising: In response to the acquired video stream sequence frames, performing image preprocessing on the initial image in the video stream sequence frames to determine the image to be processed; Inputting the image to be processed in the video stream sequence frame into the target detection model, performing multi-target detection on the image to be processed based on the target detection model, and determining the position information of the detection frame corresponding to the detection target; The video stream sequence frames are determined by extracting frames from video stream data associated with the vehicle driving condition.

3. The method according to claim 1, characterized in that The step of establishing a binding relationship between the second target and the first target according to the position information of the detection frame corresponding to the detection target determined by multi-target detection for the image to be processed includes: In the image to be processed, for each first detection frame, traverse each second detection frame, and calculate the degree of overlap between the second detection frame and the first detection frame based on position information of the first detection frame and the second detection frame, where the first detection frame is a detection frame corresponding to the first target, and the second detection frame is a detection frame corresponding to the second target; Identify the second target that directly matches the first target based on the overlap degree, and determine the second target that indirectly matches the first target based on the subordinate relationship between the second targets; A binding relationship between the first target and the second target that match each other is established.

4. The method according to claim 1, characterized in that: The updating of the binding relationship according to the tracking status of the detection target in the video stream sequence frames includes: Tracking the detected target in the video stream sequence frames based on a tracking algorithm; Identify, based on the tracking situation, whether the second target remains bound to the first target that is historically bound; In response to the second target not maintaining a binding state with the first target historically bound thereto, the binding relationship is updated.

5. The method according to claim 1, characterized in that The counting of the number of image frames of the to-be-processed images corresponding to different categories of vehicle violations of the first target for each first target includes: For each of the first targets, identifying in the video stream sequence frames an image to be processed in which a vehicle violation behavior associated with the first target exists and a violation category corresponding to the image to be processed; The number of image frames corresponding to each type of vehicle violation existing in the first target is counted based on the recognition situation.

6. The method according to claim 1, characterized in that The outputting of an alarm prompt in response to the number of image frames corresponding to at least one type of vehicle violation behavior of the first target meeting a preset condition includes: In response to the number of image frames corresponding to at least one type of vehicle violation of the first target reaching at least one matching frame number threshold, outputting an alarm prompt including location information of the alarm target and the violation category; The warning target includes the first target and a second target that triggers the vehicle's illegal behavior.

7. The method according to any one of claims 1 to 6, characterized in that: The first target is an electric vehicle object, the second target includes a human body object, a license plate object, an attachment object, a head object without a helmet, and a head object with a helmet, and the attachment object is an illegal object attached to the electric vehicle object; The binding relationship includes: the binding relationship between the human body object, the license plate object, the accessory object, the head object not wearing a helmet, the head object wearing a helmet and the electric vehicle object respectively; the binding relationship between the head object not wearing a helmet, the head object wearing a helmet and the electric vehicle object is established based on the human body object as an intermediate medium; The vehicle violations of the electric vehicle objects include: overloading violations, not wearing helmet violations, license plate violations and additional installation violations.

8. A vehicle violation identification device, characterized in that: include: a detection module, configured to perform multi-target detection on an image to be processed in a video stream sequence frame associated with a vehicle driving condition, when acquiring the video stream sequence frame associated with the vehicle driving condition, wherein the detection targets include a first target and N second targets, where N is an integer greater than or equal to 1, the first target is a vehicle object, the second target is an object associated with a vehicle violation that triggers the first target, and the image to be processed corresponds to at least one first target; An establishing module, used for establishing, for the image to be processed, a binding relationship between the second target and the first target to which it belongs according to the position information of the detection frame corresponding to the detection target determined by multi-target detection; An updating module, used for updating the binding relationship according to the tracking status of the detection target in the video stream sequence frame; A statistics module, for counting, for each first target, the number of image frames of the to-be-processed images corresponding to different categories of vehicle violations of the first target, wherein the vehicle violations of the first target are triggered based on a second target with a subordinate relationship; The output module is used to output an alarm prompt in response to the number of image frames corresponding to at least one type of vehicle violation of the first target meeting a preset condition.

9. An electronic device, characterized in that: The method comprises a processor, a memory and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the vehicle violation identification method as claimed in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the vehicle violation identification method as described in any one of claims 1 to 7 are implemented.

Images