Non-motor vehicle violation and illegal behavior intelligent identification method based on multiple features

Abstract

The invention discloses a non-motor vehicle violation and illegal behavior intelligent identification method based on multiple features, relates to the technical field of non-motor vehicle behavior identification, and solves the problem that in the prior art, non-motor vehicle violation behavior identification systems mostly depend on a single feature for identification, so that misjudgment or missed judgment is easily caused in a complex traffic scene. According to the method, the global semantic understanding capability of a large model is introduced, so that deep fusion of various characteristics such as appearance, movement, behaviors and environment is realized. Therefore, the system can understand the traffic scene more comprehensively, so that the traffic violation behavior of the non-motor vehicle can be identified more accurately. In addition, the method further improves the recognition capability of the key information through the attention mechanism and region enhanced feature extraction. The multi-feature fusion mode not only improves the recognition accuracy, but also enhances the robustness and generalization ability of the system, so that the system can better adapt to different traffic scenes and environmental conditions.

Description

Technical Field

[0001] This invention relates to the field of non-motorized vehicle behavior recognition technology, and in particular to an intelligent recognition method for non-motorized vehicle violations based on multiple features. Background Technology

[0002] With the acceleration of urbanization, non-motorized vehicles are playing an increasingly important role in urban transportation. However, violations by non-motorized vehicles (such as running red lights, driving against traffic, and illegally carrying passengers) are also becoming more frequent, posing a significant threat to traffic safety. Traditional traffic monitoring systems mainly rely on manual monitoring and simple image analysis techniques, making it difficult to efficiently and accurately identify violations by non-motorized vehicles. In recent years, with the rapid development of computer vision and deep learning technologies, artificial intelligence-based traffic monitoring systems have gradually become a research hotspot. These systems collect image or video data in real time through cameras and use deep learning algorithms to automatically identify and analyze various behaviors in traffic scenarios. However, most existing intelligent recognition methods have some limitations, such as insufficient adaptability to complex scenarios and incomplete extraction of non-motorized vehicle features, resulting in room for improvement in recognition accuracy and robustness.

[0003] Most existing non-motorized vehicle violation recognition systems rely on single features (such as appearance or motion characteristics) for identification, which can easily lead to misjudgments or missed judgments in complex traffic scenarios. For example, relying solely on appearance features makes it difficult to distinguish non-motorized vehicles with similar colors and shapes, while relying solely on motion characteristics cannot accurately determine whether a vehicle is illegally carrying passengers or not wearing a helmet. In addition, traditional methods are poorly adaptable to environmental changes (such as changes in lighting, rain cover, etc.), further reducing the accuracy and reliability of recognition.

[0004] Therefore, there is a need for an intelligent identification method for non-motorized vehicle violations based on multiple features. Summary of the Invention

[0005] To address the problem that most existing non-motorized vehicle violation recognition systems rely on single features (such as appearance or motion features) for identification, leading to misjudgments or missed judgments in complex traffic scenarios, this invention provides a multi-feature-based intelligent recognition method for non-motorized vehicle violations. This method combines multiple features such as appearance, motion, behavior, and environment, and deeply fuses and analyzes these features through the global correlation capabilities of a large model. By introducing an attention mechanism and region-enhanced feature extraction, this method can better adapt to complex scenarios, improving the accuracy and robustness of non-motorized vehicle violation recognition. Furthermore, through a multi-objective loss function and optimized feature fusion strategy, this method can better learn the importance of different features during training, further improving model performance. The specific technical solution is as follows: A multi-feature-based intelligent identification method for non-motorized vehicle violations includes the following steps: Collect image data of non-motorized vehicles and perform data annotation and preprocessing operations; Multi-feature extraction, where features include at least: appearance features, motion features, behavioral features, and environmental features; The extracted appearance, motion, behavior, and environmental features are input into the attention mechanism module to generate a region enhancement feature map. The region enhancement feature map is then concatenated with the appearance, motion, behavior, and environmental features to form a multi-feature fusion feature vector. The concatenated feature vector is then fused using a weighted summation method. By using the large model as the base model, and combining the feature fusion module with the base model, a complete recognition network can be constructed. Design a loss function and train a network that integrates a large model and multiple features using labeled data; The trained model is deployed to the monitoring system to analyze the real-time images or videos.

[0006] Preferably, the appearance features include the vehicle's color, shape, and license plate; the motion features include speed, acceleration, and driving trajectory; the behavioral features include traffic light status and types of violations; and the environmental features identify environmental information surrounding the non-motorized vehicle.

[0007] Preferably, the specific steps for constructing a complete recognition network by using a large model as the base model and combining the feature fusion module with the base model are as follows: The CLIP model was used to extract global semantic features of non-motorized vehicles using a visual encoder; in addition, the DINOv2 model was used to extract fine-grained visual features of non-motorized vehicles. The features extracted by CLIP and DINOv2 are projected through a linear layer to make their dimensions consistent. A hierarchical feature fusion algorithm is used to additively fuse the multi-layer features of the DINOv2 model.

[0008] The fused DINOv2 features are concatenated with CLIP features and then fused using an attention mechanism (such as a self-attention layer).

[0009] The features fused through the attention mechanism are input into a fully connected layer or a convolutional layer, and a classifier is added to the output of the feature fusion network to identify violations by non-motorized vehicles. The input of the classifier is the fused feature vector, and the output is the probability that a non-motorized vehicle has committed a violation.

[0010] Preferably, the loss function formula is expressed as follows: In the formula, These are weighting coefficients. It is a fusion loss. It is marginal loss. It is a regional enhancement loss.

[0011] Preferably, the following steps are also included: Adjustable weight parameters can be set to adjust the contribution of different features in the fusion process.

[0012] Preferably, the multi-feature extraction step also includes extracting spatiotemporal features, which include temporal distribution and spatial distribution. The temporal distribution is used to analyze the time distribution of non-motorized vehicle violations throughout the day, and the spatial distribution is used to identify hotspot areas of non-motorized vehicle violations.

[0013] Preferably, the following steps are also included: The model's recognition accuracy, recall, and false detection rate are evaluated using data obtained from the application. Based on the evaluation results, the model structure and loss function are adjusted.

[0014] Preferably, during the training of a network that integrates a large model and multiple features using labeled data, an appropriate optimization algorithm is selected for model training, the learning rate and weight decay are adjusted, and the model parameters are updated through backpropagation during the training process to optimize the loss function.

[0015] A computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device on which the computer-readable storage medium is located to perform the intelligent identification method for non-motorized vehicle violations based on multiple features as described above.

[0016] A processor for running a program, wherein the program executes the intelligent identification method for non-motorized vehicle violations based on multiple features as described above.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1. This invention achieves deep fusion of multiple features, including appearance, motion, behavior, and environment, by introducing the global semantic understanding capabilities of a large model. This enables the system to more comprehensively understand traffic scenarios, thereby more accurately identifying non-motorized vehicle violations. For example, DINOv2 can capture complex visual patterns and maintain high recognition accuracy even in complex environments such as changing lighting and awning obstructions. Furthermore, this method further enhances the ability to recognize key information through attention mechanisms and region-enhanced feature extraction. This multi-feature fusion approach not only improves recognition accuracy but also enhances the system's robustness and generalization ability, allowing it to better adapt to different traffic scenarios and environmental conditions.

[0018] 2. This invention offers significant advantages in terms of comprehensiveness and accuracy. Traditional methods mostly rely on single features for identification, such as judging non-motorized vehicle violations solely based on appearance features (color, shape) or motion features (speed, trajectory). However, single features often struggle to handle complex and ever-changing traffic scenarios. For example, in environments with insufficient light or complex backgrounds, relying solely on appearance features may fail to accurately distinguish the type and status of non-motorized vehicles; while relying solely on motion features makes it difficult to determine whether a vehicle is illegally carrying passengers or not wearing a helmet. The multi-feature fusion method, by comprehensively considering multiple features such as appearance, motion, behavior, and environment, can more comprehensively understand various information in traffic scenarios. The powerful semantic understanding capabilities of the DINOv2 large model further enhance its adaptability to complex scenarios, enabling it to accurately identify non-motorized vehicle violations under various conditions, significantly improving the accuracy and reliability of identification.

[0019] 3. This invention also demonstrates excellent robustness and generalization ability. Traditional recognition technologies often experience a significant drop in performance when faced with environmental changes (such as changes in lighting, awning obstructions, pedestrian interference, etc.). This is because single features are poorly adaptable to environmental changes and easily affected by external factors. However, the multi-feature fusion method, by introducing multiple features, can analyze and identify non-motorized vehicles from different perspectives. For example, even in low-light conditions, motion and behavioral features can still provide effective information to help the system accurately determine the vehicle's status. Furthermore, the global correlation capability of the DINOv2 model can better understand the relationship between non-motorized vehicles and their surrounding environment, further improving the system's robustness. Regarding generalization ability, this method, through a multi-objective loss function and optimized feature fusion strategy, can better learn the importance of different features during training, enabling the model to adapt to various traffic scenarios and environmental conditions. This allows the system to operate more stably in practical applications, reducing false positives and false negatives.

[0020] 4. This invention also boasts a higher level of intelligence and efficiency. Traditional methods typically require significant manual intervention, such as manually labeling data and adjusting parameters, which is not only time-consuming and labor-intensive but also prone to introducing human error. This method, through deep learning and automated feature extraction, can automatically learn and identify various characteristics of non-motorized vehicles, greatly reducing the need for manual intervention. Furthermore, the efficient feature extraction capability of the DINOv2 model enables the system to process large amounts of image and video data in a short time, improving the speed and efficiency of recognition. In practical applications, this means the system can monitor and identify non-motorized vehicle violations in real time, issue timely alerts, and take corresponding measures, thereby better ensuring traffic safety. Simultaneously, this method can provide traffic management departments with more accurate and comprehensive violation data, contributing to more efficient and precise traffic management decisions. Attached Figure Description

[0021] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0022] Figure 1 This is a flowchart of the method of the present invention; Figure 2 A schematic diagram illustrating the steps involved in constructing a complete recognition network; Figure 3 This is a schematic diagram illustrating the steps of fusing multiple extracted features. Detailed Implementation

[0023] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] It should be understood that, when used in this specification and the appended claims, the terms "comprising" and "including" indicate the presence of the described features, integrals, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0025] It should also be understood that the terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the invention. As used in this specification and the appended claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise.

[0026] It should also be further understood that the term "and / or" as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0027] In one embodiment of the present invention, a method for intelligent identification of non-motorized vehicle violations based on multiple features is provided, such as... Figure 1 As shown, it includes the following steps: Step 1: Collect image data of non-motorized vehicles and label and preprocess the data.

[0028] The data collection process primarily involves using traffic monitoring cameras to collect image and video data containing non-motorized vehicles. The collected data is then labeled, including information such as the location, type, and whether any violations have occurred by the non-motorized vehicles. Furthermore, data augmentation techniques (such as Mosaic and MixUp) are employed to increase data diversity and improve the model's generalization ability across different scenarios.

[0029] Step 2: Feature extraction. Features include at least: appearance features, motion features, behavioral features, spatial features, and environmental features. Appearance features include the vehicle's color, shape, and license plate; motion features include speed, acceleration, and driving trajectory; behavioral features include traffic light status and types of violations; spatiotemporal features include temporal and spatial distribution; and environmental features identify the environmental information surrounding the non-motorized vehicle.

[0030] In terms of appearance features, vehicle color and shape are extracted using image recognition technology to distinguish different types of non-motorized vehicles. License plate recognition utilizes optical character recognition (OCR) technology to identify license plate numbers in order to track and record violations.

[0031] In motion characteristics analysis, the speed and acceleration of non-motorized vehicles are calculated through radar or continuous frame image analysis to determine whether speeding occurs. Analyzing the trajectory of non-motorized vehicles can determine whether they are traveling within the designated lane or whether they are traveling against the flow of traffic.

[0032] Among the behavioral characteristics, image recognition technology is used to identify the color of traffic lights to determine whether non-motorized vehicles are crossing at red lights. Types of violations mainly include identifying whether non-motorized vehicles are crossing outside of pedestrian crossings, illegally carrying passengers, or not wearing helmets.

[0033] In the spatiotemporal characteristics, temporal distribution mainly analyzes the time distribution of non-motorized vehicle violations throughout the day, such as violations during morning and evening rush hours. Spatial distribution mainly identifies hotspots of non-motorized vehicle violations, such as urban core areas or specific intersections.

[0034] The environmental features primarily identify information about the environment surrounding non-motorized vehicles, such as whether there are awnings or other pedestrians. Specifically, a semantic segmentation model (such as DeepLabv3plus) is used to identify awnings around non-motorized vehicles, and an object detection model (such as YOLOv5s) is used to detect pedestrians around non-motorized vehicles.

[0035] Step 3: Fuse the extracted features. For example... Figure 3 As shown, the details are as follows: S1: Design an attention mechanism module to extract region enhancement features, and input the extracted appearance, motion, behavior and environment features into the attention mechanism module to generate a region enhancement feature map.

[0036] For example, the attention mechanism module can be either the Squeeze-and-Excitation (SE) module or the ChannelAttention Module (CAM).

[0037] S2: The region enhancement feature map is concatenated with appearance features, motion features, behavior features and environmental features to form a multi-feature fusion feature vector, and the concatenated feature vector is fused by weighted summation.

[0038] S3: Set adjustable weight parameters to adjust the contribution of different features in the fusion process, so that the model can better learn the importance of different features.

[0039] Step 4: Train and optimize the model. For example... Figure 2 As shown, the specific steps are as follows: S01: Using a large model (such as CLIP or DINOv2) as the base model, combine the feature fusion module with the base model to construct a complete recognition network. The specific steps are as follows: S101: The CLIP model's visual encoder is used to extract global semantic features of non-motorized vehicles. The CLIP visual encoder provides rich semantic information, which helps in understanding the overall features of non-motorized vehicles; in addition, the DINOv2 model is used to extract fine-grained visual features of non-motorized vehicles. DINOv2 excels at capturing fine-grained details in images, such as the local structure and texture of vehicles, which helps in identifying the appearance features of vehicles; S102: Project the features extracted by CLIP and DINOv2 through a linear layer to make their dimensions consistent. This step ensures that the two types of features can be fused in the same feature space; S103: Employ a hierarchical feature fusion algorithm to additively fuse the multi-layer features of the DINOv2 model. For example, the features of layers 3, 6, 9, and 12 of the DINOv2 model can be additively fused to fully utilize its fine-grained features.

[0040] S104: The fused DINOv2 features are concatenated with CLIP features and then fused using an attention mechanism (such as a self-attention layer). The attention mechanism can dynamically adjust the feature weights based on the importance of non-motorized vehicles in the image, improving the feature fusion effect.

[0041] S105: The features fused through the attention mechanism are input into a fully connected layer or convolutional layer to further extract a high-level representation of the fused features. A classifier (such as a Softmax classifier) ​​is added to the output of the feature fusion network to identify violations by non-motorized vehicles. The input to the classifier is the fused feature vector, and the output is the probability that a non-motorized vehicle has committed a violation.

[0042] S02: Design the loss function. In this embodiment, a multi-objective loss function is introduced to guide the model to better perform feature fusion.

[0043] For example, the loss function formula is expressed as follows: In the formula, These are weighting coefficients. It is a fusion loss. It is marginal loss. It is a regional enhancement loss.

[0044] S03: Train the network that integrates large models and multiple features using labeled data, select an appropriate optimization algorithm (such as AdamW or SGD) for model training, adjust hyperparameters such as learning rate and weight decay, and update model parameters through backpropagation during training to optimize the loss function and improve the model's recognition accuracy.

[0045] Step 5: Deploy the trained model into the monitoring system to analyze the real-time acquired images or videos.

[0046] Step Six: Using the data obtained from the application, evaluate the model's recognition accuracy, recall, false positive rate, and other metrics. Based on the evaluation results, adjust the feature extraction method, optimize the feature fusion strategy, and then adjust the model structure and improve the loss function to further enhance the model's recognition accuracy and robustness.

[0047] In summary, this invention achieves deep fusion of multiple features, including appearance, motion, behavior, and environment, by introducing the global semantic understanding capabilities of a large model. This enables the system to more comprehensively understand traffic scenarios, thereby more accurately identifying non-motorized vehicle violations. For example, DINOv2 can capture complex visual patterns and maintain high recognition accuracy even in complex environments such as changing lighting and canopy obstruction. Furthermore, this method further enhances the ability to identify key information through attention mechanisms and region-enhanced feature extraction. This multi-feature fusion approach not only improves recognition accuracy but also enhances the system's robustness and generalization ability, enabling it to better adapt to different traffic scenarios and environmental conditions.

[0048] Those skilled in the art will recognize that the units of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0049] In the embodiments provided by the present invention, it should be understood that the division of units is only a logical functional division. In actual implementation, there may be other division methods, such as multiple units can be combined into one unit, one unit can be split into multiple units, or some features can be ignored.

[0050] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0051] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0052] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention, and they should all be covered within the scope of the claims and specification of the present invention.

Claims

1. A method for intelligent identification of non-motorized vehicle violations based on multiple features, characterized in that, Includes the following steps: Collect image data of non-motorized vehicles and perform data annotation and preprocessing operations; Multi-feature extraction, where features include at least: appearance features, motion features, behavioral features, and environmental features; The extracted appearance, motion, behavior, and environmental features are input into the attention mechanism module to generate a region enhancement feature map. The region enhancement feature map is then concatenated with the appearance, motion, behavior, and environmental features to form a multi-feature fusion feature vector. The concatenated feature vector is then fused using a weighted summation method. By using the large model as the base model, and combining the feature fusion module with the base model, a complete recognition network can be constructed. Design a loss function and train a network that integrates a large model and multiple features using labeled data; The trained model is deployed to the monitoring system to analyze the real-time images or videos.

2. The intelligent identification method for non-motorized vehicle violations based on multiple features according to claim 1, characterized in that, Appearance features include the vehicle's color, shape, and license plate; motion features include speed, acceleration, and trajectory; behavior features include traffic light status and types of violations; and environmental features identify information about the environment surrounding the non-motorized vehicle.

3. The intelligent identification method for non-motorized vehicle violations based on multiple features according to claim 1, characterized in that, The specific steps for constructing a complete recognition network by using a large model as the base model and combining the feature fusion module with the base model are as follows: The CLIP model was used to extract global semantic features of non-motorized vehicles using a visual encoder; in addition, the DINOv2 model was used to extract fine-grained visual features of non-motorized vehicles. The features extracted by CLIP and DINOv2 are projected through a linear layer to make their dimensions consistent. A hierarchical feature fusion algorithm is used to additively fuse the multi-layer features of the DINOv2 model; The fused DINOv2 features are concatenated with CLIP features and then fused using an attention mechanism. The features fused through the attention mechanism are input into a fully connected layer or a convolutional layer, and a classifier is added to the output of the feature fusion network to identify violations by non-motorized vehicles. The input of the classifier is the fused feature vector, and the output is the probability that a non-motorized vehicle has committed a violation.

4. The intelligent identification method for non-motorized vehicle violations based on multiple features according to claim 1, characterized in that, The loss function formula is expressed as follows: In the formula, These are weighting coefficients. It is a fusion loss. It is marginal loss. It is a regional enhancement loss.

5. The intelligent identification method for non-motorized vehicle violations based on multiple features according to claim 1, characterized in that, It also includes the following steps: Adjustable weight parameters can be set to adjust the contribution of different features in the fusion process.

6. The intelligent identification method for non-motorized vehicle violations based on multiple features according to claim 1, characterized in that, The multi-feature extraction step also includes extracting spatiotemporal features, which include temporal distribution and spatial distribution. The temporal distribution is used to analyze the time distribution of non-motorized vehicle violations throughout the day, and the spatial distribution is used to identify hotspots of non-motorized vehicle violations.

7. The intelligent identification method for non-motorized vehicle violations based on multiple features according to claim 1, characterized in that, It also includes the following steps: The model's recognition accuracy, recall, and false detection rate are evaluated using data obtained from the application. Based on the evaluation results, the model structure and loss function are adjusted.

8. The intelligent identification method for non-motorized vehicle violations based on multiple features according to claim 1, characterized in that, During the training of a network that integrates a large model and multiple features using labeled data, an appropriate optimization algorithm is selected for model training, the learning rate and weight decay are adjusted, and the model parameters are updated through backpropagation during the training process to optimize the loss function.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored program, wherein, when the program is executed, it controls the device containing the computer-readable storage medium to perform the intelligent identification method for non-motorized vehicle violations based on any one of claims 1 to 8.

10. A processor, characterized in that, The processor is used to run a program, wherein the program executes the intelligent identification method for non-motorized vehicle violations based on multiple features as described in any one of claims 1 to 8.

Images