Non-motor vehicle illegal behavior detection method, device, equipment and medium
By performing head and shoulder detection and trajectory tracking on traffic monitoring videos, the correlation between the movement of non-motorized vehicles and reference objects is determined, solving the problem of monitoring illegal passenger carrying by non-motorized vehicles and improving detection accuracy and road safety.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-11-03
- Publication Date
- 2026-03-24
AI Technical Summary
In the process of urbanization, it is difficult to effectively monitor the illegal carrying of passengers by non-motorized vehicles, resulting in traffic safety hazards. Existing technologies are insufficient to cope with the large number of non-motorized vehicles, traffic police have limited human resources, and video surveillance is ineffective.
By performing head and shoulder detection and non-motorized vehicle detection on traffic monitoring videos, and using tracking algorithms to determine the trajectory information of non-motorized vehicles and reference objects, the motion correlation of the related objects is compared to determine whether non-motorized vehicles are illegally carrying passengers.
It has achieved precise monitoring of illegal passenger transport by non-motorized vehicles, improved traffic safety, and achieved a detection accuracy rate of 85% to 90%.
Smart Images
Figure CN121725433A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of intelligent transportation technology, and in particular to a method, device, equipment and medium for detecting illegal acts of non-motorized vehicles. Background Technology
[0002] With the advancement of urbanization, the urban population is increasing, and the number of motor vehicles in various places is also increasing, leading to more and more road traffic problems. In order to avoid traffic congestion and achieve the goal of convenient travel, many people choose to use non-motorized vehicles such as electric vehicles for transportation.
[0003] However, due to complex traffic conditions, inadequate road construction in various regions, and a lack of protective measures for non-motorized vehicles, people using non-motorized vehicles are at a disadvantage when traveling on the road, making them more prone to accidents and suffering more severe injuries.
[0004] For traffic safety, relevant regulations clearly prohibit the illegal carrying of passengers by non-motorized vehicles such as electric bikes. However, it is difficult to cope with the huge number of non-motorized vehicles in the city by relying solely on the limited traffic police manpower. Therefore, it is particularly important to use video surveillance to monitor the illegal carrying of passengers by non-motorized vehicles.
[0005] Therefore, there is an urgent need for a method to monitor illegal passenger transport by non-motorized vehicles in order to cope with the large number of non-motorized vehicles in cities and ensure traffic safety. Summary of the Invention
[0006] This application provides a method, device, equipment, and medium for detecting illegal non-motorized vehicle behavior, which is used to monitor the illegal carrying of passengers by non-motorized vehicles in the large urban non-motorized vehicle population and ensure traffic safety.
[0007] Firstly, this application provides a method for detecting non-motorized vehicle violations, the method comprising: Obtain traffic monitoring videos of non-motorized vehicles; Head and shoulder detection and non-motorized vehicle detection are performed on video frames in the traffic monitoring video to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video. The tracking algorithm is used to track the head and shoulder regions of the non-motorized vehicle and the at least one reference object to determine the first trajectory information of the non-motorized vehicle and the second trajectory information of the at least one reference object. Based on the first trajectory information and the second trajectory information, the positions of the non-motorized vehicle and the at least one reference object are compared to determine the associated object; the associated object includes the target object among the non-motorized vehicle and the at least one reference object. Based on the trajectory information of the associated object and other traffic objects within the preset area of the associated object, the motion correlation between the associated object and other traffic objects is determined; If the motion correlation between the associated entity and at least one other traffic object is not positive, then it is determined that the non-motorized vehicle in the associated entity is illegally carrying passengers.
[0008] In one possible implementation, the head and shoulder detection and non-motorized vehicle detection performed on the traffic monitoring video to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video includes: Using a pre-trained head and shoulder detection model, head and shoulder detection is performed on the traffic monitoring video to determine the head and shoulder region of at least one reference object in the traffic monitoring video; Using a pre-trained non-motorized vehicle detection model, non-motorized vehicles are detected in the traffic monitoring video to identify them.
[0009] In one possible implementation, the step of comparing the positions of the non-motorized vehicle and the at least one reference object based on the first trajectory information and the second trajectory information to determine the associated entity includes: For each reference object, based on the first trajectory information and the second trajectory information, the distance feature, height ratio feature, and intersection-union ratio feature between the non-motorized vehicle and the reference object are determined; the distance feature is the distance between the center point of the non-motorized vehicle and the center point of the reference object; the height ratio feature is the ratio between the height of the non-motorized vehicle and the height of the reference object; the intersection-union ratio feature is the intersection-union ratio between the bounding box of the non-motorized vehicle and the bounding box of the reference object. Based on the distance feature, the height ratio feature, and the intersection-union ratio feature, the association probability between the non-motorized vehicle and the reference object is determined; If the association probability is greater than the preset association probability, then the reference object is taken as the target object, and the non-motorized vehicle and the target object are determined to be associated entities.
[0010] In one possible implementation, determining the motion correlation between the associated entity and other traffic objects based on the trajectory information of the associated entity and other traffic objects within a preset area of the associated entity includes: Based on the trajectory information of the associated entity and other traffic objects within the preset area of the associated entity, calculate the intersection-combination ratio between the associated entity and other traffic objects; For each of the other traffic objects, perform the following operations: If the intersection-over-union ratio between the associated body and the traffic object is greater than a preset intersection-over-union ratio threshold, then the motion trajectories of the associated body and the traffic object in all video frames within a preset time window containing the video frame are obtained. Based on the motion trajectories of the associated body and the traffic object in all video frames, the motion correlation between the associated body and the traffic object is calculated.
[0011] In one possible implementation, the method further includes: If the motion correlation between the associated entity and the other traffic objects is linearly positive, then it is determined that the non-motorized vehicle in the associated entity does not have illegal passenger-carrying behavior.
[0012] In one possible implementation, calculating the motion correlation between the associated object and the traffic object based on their motion trajectories in all video frames includes: If the motion trajectories of the associated object and the traffic object in all video frames meet a preset similarity condition, then the motion correlation between the associated object and the traffic object is determined to be linearly positive.
[0013] In one possible implementation, the preset similarity condition is determined by the linear fit between the two motion trajectories in a two-dimensional image coordinate system or a world coordinate system.
[0014] Secondly, this application provides a detection device for non-motorized vehicle violations, the device comprising: The data acquisition module is used to acquire traffic monitoring videos of non-motorized vehicles; The target detection module is used to perform head and shoulder detection and non-motorized vehicle detection on video frames in the traffic monitoring video, and to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video. The trajectory tracking module is used to track the head and shoulder regions of the non-motorized vehicle and the at least one reference object using a tracking algorithm, and to determine the first trajectory information of the non-motorized vehicle and the second trajectory information of the at least one reference object. The associated object determination module is used to compare the positions of the non-motorized vehicle and the at least one reference object based on the first trajectory information and the second trajectory information to determine the associated object; the associated object includes the target object among the non-motorized vehicle and the at least one reference object; The relationship determination module is used to determine the motion correlation between the associated entity and other traffic objects within a preset area of the associated entity. The behavior determination module is used to determine that the non-motorized vehicle in the associated entity has engaged in illegal passenger-carrying behavior if the motion correlation between the associated entity and at least one other traffic object is not positive.
[0015] Thirdly, this application also provides an electronic device, which includes a processor for executing a computer program stored in a memory to implement the steps of the non-motorized vehicle violation detection method as described above.
[0016] Fourthly, this application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the detection method for non-motorized vehicle violations as described above.
[0017] Fifthly, this application provides a computer program product, including a computer program: when the computer program is executed by a processor, it implements the method for detecting non-motorized vehicle violations as described in the first aspect above.
[0018] The technical solutions provided by the embodiments of this application have at least the following beneficial effects: In this embodiment, head and shoulder detection and non-motorized vehicle detection are performed on video frames to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the video frame. A tracking algorithm is used to track the trajectories of the head and shoulder regions of non-motorized vehicles and at least one reference object to determine the first trajectory information of the non-motorized vehicle and the second trajectory information of at least one reference object. Based on the first and second trajectory information, the positions of the non-motorized vehicle and at least one reference object are compared to determine the associated entity. The associated entity includes the target object among the non-motorized vehicle and at least one reference object. Based on the trajectory information of the associated entity and other traffic objects within a preset area of the associated entity, the motion correlation between the associated entity and other traffic objects is determined. If the motion correlation between the associated entity and at least one other traffic object is not positive, it is determined that the non-motorized vehicle in the associated entity is illegally carrying passengers.
[0019] Therefore, this application detects the head and shoulder regions of non-motorized vehicles and at least one reference object in video frames, and then uses a tracking algorithm to track the trajectories of the non-motorized vehicles and at least one reference object. Based on the trajectory information, the positions of the non-motorized vehicles and at least one reference object are compared to determine the target object associated with the non-motorized vehicle. Furthermore, by analyzing the motion correlation between the associated object and other traffic objects within a preset area of the associated object, it is determined whether the non-motorized vehicle in the associated object is illegally carrying passengers. This method can more accurately monitor the illegal carrying of passengers by non-motorized vehicles than the traditional method of identifying faces on non-motorized vehicles in video frames, thereby ensuring road safety.
[0020] Other features and advantages of this application will be set forth in the description which follows, and will be apparent in part from the description, or may be learned by practicing the application. The objectives and other advantages of this application may be realized and obtained by means of the structures particularly pointed out in the written description, claims, and drawings. Attached Figure Description
[0021] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application.
[0022] Figure 1 This is a schematic diagram illustrating an application scenario of a method for detecting non-motorized vehicle violations provided in an embodiment of this application. Figure 2 A flowchart illustrating a method for detecting non-motorized vehicle violations provided in this application embodiment; Figure 3 A schematic diagram of a target detection process provided in an embodiment of this application; Figure 4 A schematic flowchart illustrating the training process of a head and shoulder detection model provided in an embodiment of this application; Figure 5 A flowchart illustrating a method for determining an associated entity provided in an embodiment of this application; Figure 6 A flowchart illustrating a motion correlation determination method provided in an embodiment of this application; Figure 7 A schematic diagram illustrating the effect of detecting illegal passenger-carrying behavior of a two-wheeled vehicle, provided in an embodiment of this application; Figure 8 A schematic diagram illustrating the effect of detecting illegal passenger transport by a tricycle, provided in an embodiment of this application; Figure 9 A schematic diagram of a non-motorized vehicle violation detection device provided in an embodiment of this application; Figure 10 This is a schematic diagram of an electronic device structure provided in an embodiment of this application. Detailed Implementation
[0023] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0024] It should be noted that the terms "comprising" and "having" and their variations used in this application are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such process, method, product, or device.
[0025] The terms "first" and "second" used in this document are for descriptive purposes only and should not be construed as indicating relative importance or implying the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more.
[0026] The word “exemplary” as used below means “serving as an example, embodiment, or illustration.” Any embodiment illustrated as an “exemplary” need not be construed as superior to or better than other embodiments.
[0027] With the advancement of urbanization, the urban population is increasing, and the number of motor vehicles in various places is also increasing, leading to more and more road traffic problems. In order to avoid traffic congestion and achieve the goal of convenient travel, many people choose to use non-motorized vehicles such as electric vehicles for transportation.
[0028] However, due to complex traffic conditions, inadequate road construction in various regions, and a lack of protective measures for non-motorized vehicles, people using non-motorized vehicles are at a disadvantage when traveling on the road, making them more prone to accidents and suffering more severe injuries.
[0029] For traffic safety, relevant regulations clearly prohibit the illegal carrying of passengers by non-motorized vehicles such as electric bikes. However, it is difficult to cope with the huge number of non-motorized vehicles in the city by relying solely on the limited traffic police manpower. Therefore, it is particularly important to use video surveillance to monitor the illegal carrying of passengers by non-motorized vehicles.
[0030] Therefore, there is an urgent need for a method to monitor illegal passenger transport by non-motorized vehicles in order to cope with the large number of non-motorized vehicles in cities and ensure traffic safety.
[0031] In view of this, this application provides a method, device, equipment and medium for detecting illegal non-motorized vehicle behavior, which is used to monitor the illegal carrying of passengers by non-motorized vehicles in the large urban non-motorized vehicle population and ensure traffic safety.
[0032] The inventive concept of this application can be summarized as follows: by detecting the head and shoulder regions of non-motorized vehicles and at least one reference object in a video frame, and then using a tracking algorithm to track the trajectory of the non-motorized vehicles and at least one reference object, the position of the non-motorized vehicles and at least one reference object is compared based on the trajectory information to determine the target object associated with the non-motorized vehicles. Furthermore, by the motion correlation between the associated body and other traffic objects within a preset area of the associated body, it is determined whether the non-motorized vehicles in the associated body are illegally carrying passengers.
[0033] The method for detecting non-motorized vehicle violations provided in this application is applied to electronic devices, such as PCs, mobile terminals, terminal devices, and servers. Furthermore, the method for detecting non-motorized vehicle violations provided in this application can be applied to distributed software platforms, such as blockchain.
[0034] After introducing the main inventive ideas of the embodiments of this application, combined with Figure 1 This application describes the application scenarios of the detection method for non-motorized vehicle violations provided in this application.
[0035] like Figure 1 As shown, the application scenarios for the non-motorized vehicle violation detection method provided in this application include traffic monitoring, terminal equipment, and users.
[0036] Traffic monitoring is used to collect traffic monitoring videos and send them to terminal devices; the traffic monitoring videos include non-motorized vehicles. The terminal device is used to perform head and shoulder detection and non-motorized vehicle detection on video frames in traffic monitoring videos to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video; it uses a tracking algorithm to track the trajectories of the head and shoulder regions of non-motorized vehicles and at least one reference object to determine the first trajectory information of the non-motorized vehicle and the second trajectory information of at least one reference object; based on the first trajectory information and the second trajectory information, it compares the positions of the non-motorized vehicle and at least one reference object to determine the associated entity; the associated entity includes the target object among the non-motorized vehicle and at least one reference object; based on the trajectory information of the associated entity and other traffic objects within a preset area of the associated entity, it determines the motion correlation between the associated entity and other traffic objects; if the motion correlation between the associated entity and at least one other traffic object is not positive, it is determined that the non-motorized vehicle in the associated entity is illegally carrying passengers.
[0037] If the terminal device determines that a non-motorized vehicle in the associated network is illegally carrying passengers, it can send a warning message to the user so that the user can take action. In this embodiment, the user is a traffic police officer or other law enforcement personnel.
[0038] To further illustrate the technical solutions provided in the embodiments of this application, a detailed description is provided below in conjunction with the accompanying drawings and specific implementation methods. Although the embodiments of this application provide method operation steps as shown in the following embodiments or drawings, the method may include more or fewer operation steps based on conventional or non-inventive methods. In steps where there is no logically necessary causal relationship, the execution order of these steps is not limited to the execution order provided in the embodiments of this application.
[0039] See Figure 2 This is a flowchart illustrating a method for detecting non-motorized vehicle violations according to an embodiment of this application. The method can be specifically executed as follows: Figure 2 The steps shown are as follows: In step S201, traffic monitoring video of non-motorized vehicles is acquired, and head and shoulder detection and non-motorized vehicle detection are performed on the video frames in the traffic monitoring video to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video.
[0040] In this application, traffic monitoring video includes real-time video streams transmitted through on-site front-end equipment, i.e., monitoring terminals, and stored offline video. Real-time video streams are accessed via RTSP or a video gateway to perform real-time detection of intersections in specified directions and areas, thereby acquiring the real-time video streams.
[0041] The compressed video stream is then decoded into video frames using a dedicated hardware decoder, such as a GPU hardware decoding chip. This hardware can efficiently process video data, providing raw data for subsequent processing. Finally, video frames are extracted for head and shoulder detection and non-motorized vehicle detection.
[0042] In this application, before performing head and shoulder detection and non-motorized vehicle detection on the video frames in traffic monitoring videos, preprocessing can be performed on the original video frames. Specifically, this can be done as follows: Preprocessing is performed on the decoded video frames through illumination correction and noise reduction. Furthermore, preprocessed video frames can be cropped, scaled, or otherwise modified to improve image quality.
[0043] In one possible implementation, head and shoulder detection and non-motorized vehicle detection are performed on traffic monitoring videos to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video. Specifically, this is performed as follows: Figure 3 The steps shown are as follows: In step S301, a pre-trained head and shoulder detection model is used to perform head and shoulder detection on the traffic monitoring video to determine the head and shoulder region of at least one reference object in the traffic monitoring video. In step S302, a pre-trained non-motorized vehicle detection model is used to detect non-motorized vehicles in the traffic monitoring video to identify non-motorized vehicles in the traffic monitoring video.
[0044] The training of the head and shoulder detection model involves first cropping and scaling the preprocessed video frames to create an original dataset. Then, annotation tools are used to label and classify the original dataset, converting it into text data that the head and shoulder detection model can process. Finally, the labeled original dataset is input into the head and shoulder detection model for training, resulting in a pre-trained model.
[0045] The head and shoulder detection models used in this application include, but are not limited to, algorithms such as Faster R-CNN, Cascade R-CNN, YOLOv5, and YOLOv8.
[0046] like Figure 4 The diagram shows the training process of the head and shoulder detection model in this application.
[0047] In step S1, data acquisition is performed. This involves acquiring the real-time video stream transmitted from the monitoring terminal and the saved offline video.
[0048] In step S2, data cleaning is performed. The decoded video frames are cropped, scaled, and then integrated into the original dataset.
[0049] In step S3, data annotation is performed. The original dataset is labeled and classified using annotation tools, and then converted into text data that the head and shoulder detection model can run.
[0050] In step S4, the head and shoulder detection model is trained. The labeled raw dataset is input into the head and shoulder detection model for training, and the parameters of the head and shoulder detection model are adjusted.
[0051] In step S5, it is determined whether the head and shoulder detection model has met the preset training conditions; if the preset training conditions are met, then in step S6, the pre-trained head and shoulder detection model is obtained; if the preset training conditions are not met, then steps S1-S5 are executed again.
[0052] Among them, the preset training conditions can be set according to actual needs, such as when the number of training rounds reaches a certain number, or when the accuracy of the head and shoulder detection model is greater than the accuracy threshold.
[0053] The training process for the non-motorized vehicle detection model in this application is the same as that for the head and shoulder detection model, and will not be described in detail here.
[0054] In step S202, a tracking algorithm is used to track the head and shoulder regions of the non-motorized vehicle and at least one reference object to determine the first trajectory information of the non-motorized vehicle and the second trajectory information of at least one reference object.
[0055] Specifically, the head and shoulder regions of the non-motorized vehicle and at least one reference object obtained after detection are tracked using the deepsort algorithm to obtain the first trajectory information of the non-motorized vehicle and the second trajectory information of at least one reference object.
[0056] If there are multiple reference objects, the deepsort algorithm is used to track the head and shoulders of each reference object to obtain the second trajectory information of each reference object.
[0057] In step S203, based on the first trajectory information and the second trajectory information, the positions of the non-motorized vehicle and at least one reference object are compared to determine the associated body; the associated body includes the target object among the non-motorized vehicle and at least one reference object.
[0058] In one possible implementation, multiple reference objects may appear in a video frame. Therefore, it is necessary to determine the association between the reference objects and the non-motorized vehicle, filter out reference objects unrelated to the non-motorized vehicle, and identify reference objects related to the non-motorized vehicle. Therefore, based on the first trajectory information and the second trajectory information, the positions of the non-motorized vehicle and at least one reference object are compared to determine the associated entity. For each reference object, the process can be executed as follows: Figure 5 As shown: In step S501, based on the first trajectory information and the second trajectory information, the distance characteristics, height ratio characteristics, and intersection-to-merge ratio characteristics between the non-motorized vehicle and the reference object are determined.
[0059] Among them, the distance feature is the distance between the center point of the non-motorized vehicle and the center point of the reference object; the height ratio feature is the ratio between the height of the non-motorized vehicle and the height of the reference object; and the intersection-union ratio feature is the intersection-union ratio between the bounding box of the non-motorized vehicle and the bounding box of the reference object.
[0060] The center point of the reference object is the center point of the head and shoulder region of the reference object; the height of the reference object is the height of the head and shoulder region of the reference object; the bounding box of the reference object is the bounding box of the head and shoulder region of the reference object.
[0061] In step S502, the association probability between the non-motorized vehicle and the reference object is determined based on distance features, height ratio features, and intersection-to-union ratio features.
[0062] In step S503, if the association probability is greater than the preset association probability, the reference object is taken as the target object, and the non-motorized vehicle and the target object are determined to be associated entities.
[0063] In practice, based on the non-motorized vehicles detected by the non-motorized vehicle detection model and at least one reference object detected by the head and shoulder detection model, a correlation body between the head and shoulder regions of the non-motorized vehicle and the target object is constructed. The distance characteristics, height ratio characteristics, and intersection-union ratio (IOU) characteristics of the head and shoulder regions of each reference object in the non-motorized vehicle region are represented by a multiple linear regression model to obtain the correlation probability between each reference object and the non-motorized vehicle. Reference objects with correlation probabilities less than the preset correlation probability are removed, i.e., reference objects that do not belong to the non-motorized vehicle category, thus obtaining the correlation body between the head and shoulder regions of the non-motorized vehicle and the target object.
[0064] The probability of association between each reference object and a non-motorized vehicle is represented by a multiple linear regression model (1): (1) in, It is the probability of association between the reference object and the non-motorized vehicle; It is the distance characteristic between the non-motorized vehicle and the reference object; It is the height ratio characteristic between non-motorized vehicles and reference objects; It is the intersection-combination ratio characteristic between non-motorized vehicles and reference objects; , , These are the weight coefficients corresponding to the distance feature, height ratio feature, and crossover ratio feature, respectively. They are obtained by the multiple linear regression model through learning and represent the importance of the distance feature, height ratio feature, and crossover ratio feature to the association probability. It is the intercept of the multiple linear regression model.
[0065] In step S204, the motion correlation between the associated body and other traffic objects within the preset area of the associated body is determined.
[0066] In one possible implementation, to avoid the non-motorized vehicle in the associated entity being mistakenly associated with the target object, this application can also exclude mistakenly associated entities through spatiotemporal information.
[0067] Therefore, based on the trajectory information of the associated object and other traffic objects within its preset area, determining the motion correlation between the associated object and other traffic objects can be performed as follows: Figure 6 The steps shown are as follows: In step S601, the intersection-merger ratio between the associated body and other traffic objects within the preset area of the associated body is calculated based on the trajectory information of the associated body and other traffic objects.
[0068] Other traffic objects within the preset area include people, motor vehicles, and other non-motorized vehicles near the associated object. The preset area can be set according to actual needs, such as within 5 meters of the associated object.
[0069] For each of the other traffic objects, perform steps S602 and S603 respectively: In step S602, if the cross-over ratio between the associated body and the traffic object is greater than the preset cross-over ratio threshold, then the motion trajectories of the associated body and the traffic object in all video frames within a preset time window containing video frames are obtained.
[0070] The preset intersection-over-union (IoU) threshold can be set according to actual needs, and this application does not impose any restrictions on it. The preset time window can also be set according to actual needs, such as including the 20 frames before and after the video frame.
[0071] Specifically, the cross-over ratio (CUP) between the associated object and the traffic object is greater than the preset CUP threshold when the average CUP of the bounding boxes of the associated object and the traffic object is greater than the preset CUP threshold within N consecutive frames of a preset time window.
[0072] In step S603, the motion correlation between the associated objects and traffic objects is calculated based on the motion trajectories of the associated objects and traffic objects in all video frames.
[0073] Specifically, based on the motion trajectories of the associated objects and traffic objects in all video frames, the motion correlation between the associated objects and traffic objects is calculated as follows: within N consecutive frames of a preset time window, the motion correlation between the motion trajectories of the associated objects and traffic objects is linearly positive.
[0074] In practice, after confirming the associated entity, the trajectory information of pedestrians, motor vehicles, and other non-motorized vehicles around the non-motorized vehicle is obtained. Then, based on the trajectory information of the associated entity and nearby pedestrians, motor vehicles, and other non-motorized vehicles, the IOU between the associated entity and nearby pedestrians, motor vehicles, and other non-motorized vehicles is determined.
[0075] If the Intersection over Union (IOU) between the associated object and nearby pedestrians, motor vehicles, and other non-motorized vehicles exceeds a preset threshold, then the motion trajectories of nearby pedestrians, motor vehicles, and other non-motorized vehicles within the preceding and following 20 frames containing video frames are further obtained; then the motion correlation between the associated object and nearby pedestrians, motor vehicles, and other non-motorized vehicles is calculated based on the motion trajectories of the associated object and nearby pedestrians, motor vehicles, and other non-motorized vehicles within the preceding and following 20 frames.
[0076] In step S205, if the motion correlation between the associated entity and at least one other traffic object is not positive, it is determined that the non-motorized vehicle in the associated entity is illegally carrying passengers.
[0077] In one possible implementation, if the motion correlation between the associated entity and other traffic objects is linearly positive, then it is determined that there is no illegal passenger-carrying behavior among the non-motorized vehicles in the associated entity.
[0078] Specifically, if within N consecutive frames of a preset time window, the average intersection-union ratio (IUR) of the bounding boxes of the associated object and the traffic object is greater than a preset IUR threshold, and the motion correlation between the motion trajectories of the associated object and the traffic object is linearly positive, then it is determined that the non-motorized vehicle in the associated object is falsely associated with the target object. In this case, the subordinate relationship of the associated object is removed, and it is determined that the non-motorized vehicle in the associated object does not have illegal passenger-carrying behavior. Otherwise, it is determined that the non-motorized vehicle in the associated object has illegal passenger-carrying behavior.
[0079] In one possible implementation, calculating the motion correlation between the associated objects and traffic objects based on their motion trajectories across all video frames can be performed as follows: If the motion trajectories of the associated objects and traffic objects in all video frames meet the preset similarity conditions, then the motion correlation between the associated objects and traffic objects is determined to be linear and positive.
[0080] For example, the motion trajectories of the associated object and other traffic objects within 20 frames satisfy... If there is a positive correlation, then the motion correlation between the associated entity and the traffic object is determined to be a linear positive correlation.
[0081] The preset similarity condition is determined by the linear fit between the two motion trajectories in the two-dimensional image coordinate system or the world coordinate system.
[0082] In this application, the motion correlation between the associated body and the traffic object can also be determined by calculating the Pearson coefficient between their motion trajectories. Specifically, this can be done as follows: First, calculate the correlation coefficients of the two motion trajectory sequences of the associated object and the traffic object in the X and Y directions, and then take their average or minimum values. If the Pearson coefficient is greater than a threshold, such as 0.8, the motion correlation between the associated object and the traffic object is considered to be linear and positive.
[0083] In this application, the motion correlation between the associated body and the traffic object can also be determined by calculating the trajectory distance between their motion trajectories. Specifically, this can be performed as follows: The morphological similarity between the motion trajectories of the associated object and the traffic object is calculated using dynamic time warping or Euclidean distance. If the trajectory distance is less than a threshold, the motion correlation between the associated object and the traffic object is considered to be linearly positive.
[0084] In this application, the motion correlation between the associated body and the traffic object can also be determined by calculating the similarity of the velocity vectors between their trajectories. Specifically, this can be done as follows: First, calculate the velocity vectors of the two motion trajectories of the associated object and the traffic object across all video frames (between the first and last 20 frames), and analyze the cosine value of the angle between the two velocity vectors. If the average value is close to 1, then the motion correlation between the associated object and the traffic object is determined to be linear and positive.
[0085] This not only ensures the security and timeliness of traffic data, but also allows for more accurate monitoring of illegal passenger transport by non-motorized vehicles, thereby guaranteeing road safety.
[0086] It should be noted that the detection method for non-motorized vehicle violations provided in this application is applied to the detection of illegal passenger carrying on two-wheeled vehicles, tricycles, and trucks. Based on extensive experiments, the detection accuracy of the non-motorized vehicle violation detection method provided in this application is between 85% and 90%. Figure 7 The image shows the detection results of two-wheeled vehicles illegally carrying passengers. The two-wheeled vehicles within the detection frame are found to be illegally carrying passengers. Figure 8 The image shows the detection results of tricycles illegally carrying passengers. The tricycles within the detection frame are found to be illegally carrying passengers.
[0087] In one possible implementation, if it is determined that a non-motorized vehicle in the associated entity is illegally carrying passengers, a warning message is sent to the user so that the user can take relevant actions against the non-motorized vehicle and the target object in the associated entity based on the warning message.
[0088] In practice, after detecting whether non-motorized vehicles in the associated group are illegally carrying passengers, the associated group and the corresponding detection results are uploaded to the database for storage. The associated group and the corresponding detection results are also sent to traffic police and other law enforcement personnel to remind them that non-motorized vehicles in the associated group are illegally carrying passengers, so that traffic police and other law enforcement personnel can deal with illegal passenger carrying in a timely manner.
[0089] Based on the foregoing description, this application detects the head and shoulder regions of non-motorized vehicles and at least one reference object in a video frame, and then uses a tracking algorithm to track the trajectories of the non-motorized vehicles and at least one reference object. Based on the trajectory information, the positions of the non-motorized vehicles and at least one reference object are compared to determine the target object associated with the non-motorized vehicle. Furthermore, by analyzing the motion correlation between the associated object and other traffic objects within a preset area of the associated object, it is determined whether the non-motorized vehicle in the associated object is illegally carrying passengers. This method can more accurately monitor the illegal carrying of passengers by non-motorized vehicles than the traditional method of identifying faces on non-motorized vehicles in video frames, thereby ensuring road safety.
[0090] Based on the same inventive concept, this application also provides a device for detecting non-motorized vehicle violations. Figure 9 This is a schematic diagram of a non-motorized vehicle violation detection device provided in an embodiment of this application. The device includes a data acquisition module 901, a target detection module 902, a trajectory tracking module 903, a related entity determination module 904, a relationship determination module 905, and a behavior determination module 906. The data acquisition module 901 is used to acquire traffic monitoring videos of non-motorized vehicles; The target detection module 902 is used to perform head and shoulder detection and non-motorized vehicle detection on video frames in traffic monitoring videos, and to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video. The trajectory tracking module 903 is used to track the head and shoulder regions of a non-motorized vehicle and at least one reference object using a tracking algorithm, and to determine the first trajectory information of the non-motorized vehicle and the second trajectory information of at least one reference object. The associated body determination module 904 is used to compare the positions of the non-motorized vehicle and at least one reference object based on the first trajectory information and the second trajectory information to determine the associated body; the associated body includes the target object among the non-motorized vehicle and at least one reference object. The relationship determination module 905 is used to determine the motion correlation between the associated object and other traffic objects based on the trajectory information of the associated object and other traffic objects within the preset area of the associated object. The behavior determination module 906 is used to determine that a non-motorized vehicle in the associated entity is illegally carrying passengers if the motion correlation between the associated entity and at least one other traffic object is not positive.
[0091] In one possible implementation, the target detection module 902 is specifically used for: Using a pre-trained head and shoulder detection model, head and shoulder detection is performed on the traffic monitoring video to determine the head and shoulder region of at least one reference object in the traffic monitoring video; Using a pre-trained non-motorized vehicle detection model, non-motorized vehicles are detected in the traffic monitoring video to identify them.
[0092] In one possible implementation, the associated entity determination module 904 is specifically used for: For each reference object, based on the first trajectory information and the second trajectory information, the distance feature, height ratio feature, and intersection-union ratio feature between the non-motorized vehicle and the reference object are determined; the distance feature is the distance between the center point of the non-motorized vehicle and the center point of the reference object; the height ratio feature is the ratio between the height of the non-motorized vehicle and the height of the reference object; the intersection-union ratio feature is the intersection-union ratio between the bounding box of the non-motorized vehicle and the bounding box of the reference object. Based on the distance feature, the height ratio feature, and the intersection-union ratio feature, the association probability between the non-motorized vehicle and the reference object is determined; If the association probability is greater than the preset association probability, then the reference object is taken as the target object, and the non-motorized vehicle and the target object are determined to be associated entities.
[0093] In one possible implementation, the relationship determination module 905 is specifically used for: Based on the trajectory information of the associated entity and other traffic objects within the preset area of the associated entity, calculate the intersection-combination ratio between the associated entity and other traffic objects; For each of the other traffic objects, perform the following operations: If the intersection-over-union ratio between the associated body and the traffic object is greater than a preset intersection-over-union ratio threshold, then the motion trajectories of the associated body and the traffic object in all video frames within a preset time window containing the video frame are obtained. Based on the motion trajectories of the associated body and the traffic object in all video frames, the motion correlation between the associated body and the traffic object is calculated.
[0094] In one possible implementation, the behavior determination module 906 is further configured to: If the motion correlation between the associated entity and the other traffic objects is linearly positive, then it is determined that the non-motorized vehicle in the associated entity does not have illegal passenger-carrying behavior.
[0095] In one possible implementation, the relationship determination module 905 is specifically used for: If the motion trajectories of the associated object and the traffic object in all video frames meet a preset similarity condition, then the motion correlation between the associated object and the traffic object is determined to be linearly positive.
[0096] In one possible implementation, the relationship determination module 905 is specifically used for: The preset similarity conditions are determined by the linear fit between the two motion trajectories in the two-dimensional image coordinate system or the world coordinate system.
[0097] Based on the same inventive concept, this application also provides an electronic device. Figure 10 This application provides a schematic diagram of an electronic device structure, such as... Figure 10 As shown, it includes: processor 1001, communication interface 1002, memory 1003 and communication bus 1004, wherein processor 1001, communication interface 1002 and memory 1003 communicate with each other through communication bus 1004. The memory 1003 stores a computer program, which, when executed by the processor 1001, causes the processor 1001 to perform the steps of any of the non-motor vehicle violation detection methods provided in the embodiments of this application.
[0098] Since the problem-solving method of the above-mentioned electronic device is similar to the detection method of non-motorized vehicle violations, the implementation of the above-mentioned electronic device can be referred to the embodiments of the method, and the repeated parts will not be described again.
[0099] The communication bus mentioned in the aforementioned electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not indicate that there is only one bus or one type of bus. The communication interface 1302 is used for communication between the aforementioned electronic device and other devices. The memory can include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.
[0100] The processors mentioned above can be general-purpose processors, including central processing units, network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits, field-programmable gate arrays or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.
[0101] Based on the above embodiments, this invention also provides a computer-readable storage medium storing a computer program executable by a processor. When the program runs on the processor, it causes the processor to perform the steps of any of the non-motorized vehicle violation detection methods provided in this application.
[0102] Based on the same inventive concept, embodiments of this application provide a computer program product, including a computer program, which, when executed by a processor, implements the steps of any of the non-motorized vehicle violation detection methods provided in embodiments of this application.
[0103] Since the principle of the computer-readable storage medium in solving the problem is similar to the method for detecting non-motorized vehicle violations, the implementation of the computer-readable storage medium can be found in the embodiments of the method, and repeated details will not be repeated.
[0104] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0105] This application is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0106] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0107] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.
[0108] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the spirit and scope of this application. Therefore, if such modifications and variations fall within the scope of the claims of this application and their equivalents, this application also intends to include such modifications and variations.
Claims
1. A method for detecting non-motorized vehicle violations, characterized in that, The method includes: Obtain traffic monitoring videos of non-motorized vehicles; Head and shoulder detection and non-motorized vehicle detection are performed on video frames in the traffic monitoring video to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video. A tracking algorithm is used to track the head and shoulder regions of the non-motorized vehicle and the at least one reference object to determine the first trajectory information of the non-motorized vehicle and the second trajectory information of the at least one reference object. Based on the first trajectory information and the second trajectory information, the positions of the non-motorized vehicle and the at least one reference object are compared to determine the associated object; the associated object includes the target object among the non-motorized vehicle and the at least one reference object. Based on the trajectory information of the associated object and other traffic objects within the preset area of the associated object, the motion correlation between the associated object and other traffic objects is determined; If the motion correlation between the associated entity and at least one other traffic object is not positive, then it is determined that the non-motorized vehicle in the associated entity is illegally carrying passengers.
2. The method according to claim 1, characterized in that, The step of performing head and shoulder detection and non-motorized vehicle detection on the traffic monitoring video to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video includes: Using a pre-trained head and shoulder detection model, head and shoulder detection is performed on the traffic monitoring video to determine the head and shoulder region of at least one reference object in the traffic monitoring video; Using a pre-trained non-motorized vehicle detection model, non-motorized vehicles are detected in the traffic monitoring video to identify them.
3. The method according to claim 1, characterized in that, The step of comparing the positions of the non-motorized vehicle and the at least one reference object based on the first trajectory information and the second trajectory information to determine the associated entity includes: For each reference object, based on the first trajectory information and the second trajectory information, the distance feature, height ratio feature, and intersection-union ratio feature between the non-motorized vehicle and the reference object are determined; the distance feature is the distance between the center point of the non-motorized vehicle and the center point of the reference object; the height ratio feature is the ratio between the height of the non-motorized vehicle and the height of the reference object; the intersection-union ratio feature is the intersection-union ratio between the bounding box of the non-motorized vehicle and the bounding box of the reference object. Based on the distance feature, the height ratio feature, and the intersection-union ratio feature, the association probability between the non-motorized vehicle and the reference object is determined; If the association probability is greater than the preset association probability, then the reference object is taken as the target object, and the non-motorized vehicle and the target object are determined to be associated entities.
4. The method according to claim 1, characterized in that, The step of determining the motion correlation between the associated entity and other traffic objects within a preset area based on the trajectory information of the associated entity and other traffic objects includes: Based on the trajectory information of the associated entity and other traffic objects within the preset area of the associated entity, calculate the intersection-combination ratio between the associated entity and other traffic objects; For each of the other traffic objects, perform the following operations: If the intersection-over-union ratio between the associated body and the traffic object is greater than a preset intersection-over-union ratio threshold, then the motion trajectories of the associated body and the traffic object in all video frames within a preset time window containing the video frame are obtained. Based on the motion trajectories of the associated body and the traffic object in all video frames, the motion correlation between the associated body and the traffic object is calculated.
5. The method according to claim 1, characterized in that, The method further includes: If the motion correlation between the associated entity and the other traffic objects is linearly positive, then it is determined that the non-motorized vehicle in the associated entity does not have illegal passenger-carrying behavior.
6. The method according to claim 5, characterized in that, The step of calculating the motion correlation between the associated object and the traffic object based on the motion trajectories of the associated object and the traffic object in all video frames includes: If the motion trajectories of the associated object and the traffic object in all video frames meet a preset similarity condition, then the motion correlation between the associated object and the traffic object is determined to be linearly positive.
7. The method according to claim 6, characterized in that, The preset similarity condition is determined by the linear fit of the two motion trajectories in the two-dimensional image coordinate system or the world coordinate system.
8. A device for detecting non-motorized vehicle violations, characterized in that, The device includes: The data acquisition module is used to acquire traffic monitoring videos of non-motorized vehicles; The target detection module is used to perform head and shoulder detection and non-motorized vehicle detection on video frames in the traffic monitoring video, and to determine the head and shoulder regions of non-motorized vehicles and at least one reference object in the traffic monitoring video. The trajectory tracking module is used to track the head and shoulder regions of the non-motorized vehicle and the at least one reference object using a tracking algorithm, and to determine the first trajectory information of the non-motorized vehicle and the second trajectory information of the at least one reference object. The associated object determination module is used to compare the positions of the non-motorized vehicle and the at least one reference object based on the first trajectory information and the second trajectory information to determine the associated object; the associated object includes the target object among the non-motorized vehicle and the at least one reference object; The relationship determination module is used to determine the motion correlation between the associated entity and other traffic objects within a preset area of the associated entity. The behavior determination module is used to determine that the non-motorized vehicle in the associated entity has engaged in illegal passenger-carrying behavior if the motion correlation between the associated entity and at least one other traffic object is not positive.
9. An electronic device, characterized in that, The electronic device includes a processor, which executes a computer program stored in a memory to implement the steps of the non-motorized vehicle violation detection method as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, It stores a computer program, which, when executed by a processor, implements the steps of the method for detecting non-motorized vehicle violations as described in any one of claims 1-7.