Method for detecting passenger and freight mixed loading behavior of minibus

Abstract

According to the method for detecting the passenger and freight mixed loading behavior of the minibus, the minibus is recognized through the minibus side shooting target recognition model, then the minibus side row picture is detected based on the vehicle window side shooting target recognition model, the vehicle window target is recognized, and the passenger and freight mixed loading behavior of the minibus is obtained. And vehicle window targets in all vehicle window areas are identified through a cargo carrying feature identification model, vehicle windows with cargo carrying features are found, and real-time early warning information is generated in combination with vehicle license plate information and equipment information of roadside infrared monitoring equipment. Pictures are collected in real time based on the roadside infrared monitoring equipment, visible light interference to the view of a driver is avoided, and it is ensured that the to-be-recognized image can be rapidly and safely obtained; new equipment does not need to be added in the whole process, and the recognition rate of the passenger and freight mixed loading minibus can be rapidly and accurately improved based on low cost.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent traffic control, and in particular to a method for detecting mixed passenger and cargo loading behavior in a van. Background Art

[0002] According to relevant regulations, passenger cars are not allowed to carry cargo except for luggage racks on the outside of the car body and built-in luggage boxes. When a passenger and cargo van is braking, accelerating, or turning, the cargo inside the car may be bumped, hit, punctured, or otherwise damaged, which can easily cause casualties. If the cargo is too heavy, it will affect the vehicle's driving and braking functions, creating serious safety hazards.

[0003] Although the video or image acquisition equipment installed at the checkpoints on the highway can identify abnormal behaviors such as vehicle violations based on image recognition technology, most of the road monitoring equipment installed at the checkpoints only captures the front or rear of the vehicle, and it is difficult to capture the passengers and cargo in the vehicle. For mixed passenger and cargo behavior, road management personnel often use road inspections as a means of control, which costs a lot of manpower and is inefficient. Summary of the invention

[0004] In order to solve the problem of high manpower and time costs in the prior art for discovering and investigating the mixed loading of passengers and cargo in vans, the present invention provides a detection method for the mixed loading of passengers and cargo in vans, which can automatically identify the mixed loading of passengers and cargo in vans by using existing equipment, has low implementation cost, and effectively improves the recognition rate of this behavior.

[0005] The technical solution of the present invention is as follows: a method for detecting mixed passenger and cargo loading behavior in a van, characterized in that it comprises the following steps: S1: Determine a detection area, in which a roadside infrared monitoring device is installed; Taking the van as the target vehicle, the right side picture of the target vehicle's forward direction is collected based on the roadside infrared monitoring equipment, which is recorded as: the picture to be analyzed; S2: marking the vehicle outline of the target vehicle in the image to be analyzed to obtain a side-shot vehicle target detection image; Based on the side-view vehicle target detection pictures, construct a training set for side-view target detection of vans; Building a van side-shot target recognition model based on a deep learning algorithm, and training the van side-shot target recognition model based on the van side-shot target detection training set to obtain the trained van side-shot target recognition model; S3: marking all vehicle windows in the side-view vehicle target detection image to obtain a side-view vehicle window detection image; Based on the side-view window detection pictures, construct a training set for window side-view target detection; Based on the deep learning algorithm, a vehicle window side-shot target recognition model is constructed, and the vehicle window side-shot target recognition model is trained based on the vehicle window side-shot target detection training set to obtain the trained vehicle window side-shot target recognition model; S4: Classifying the side-shot vehicle target detection images according to the characteristics of loaded and unloaded vehicles to obtain loaded vehicle images and unloaded vehicle images; For the cargo vehicle pictures, the types of the cargo features in the vehicle are classified to confirm the types of cargo features in the vehicles passing through the detection area; for each type of cargo vehicle picture, the window positions of the cargo features in the cargo vehicle pictures are marked to obtain cargo classification feature pictures; For the image of the vehicle without cargo, mark the position of the vehicle windows in the image of the vehicle without cargo to obtain a feature image of the vehicle without cargo; S5: Build a cargo feature recognition model based on deep learning algorithm; S6: Find all types of cargo classification feature images and non-cargo feature images in the historical image set, and construct a training set and a test set; train the cargo feature recognition model based on the training set to obtain the trained cargo feature recognition model; S7: Based on the roadside infrared monitoring device in the detection area, a side-shot image of the vehicle is acquired in real time, which is recorded as a side-shot image to be detected; S8: inputting the side-shot image to be detected into the van side-shot target recognition model in real time, and the van side-shot target recognition model outputs an image with a side-shot van, which is recorded as: a vehicle window image to be recognized; S9: inputting the vehicle window image to be identified into the vehicle window side-shot target recognition model, and outputting all the identified vehicle window elements according to the output of the vehicle window side-shot target recognition model, which is recorded as: a vehicle window target set to be identified; S10: taking each vehicle window in the target set of vehicle windows to be identified as a target vehicle window to be identified, inputting the window into the cargo feature recognition model, and outputting a vehicle window with vehicle cargo features, recorded as a cargo vehicle window; S11: Based on each of the truck windows, find the corresponding vehicle license plate information, push it together with the information of the roadside infrared monitoring equipment to the management platform, and generate warning information.

[0006] It is further characterized by: In step S1, a roadside infrared monitoring device and a checkpoint image acquisition device located in the direction of road travel are installed in the detection area at the same time; Step S7 also includes the following steps: Based on the said bayonet image acquisition device, the forward or reverse pictures of all target vehicles passing by are collected, which are recorded as: pictures for confirming vehicle license plate information; Step S11 also includes the following steps: Find the corresponding vehicle license plate information confirmation image for each of the truck windows, identify the corresponding vehicle license plate type and license plate number information; push the vehicle license plate type, license plate number, the side-shot image to be detected, the vehicle license plate information confirmation image, the device number and device location information of the roadside infrared monitoring device to the management platform, and generate warning information; In step S6, during the process of training the cargo feature recognition model based on the training set, after each round of training, the cargo feature recognition model is tested using all the feature test set images, and the accuracy of the model output is statistically calculated. If the accuracy is not less than the expected accuracy threshold, the model training is completed, otherwise, the training set and model training are continued; In step S6, the method for constructing the training set and the test set includes the following steps: a1: Confirm the type of cargo characteristics in the vehicle; a2: Determine the proportion of each type of cargo-carrying and non-carrying features in the total image set; a3: Divide the pictures in the historical picture set into 80% and 20% ratios to obtain the training set and the test set; a4: Ensure that the proportion of each type of in-vehicle cargo feature and unloaded cargo feature in the test set is the same as the corresponding proportion in the training set.

[0007] The present application provides a method for detecting the mixed passenger and cargo behavior of a van. The van is identified by a van side-shot target recognition model, and then the van side row image is detected based on the window side-shot target recognition model to identify the window target. Then, the window target of all the window areas is identified by the cargo feature recognition model to find the window with cargo features, that is, the cargo truck window, and then the real-time warning information is generated in combination with the vehicle license plate information and the equipment information of the roadside infrared monitoring equipment. The present method makes full use of the pictures collected in real time by the roadside infrared monitoring equipment, uses the cargo feature recognition model to accurately find the van with illegal cargo features, and issues warning information in real time; the present application is based on the real-time collection of pictures by the roadside infrared monitoring equipment, and there is no visible light interference to the driver's field of view, ensuring that the image to be identified can be obtained quickly and safely; the whole process does not require the addition of new equipment, and can quickly and accurately improve the recognition rate of the mixed passenger and cargo van at a relatively low cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Figure 1 Schematic diagram of the flow of the method for detecting mixed passenger and cargo loading behavior in a van. DETAILED DESCRIPTION

[0009] The existing camera image acquisition equipment on the road can only capture the forward or reverse images of the vehicle, and cannot fully capture the internal features of the vehicle. The infrared monitoring equipment on the roadside cannot collect the license plate information of the vehicle. Therefore, this application is based on the infrared monitoring equipment installed on the roadside to collect the side images of the vehicle and identify whether the vehicle is loaded with a large amount of cargo. At the same time, the camera image acquisition equipment is used to collect the license plate information of the van, generate an early warning and push it to the management platform for subsequent tracking and processing.

[0010] like Figure 1 As shown, the present application includes a method for detecting mixed passenger and cargo loading behavior in a van, which includes the following steps.

[0011] S1: Determine the detection area, in which the roadside infrared monitoring equipment and the checkpoint image acquisition equipment located in the road driving direction are installed at the same time. When the checkpoint image acquisition equipment captures the picture of the passing vehicle, the roadside infrared monitoring equipment simultaneously captures the side picture.

[0012] Taking the van as the target vehicle, the forward or reverse images of all target vehicles passing by are collected based on the bayonet image acquisition device; Based on the roadside infrared monitoring equipment, collect the right side picture of the target vehicle's forward direction, which is recorded as: picture to be analyzed.

[0013] In this application, the side image of the vehicle is collected by roadside infrared monitoring equipment, which can capture the situation of people and cargo in the vehicle. At the same time, there is no visible light interference to the driver's field of vision, which will not affect the normal driving of the vehicle. There is no need to add other new equipment, which effectively controls the cost of implementing the solution.

[0014] S2: marking the vehicle outline of the target vehicle in the image to be analyzed to obtain a side-shot vehicle target detection image; Based on the side-view vehicle target detection images, i.e., the side-view images of all vans, a training set for van side-view target detection is constructed; A van side-view target recognition model is constructed based on a deep learning algorithm, and the van side-view target recognition model is trained based on a training set for van side-view target detection to obtain a trained van side-view target recognition model.

[0015] S3: marking all vehicle windows in the side-view vehicle target detection image to obtain a side-view vehicle window detection image; Based on the side-view car window detection images, a training set for side-view car window target detection is constructed; Based on the deep learning algorithm, a car window side camera target recognition model is constructed. The car window side camera target recognition model is trained based on the training set for car window side camera target detection to obtain a trained car window side camera target recognition model.

[0016] S4: Classify the side-shot vehicle target detection images according to the characteristics of cargo and no cargo, and obtain cargo vehicle images and no cargo vehicle images; for the cargo vehicle images, classify the types of cargo characteristics inside the vehicle, and confirm the types of cargo characteristics inside the vehicles passing through the detection area; for each type of cargo vehicle image, mark the window position of the cargo vehicle image with cargo characteristics, and obtain the cargo classification feature image; For the unloaded vehicle picture, the window positions in the unloaded vehicle picture are marked to obtain the unloaded feature picture.

[0017] In this application, a sufficient number of unloaded and loaded pictures of vans are selected from the side pictures of the vehicle captured by roadside monitoring equipment, and the types of common in-vehicle loading characteristics in the local area are determined based on the specific characteristics of the mixed passengers and cargo in the pictures, ensuring that the accuracy of subsequent detection can be improved.

[0018] S5: Build a cargo feature recognition model based on deep learning algorithms. In specific applications, build a cargo feature recognition model based on existing deep learning algorithms such as AlexNet, VGG or ResNet.

[0019] S6: In the historical image collection, find all types of cargo classification feature images and non-cargo feature images, and construct a training set and a test set; train the cargo feature recognition model based on the training set to obtain a trained cargo feature recognition model.

[0020] In step S6, during the training of the cargo feature recognition model based on the training set, after each round of training, the cargo feature recognition model is tested using all the feature test set images, and the accuracy of the model output is statistically calculated. If the accuracy is not less than the expected accuracy threshold, the model training is completed, otherwise the training set and model training are continued. The specific accuracy threshold is adaptively set according to the actual detection accuracy requirements. In this embodiment, the accuracy threshold is set to 90%.

[0021] In step S6, the method for constructing a training set and a test set includes the following steps: a1: Confirm the type of cargo characteristics in the vehicle; a2: Determine the proportion of each type of cargo-carrying and non-carrying features in the total image set; a3: Divide the pictures in the historical picture set into a ratio of 80% and 20% to obtain the training set and the test set; a4: Ensure that the proportion of each type of in-vehicle cargo feature and unloaded cargo feature in the test set is the same as the corresponding proportion in the training set.

[0022] In this embodiment, City A is selected as a case site. The city has built multiple sets of roadside infrared capture equipment. The side vehicle pictures of vans captured by various roadside monitoring equipment in June 2024 are collected and summarized, and the vehicle contours in the pictures are marked with rectangular frames. The pictures where the vehicle contours cannot be marked are discarded to obtain pictures for side-shot vehicle target detection.

[0023] For the side-view vehicle target detection pictures, all the windows in the pictures are marked respectively, that is, each window in the picture is marked one by one with a rectangular frame to obtain the side-view vehicle window detection pictures. Based on the side-view vehicle target detection pictures, the training of the van side-view target recognition model is completed to obtain the trained van side-view target recognition model. Based on the side-view vehicle window detection pictures, the training of the vehicle window side-view target recognition model is completed to obtain the trained vehicle window side-view target recognition model.

[0024] A sufficient number of van unloaded and loaded images are selected from the side images of vehicles captured by roadside monitoring equipment. From the mixed loading of passengers and cargo in the images, the main cargo characteristics of vehicles in City A are box type, can type, woven bag type, and long strip type. In order to ensure the effectiveness of training, in this embodiment, when selecting images from historical data, the minimum number of images of each type of cargo characteristics in the vehicle is no less than 1,000, and the minimum number of images of unloaded characteristics is no less than 1,000.

[0025] According to the cargo characteristics and unloaded characteristics of box-type, can-type, woven bag-type, and long strip-type, the side-view cargo pictures of local vans were sorted and organized. The window positions with cargo characteristics in the cargo vehicle pictures were marked, and the window positions in the unloaded vehicle pictures were marked. Based on the unloaded characteristics and each type of cargo characteristics, a training set and a test set of window pictures were created, with the number of pictures in the training set accounting for 80% and the number of pictures in the test set accounting for 20%. From the overall situation of the side-view vehicle pictures in City A, the number of unloaded vehicle pictures accounted for 75%, and the box-type, can-type, woven bag-type, and long strip-type cargo characteristics accounted for 10%, 5%, 5%, and 5% respectively. The proportion of each type of characteristic pictures in the training set and the test set refers to the above proportion settings.

[0026] A cargo feature recognition model is constructed based on a deep learning algorithm. All images in the cargo feature training set are used for training to obtain a trained cargo feature recognition model. The current cargo features of concern, such as box type, can type, woven bag type, and long strip type, can be identified through this model.

[0027] During the training process, all cargo feature test set images are used to test the cargo feature recognition model, and the accuracy of the recognition results output by the statistical model is calculated. In this embodiment, the preset model recognition accuracy threshold is set to 90%. If the statistically obtained accuracy is not less than the preset value, the model training is completed, otherwise the model training continues until the recognition result of the cargo feature recognition model meets the preset accuracy threshold requirement.

[0028] S7: Based on the roadside infrared monitoring equipment and the checkpoint image acquisition equipment in the detection area, a side shot image of the vehicle and a forward or reverse image of the vehicle are obtained in real time, which are recorded as: the side shot image to be detected and the image for confirming the vehicle license plate information; S8: Input the side-shot image to be detected into the van side-shot target recognition model in real time. The van side-shot target recognition model outputs an image with a side-shot van, which is recorded as: vehicle window image to be recognized.

[0029] S9: Input the vehicle window image to be identified into the vehicle window side-shot target recognition model, and output all the identified vehicle window elements according to the output of the vehicle window side-shot target recognition model, which is recorded as: the vehicle window target set to be identified.

[0030] S10: taking each vehicle window in the target set of vehicle windows to be identified as a target vehicle window to be identified, inputting the window into the cargo feature recognition model, and outputting the vehicle window with the vehicle cargo feature, which is recorded as: cargo vehicle window.

[0031] S11: Based on each truck window, find the corresponding vehicle license plate information, push it to the management platform together with the information of the roadside infrared monitoring device, and generate warning information. Specifically, find the corresponding vehicle license plate information confirmation image for each truck window, identify the corresponding vehicle license plate type and license plate number information; push the vehicle license plate type, license plate number, side-shot image to be detected, vehicle license plate information confirmation image, equipment number and equipment location information of the roadside infrared monitoring device to the management platform, and generate warning information.

[0032] After using the technical solution of the present invention, the side pictures of the vehicle captured by the roadside infrared monitoring equipment are collected and annotated for the side-shot target recognition and detection of the van; the vehicle target detection of the van is annotated with pictures for window target detection; the vehicle target detection model and the window target detection model are trained; the unloaded pictures and loaded pictures in the van are selected from the side pictures of the vehicle and the types of cargo features in the vehicle are determined; the picture training set and the test set are created based on the unloaded features and each type of cargo features; the cargo feature recognition model is obtained by training; the side pictures of the vehicle are obtained in real time and used as model input. If there are cargo features, a real-time warning is generated and pushed. The technical solution of the present invention has no visible light interference on the driver's field of vision, and can make full use of the captured pictures of the roadside infrared monitoring equipment to provide the traffic management department with accurate real-time warnings of passenger and cargo mixed loading behaviors, providing strong support for strengthening road traffic safety management.

Claims

1. A method for detecting mixed loading of passengers and cargo in a van, characterized in that: It includes the following steps: S1: Determine a detection area, in which a roadside infrared monitoring device is installed; Taking the van as the target vehicle, the right side picture of the target vehicle's forward direction is collected based on the roadside infrared monitoring equipment, which is recorded as: the picture to be analyzed; S2: marking the vehicle outline of the target vehicle in the image to be analyzed to obtain a side-shot vehicle target detection image; Based on the side-view vehicle target detection pictures, construct a training set for side-view target detection of vans; Building a van side-shot target recognition model based on a deep learning algorithm, and training the van side-shot target recognition model based on the van side-shot target detection training set to obtain the trained van side-shot target recognition model; S3: marking all vehicle windows in the side-view vehicle target detection image to obtain a side-view vehicle window detection image; Based on the side-view window detection pictures, construct a training set for window side-view target detection; Based on the deep learning algorithm, a vehicle window side-shot target recognition model is constructed, and the vehicle window side-shot target recognition model is trained based on the vehicle window side-shot target detection training set to obtain the trained vehicle window side-shot target recognition model; S4: Classifying the side-shot vehicle target detection images according to the characteristics of loaded and unloaded vehicles to obtain loaded vehicle images and unloaded vehicle images; For the cargo vehicle image, classify the types of the cargo features inside the vehicle to confirm the types of the cargo features inside the vehicle passing through the detection area; For each category of cargo vehicle images, mark the window positions with cargo features in the cargo vehicle images to obtain cargo classification feature images; For the image of the vehicle without cargo, mark the position of the vehicle windows in the image of the vehicle without cargo to obtain a feature image of the vehicle without cargo; S5: Build a cargo feature recognition model based on deep learning algorithm; S6: Find all types of cargo classification feature images and no cargo feature images in the historical image set, and construct a training set and a test set; Training the cargo feature recognition model based on the training set to obtain a trained cargo feature recognition model; S7: Based on the roadside infrared monitoring device in the detection area, a side-shot image of the vehicle is acquired in real time, which is recorded as a side-shot image to be detected; S8: inputting the side-shot image to be detected into the van side-shot target recognition model in real time, and the van side-shot target recognition model outputs an image with a side-shot van, which is recorded as: a vehicle window image to be recognized; S9: inputting the vehicle window image to be identified into the vehicle window side-shot target recognition model, and outputting all the identified vehicle window elements according to the output of the vehicle window side-shot target recognition model, which is recorded as: a vehicle window target set to be identified; S10: taking each vehicle window in the target set of vehicle windows to be identified as a target vehicle window to be identified, inputting the window into the cargo feature recognition model, and outputting a vehicle window with vehicle cargo features, recorded as a cargo vehicle window; S11: Based on each of the truck windows, find the corresponding vehicle license plate information, push it together with the information of the roadside infrared monitoring equipment to the management platform, and generate warning information.

2. According to claim 1, a method for detecting mixed loading of passengers and cargo in a van is characterized by: In step S1, a roadside infrared monitoring device and a checkpoint image acquisition device located in the direction of road travel are installed in the detection area at the same time.

3. According to claim 1, a method for detecting mixed loading of passengers and cargo in a van is characterized by: Step S7 also includes the following steps: Based on the said bayonet image acquisition device, the forward or reverse pictures of all target vehicles passing by are collected and recorded as: pictures for confirming vehicle license plate information.

4. According to claim 1, a method for detecting mixed loading of passengers and cargo in a van is characterized by: Step S11 also includes the following steps: For each of the truck windows, the corresponding image for confirming the vehicle license plate information is found, and the corresponding vehicle license plate type and license plate number information are identified; the vehicle license plate type, license plate number, the side-shot image to be detected, the image for confirming the vehicle license plate information, the equipment number and equipment location information of the roadside infrared monitoring equipment are pushed together to the management platform to generate early warning information.

5. According to claim 1, a method for detecting mixed loading of passengers and cargo in a van is characterized by: In step S6, during the process of training the cargo feature recognition model based on the training set, after each round of training, the cargo feature recognition model is tested using all the feature test set images, and the accuracy of the model output is statistically calculated. If the accuracy is not less than the expected accuracy threshold, the model training is completed, otherwise the training set and model training continue to be constructed.

6. The method for detecting mixed loading of passengers and cargo in a van according to claim 5, characterized in that: In step S6, the method for constructing the training set and the test set includes the following steps: a1: Confirm the type of cargo characteristics in the vehicle; a2: Determine the proportion of each type of cargo-carrying and non-carrying features in the total image set; a3: Divide the pictures in the historical picture set into 80% and 20% ratios to obtain the training set and the test set; a4: Ensure that the proportion of each type of in-vehicle cargo feature and unloaded cargo feature in the test set is the same as the corresponding proportion in the training set.

Images