Vehicle detection method, terminal and computer-readable storage medium thereof

Abstract

The present invention discloses a vehicle detection method, a terminal and a computer-readable storage medium thereof, wherein the vehicle detection method comprises: obtaining an image to be detected; detecting a face image and its confidence, a non-motor vehicle image and its confidence, and the type of each non-motor vehicle in the image to be detected; using the confidence of the face image and the confidence of the non-motor vehicle image to form at least one group of human-vehicle detection pairs with the face image and the non-motor vehicle image; comparing the face image of each group of human-vehicle detection pairs with the face images in other human-vehicle detection pairs, and determining the face images with a coincidence degree higher than a first threshold value as the same passenger; and comparing each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, and determining the non-motor vehicle images with a coincidence degree higher than a second threshold value as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle. The present invention effectively improves the determination result of the non-motor vehicle carrying situation and improves the detection accuracy.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a vehicle detection method, a terminal and a computer-readable storage medium thereof. Background Art

[0002] In my country, non-motor vehicles are very important means of transportation, and many people use them for short-distance travel. Due to the difficulty of supervision and the lack of protective equipment, non-motor vehicles are particularly vulnerable to injury when participating in road traffic, and the illegal carrying of passengers has exacerbated this phenomenon. At present, with the development of computer vision technology, image-based intelligent applications are becoming more and more popular, but the application of such technologies in non-motor vehicle scenarios still has many limitations. Due to the lack of a reliable mechanism to determine the coupling relationship between people and non-motor vehicles, this method is prone to false alarms in dense scenes; and for a frame of the picture, two modules must be used to detect it separately, which has a high time complexity. Summary of the invention

[0003] The main technical problem solved by the present invention is to provide a vehicle detection method, a terminal and a computer-readable storage medium thereof, so as to solve the problem of inaccurate judgment of non-motorized vehicle manned conditions in the prior art.

[0004] To solve the above technical problems, the first technical solution adopted by the present invention is: to provide a vehicle detection method, which includes: acquiring an image to be detected; detecting a face image and its confidence, a non-motor vehicle image and its confidence, and the type of each non-motor vehicle in the image to be detected; using the confidence of the face image and the confidence of the non-motor vehicle image to form at least one group of human-vehicle detection pairs with the face image and the non-motor vehicle image; comparing the face image of each group of human-vehicle detection pairs with the face images in other human-vehicle detection pairs, and determining the face images with a degree of overlap higher than a first threshold as the same passenger; and comparing each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, and determining the non-motor vehicle images with a degree of overlap higher than a second threshold as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle.

[0005] Among them, the facial images of each group of person-vehicle detection pairs are compared with the facial images in other person-vehicle detection pairs, and the facial images with a degree of overlap higher than a first threshold are determined to be the same passenger; and each non-motor vehicle image is compared with the non-motor vehicle images of other person-vehicle detection pairs, and the non-motor vehicle images with a degree of overlap higher than a second threshold are determined to be the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle. The step also includes: if the non-motor vehicle images in different person-vehicle detection pairs correspond to the same non-motor vehicle, and the passengers corresponding to the facial images are different passengers, it is determined that different users are riding the same non-motor vehicle.

[0006] Among them, the step of using the confidence of the face image and the confidence of the non-motor vehicle image to form at least one group of human-vehicle detection pairs with the face image and the non-motor vehicle image includes: weighting the confidence of each face image with the confidence of each non-motor vehicle image in turn to obtain the confidence score of each group of face images and non-motor vehicle images; combining the face image and the non-motor vehicle image whose confidence score is not less than the preset score to determine it as a human-vehicle detection pair.

[0007] Among them, the vehicle detection method also includes: deleting face images and non-motor vehicle images corresponding to confidence scores less than preset scores; and storing confidence scores not less than preset scores in a set order.

[0008] Among them, the step of detecting the face image and its confidence, the non-motor vehicle image and its confidence, and the type of each non-motor vehicle in the image to be detected includes: detecting the face image frame and its confidence, the non-motor vehicle image frame and its confidence, and the type of each non-motor vehicle in the image to be detected; using the confidence of the face image and the confidence of the non-motor vehicle image to form at least one group of human-vehicle detection pairs with the face image frame and the non-motor vehicle image frame, including: using the confidence of the face image and the confidence of the non-motor vehicle image to form at least one group of human-vehicle detection frame pairs with the face image frame; comparing the face image of each group of human-vehicle detection pairs with the face images in other human-vehicle detection pairs, and comparing the faces with a higher overlap than the first The steps of determining facial images with a first threshold as the same passenger; and comparing each non-motor vehicle image with non-motor vehicle images of other human-vehicle detection pairs, determining non-motor vehicle images with a degree of overlap higher than a second threshold as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle, include: calculating the intersection-and-union ratio of the facial image frame of each group of human-vehicle detection pairs with the facial image frames in other human-vehicle detection pairs, and determining the passengers corresponding to the facial image frames with an intersection-and-union ratio higher than the first threshold as the same passenger; and, calculating the intersection-and-union ratio of each non-motor vehicle image frame with the non-motor vehicle image frames of other human-vehicle detection pairs, and determining the passengers corresponding to the facial image frames with an intersection-and-union ratio higher than the second threshold as the same non-motor vehicle.

[0009] The vehicle detection method further includes: determining whether the non-motor vehicle is overloaded according to the type of the non-motor vehicle and the corresponding number of passengers.

[0010] Among them, the steps of determining whether a non-motor vehicle is overloaded based on the type of non-motor vehicle and the corresponding number of passengers specifically include: determining the maximum number of passengers for the non-motor vehicle based on the type of non-motor vehicle; judging whether the number of passengers corresponding to the non-motor vehicle exceeds the maximum number of passengers; if so, determining that the non-motor vehicle is illegally carrying passengers.

[0011] Among them, the step of detecting the face image and its confidence, the non-motor vehicle image and its confidence, and the type of each non-motor vehicle in the image to be detected includes: detecting the image to be detected through a deep learning target detection model to obtain the face image and its confidence, the non-motor vehicle image corresponding to the face image and its confidence, and the type of non-motor vehicle.

[0012] Among them, before the step of detecting the face image and its confidence level, the non-motor vehicle image and its confidence level, and the type of each non-motor vehicle in the image to be detected, the step also includes: training a deep learning target detection model.

[0013] Among them, the steps of training the deep learning target detection model specifically include: obtaining a training sample set, the training sample set includes multiple image samples, and the image samples are images marked with real face image area frames, non-motor vehicle real area frames and real non-motor vehicle categories; inputting the image samples into the initial detection model for feature extraction to obtain a basic feature map; detecting the basic feature map to obtain a face image prediction frame and a non-motor vehicle prediction frame paired with the face image; mapping the paired face image prediction frames and non-motor vehicle prediction frames to the basic feature map to obtain paired face image feature maps and non-motor vehicle feature maps; obtaining paired face image prediction area frames, non-motor vehicle prediction area frames, and non-motor vehicle prediction categories through feature fusion of the paired face image feature maps and non-motor vehicle feature maps; constructing a loss function through the face image prediction area frame and the face image real area frame, the non-motor vehicle prediction area frame and the non-motor vehicle real area frame, and the non-motor vehicle prediction category and the non-motor vehicle real category; and iteratively training the initial detection model using the loss function to obtain a deep learning target detection model.

[0014] To solve the above technical problems, the second technical solution adopted by the present invention is: to provide a terminal, the terminal includes a memory, a processor and a computer program stored in the memory and running on the processor, and the processor is used to implement the steps in the above vehicle detection method.

[0015] In order to solve the above technical problems, the third technical solution adopted by the present invention is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the above vehicle detection method are implemented.

[0016] The beneficial effects of the present invention are as follows: different from the prior art, a vehicle detection method, a terminal and a computer-readable storage medium thereof are provided, the vehicle detection method comprising: acquiring an image to be detected; detecting a face image and its confidence level, a non-motor vehicle image and its confidence level and the type of each non-motor vehicle in the image to be detected; using the confidence level of the face image and the confidence level of the non-motor vehicle image to form at least one group of human-vehicle detection pairs from the face image and the non-motor vehicle image; comparing the face images of each group of human-vehicle detection pairs with the face images in other human-vehicle detection pairs, and determining the face images with a degree of overlap higher than a first threshold as the same passenger; and comparing each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, and determining the non-motor vehicle images with a degree of overlap higher than a second threshold as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle. The vehicle detection method of the present invention detects paired facial images and non-motor vehicles, and eliminates human-vehicle detection pairs with repeated facial images by comparing the facial images of each group of human-vehicle detection pairs with the facial images of other human-vehicle detection pairs; and screens out human-vehicle detection pairs with repeated non-motor vehicle images by comparing each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, thereby effectively improving the judgment result of the non-motor vehicle carrying situation and improving the detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 It is a flow chart of an embodiment of a vehicle detection method provided by the present invention;

[0018] Figure 2 It is a flow chart of a specific embodiment of the vehicle detection method provided by the present invention;

[0019] Figure 3 yes Figure 2 A schematic flow chart of a specific embodiment of step S21 in the provided vehicle detection method;

[0020] Figure 4 yes Figure 2 A schematic flow chart of a specific embodiment of step S29 in the vehicle detection method provided;

[0021] Figure 5 It is a schematic block diagram of an implementation method of a terminal of the present application;

[0022] Figure 6 It is a schematic block diagram of an implementation of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0023] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0024] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0025] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0026] In order to enable those skilled in the art to better understand the technical solution of the present invention, a target tracking method provided by the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0027] See also Figure 1 , Figure 1 It is a flow chart of an embodiment of the vehicle detection method provided by the present invention. For the convenience of description, the embodiment of the present application is described with a terminal as the execution subject. It should be understood that the execution subject of the vehicle detection method is not limited to the terminal, but can also be applied to devices with image processing functions such as terminal devices. The specific steps of the method are described as follows.

[0028] S11: Acquire the image to be detected.

[0029] Specifically, the acquired images to be detected at least include human face images and non-motor vehicle images.

[0030] S12: Detecting the face image and its confidence level, the non-motor vehicle image and its confidence level, and the type of each non-motor vehicle in the image to be detected.

[0031] Specifically, the face image frame and its confidence, the non-motor vehicle image frame and its confidence, and the type of each non-motor vehicle in the image to be detected are detected. In a specific embodiment, the image to be detected is detected by a deep learning target detection model to obtain a face image and its confidence, a non-motor vehicle image corresponding to the face image and its confidence, and the type of non-motor vehicle.

[0032] In an optional embodiment, the step of detecting the face image and its confidence, the non-motor vehicle image and its confidence, and the type of each non-motor vehicle in the image to be detected also includes training a deep learning target detection model. In a specific embodiment, a training sample set is obtained, the training sample set includes multiple image samples, and the image samples are images marked with a real area frame of a face image, a real area frame of a non-motor vehicle, and a real category of a non-motor vehicle; the image samples are input into the initial detection model for feature extraction to obtain a basic feature map; the basic feature map is detected to obtain a face image prediction frame and a non-motor vehicle prediction frame paired with the face image; the paired face image prediction frame and the non-motor vehicle prediction frame are mapped to the basic feature map to obtain a paired face image feature map and non-motor vehicle feature map; the paired face image feature map and the non-motor vehicle feature map are subjected to feature fusion to obtain a paired face image prediction area frame, a non-motor vehicle prediction area frame, and a non-motor vehicle prediction category; a loss function is constructed by the face image prediction area frame and the face image real area frame, the non-motor vehicle prediction area frame and the non-motor vehicle real area frame, and the non-motor vehicle prediction category and the non-motor vehicle real category; the initial detection model is iteratively trained using the loss function to obtain a deep learning target detection model.

[0033] S13: Using the confidence of the human face image and the confidence of the non-motor vehicle image, the human face image and the non-motor vehicle image are combined to form at least one group of human-vehicle detection pairs.

[0034] Specifically, the confidence of each face image is weighted with the confidence of each non-motor vehicle image in turn to obtain the confidence score of each group of face images and non-motor vehicle images; the face image and non-motor vehicle image whose confidence score is not less than the preset score are combined to determine the human-vehicle detection pair. In an optional embodiment, the face images and non-motor vehicle images corresponding to the confidence scores less than the preset scores are deleted; and the confidence scores not less than the preset scores are stored in a set order. In a specific embodiment, the face image frame and the non-motor vehicle image frame are used to form at least one group of human-vehicle detection frame pairs using the confidence of the face image and the confidence of the non-motor vehicle image.

[0035] S14: Compare the facial images of each group of person-vehicle detection pairs with the facial images of other person-vehicle detection pairs, and determine the facial images with a degree of overlap higher than a first threshold as the same passenger; and compare each non-motor vehicle image with the non-motor vehicle images of other person-vehicle detection pairs, and determine the non-motor vehicle images with a degree of overlap higher than a second threshold as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle.

[0036] Specifically, if the non-motor vehicle images in different person-vehicle detection pairs correspond to the same non-motor vehicle, and the passengers corresponding to the face images are different passengers, it is determined that different users are riding in the same non-motor vehicle.

[0037] In a specific embodiment, the intersection-and-union ratio of the face image frame of each group of person-vehicle detection pairs with the face image frames in other person-vehicle detection pairs is calculated respectively, and the passengers corresponding to the face image frames whose intersection-and-union ratio is higher than a first threshold are determined to be the same passenger; and the intersection-and-union ratio of each non-motor vehicle image frame with the non-motor vehicle image frames in other person-vehicle detection pairs is calculated respectively, and the passengers corresponding to the face image frames whose intersection-and-union ratio is higher than a second threshold are determined to be the same non-motor vehicle.

[0038] In an optional embodiment, whether the non-motor vehicle is overloaded is determined based on the type of the non-motor vehicle and the corresponding number of passengers. In a specific embodiment, the maximum number of passengers of the non-motor vehicle is determined based on the type of the non-motor vehicle; it is determined whether the number of passengers corresponding to the non-motor vehicle exceeds the maximum number of passengers; if it exceeds, it is determined that the non-motor vehicle is illegally carrying passengers.

[0039] The vehicle detection method provided in this embodiment includes: acquiring an image to be detected; detecting a face image and its confidence level, a non-motor vehicle image and its confidence level, and the type of each non-motor vehicle in the image to be detected; using the confidence level of the face image and the confidence level of the non-motor vehicle image to form at least one group of human-vehicle detection pairs from the face image and the non-motor vehicle image; comparing the face images of each group of human-vehicle detection pairs with the face images in other human-vehicle detection pairs, and determining the face images with a degree of overlap higher than a first threshold as the same passenger; and comparing each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, and determining the non-motor vehicle images with a degree of overlap higher than a second threshold as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle. The vehicle detection method of the present invention detects paired facial images and non-motor vehicles, and eliminates human-vehicle detection pairs with repeated facial images by comparing the facial images of each group of human-vehicle detection pairs with the facial images of other human-vehicle detection pairs; and screens out human-vehicle detection pairs with repeated non-motor vehicle images by comparing each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, thereby effectively improving the judgment result of the non-motor vehicle carrying situation and improving the detection accuracy.

[0040] See also Figure 2 , Figure 3 , Figure 4 , Figure 2 It is a flow chart of a specific embodiment of the vehicle detection method provided by the present invention; Figure 3 yes Figure 2 A schematic flow chart of a specific embodiment of step S21 in the provided vehicle detection method; Figure 4 yes Figure 2A flowchart of a specific embodiment of step S29 in the vehicle detection method is provided. For ease of description, the embodiment of the present application is described with the terminal as the execution subject. It should be understood that the execution subject of the vehicle detection method is not limited to the terminal, but can also be applied to devices with image processing functions such as terminal devices. The specific steps of the method are described as follows.

[0041] S21: Train a deep learning object detection model.

[0042] Specifically, see Figure 3 ,The steps of training a deep learning object detection model include the following sub-steps:

[0043] S211: Acquire a training sample set, where the training sample set includes a plurality of image samples, where the image samples are images marked with a real face image region frame, a real non-motor vehicle region frame, and a real category of a non-motor vehicle.

[0044] Specifically, a plurality of image samples in a traffic video are obtained to form a training sample set, and the image samples include at least a pair of face images and non-motor vehicle image samples. There is a coupling relationship between the face image and the non-motor vehicle, that is, the person corresponding to the face image drives a non-motor vehicle or rides a non-motor vehicle. The face image in each frame of the image sample and the non-motor vehicle that has a coupling relationship with the face image are labeled to obtain a real area frame of the face image, a real area frame of the non-motor vehicle, and a real category of the non-motor vehicle. In another optional embodiment, the real category of the face image can also be labeled.

[0045] S212: Input the image sample into the initial detection model to extract features and obtain a basic feature map.

[0046] Specifically, the unlabeled image samples are input into the initial detection model, and the feature extraction network is used to extract features from the input image samples to obtain the corresponding basic feature map.

[0047] S213: Detect the basic feature map to obtain a facial image prediction frame and a non-motor vehicle prediction frame paired with the facial image.

[0048] Specifically, the basic feature network obtained in S212 is further subjected to feature map extraction using the dual-target positioning network, and the position of the face image in the feature map and the position of the non-motor vehicle coupled to the face image are predicted to obtain a face image prediction frame and a non-motor vehicle prediction frame, and there is a coupling relationship between the face image prediction frame and the non-motor vehicle prediction frame. In a specific embodiment, the basic feature map is subjected to feature map extraction using the Double Anchor RPN (Double Anchor Region Proposal Network) to obtain a pair of face image prediction frames and non-motor vehicle prediction frames.

[0049] S214: Mapping the paired face image prediction frames and the non-motor vehicle prediction frames onto the basic feature map to obtain paired face image feature maps and non-motor vehicle feature maps.

[0050] Specifically, the face image prediction frame and the non-motor vehicle prediction frame having a coupling relationship are regressed to the basic feature map to obtain the corresponding face image feature map and the non-motor vehicle feature map. In a specific embodiment, the paired face detection frame and the non-motor vehicle prediction frame are mapped to the basic feature map, and the face feature map and the non-motor vehicle feature map paired with the face feature map are obtained by ROI (Region Of Interest) extraction. Among them, the ROI extraction method includes at least one of ROI Pooling (region of interest pooling), ROI Align (region of interest alignment), and ROI Warp (region of interest deformation).

[0051] When there are multiple pairs of face feature maps and non-motor vehicle feature maps, all face feature maps have the same size, and all non-motor vehicle feature maps have the same size. For example, among all pairs of non-motor vehicle feature maps and face feature maps, the size of all face feature maps is 3*3, and the size of all non-motor vehicle feature maps is 5*5.

[0052] S215: The paired facial image feature maps and non-motor vehicle feature maps are subjected to feature fusion to obtain paired facial image prediction region frames, non-motor vehicle prediction region frames, and non-motor vehicle prediction categories.

[0053] Specifically, the paired face feature maps and non-motor vehicle feature maps are feature fused to obtain a face prediction region frame, a non-motor vehicle prediction region frame, and a prediction category of the non-motor vehicle; the confidence of the face prediction region frame and the confidence of the non-motor vehicle prediction region frame are also detected. In an optional embodiment, the prediction category of the face can also be obtained after feature fusion. Specifically, the prediction category of the non-motor vehicle can be a motorcycle, a tricycle, a bicycle, an electric bicycle, etc. The category of the face can be a man, a woman, an adult, a child, a person wearing a helmet, a person not wearing a helmet, etc.

[0054] S216: Constructing a loss function through the predicted face image region frame and the real face image region frame, the predicted non-motor vehicle region frame and the real non-motor vehicle region frame, and the predicted non-motor vehicle category and the real non-motor vehicle category.

[0055] Specifically, the Smooth L1 regression loss function is used to calculate the error value between the predicted region frame of the face image and the real region frame of the face image; the cross entropy loss function is used to calculate the error value between the predicted category of the face image and the real category of the labeled face image. The Smooth L1 regression loss function is used to calculate the error value between the predicted region frame of the non-motor vehicle and the real region frame of the non-motor vehicle; the cross entropy loss function is used to calculate the error value between the predicted category of the non-motor vehicle and the real category of the non-motor vehicle. In another optional embodiment, the loss function is constructed by the predicted region frame of the face image and the real region frame of the face image, the predicted region frame of the non-motor vehicle and the real region frame of the non-motor vehicle, and the predicted category of the non-motor vehicle and the real category of the non-motor vehicle. The loss function is at least composed of the sum of the loss function of the face image region frame, the loss function of the non-motor vehicle region frame, the face image category loss function and the non-motor vehicle category loss function.

[0056] In a specific embodiment, the loss function composed of the sum of the loss function of the face image area frame, the loss function of the non-motor vehicle image area frame, the face image category loss function and the non-motor vehicle category loss function is shown in the following formula (1).

[0057] L RPN =L cls +L reg H +L reg V (1)

[0058] Where: L cls The loss branch for face and non-motor vehicle classification; L reg H is the face box loss branch; L reg V is the loss branch of the non-motor vehicle frame; L RPN is the predicted regional loss value.

[0059] S217: Use the loss function to iteratively train the initial detection model to obtain a deep learning target detection model.

[0060] Specifically, the initial detection model is iteratively trained by obtaining the error value between the predicted area box of the face image and the real area box of the face image, the error value between the predicted area box of the non-motor vehicle and the real area box of the non-motor vehicle, the error value between the predicted category of the non-motor vehicle and the real category of the non-motor vehicle, and the error value between the predicted category of the face image and the real category of the face image to obtain a deep learning target detection model.

[0061] In an optional embodiment, the initial detection model is reversely conducted according to the result of the above-mentioned loss function, and the weight of the initial detection model is corrected according to the loss value fed back by the loss function. In an optional embodiment, the parameters in the Double Anchor RPN in the initial detection model can also be corrected to realize the training of the initial detection model.

[0062] The image samples are input into the initial detection model, and the initial detection model detects the face images and non-motor vehicles in the image samples. When the accuracy of the detection results of the initial detection model reaches the preset accuracy threshold, which can be set by yourself, such as 99%, 95%, etc., the training of the initial detection model is stopped and the deep learning target detection model is obtained.

[0063] S22: Acquire the image to be detected.

[0064] Specifically, the image to be detected can be obtained by a snapshot camera on the road, or a video frame provided by a surveillance camera in traffic scenes such as electric police, checkpoints, entrances and exits, etc. The image to be detected can also be a video frame or image sample obtained through other channels.

[0065] S23: Detect the image to be detected by the deep learning target detection model to obtain a face image and its confidence, a non-motor vehicle image corresponding to the face image and its confidence, and the type of the non-motor vehicle.

[0066] Specifically, the image sample or video frame obtained in S22 is input into the deep learning target detection model to obtain the face image region frame and its confidence in the face image region frame, the non-motor vehicle region frame paired with the face image region frame and its confidence in the non-motor vehicle region frame. The type of non-motor vehicle can also be obtained for use in subsequent steps. The face image and the non-motor vehicle image with a coupling relationship constitute a person-vehicle detection frame pair.

[0067] In a specific embodiment, a set R={} is pre-established, and a non-motor vehicle detection frame set B V ={}, face detection frame set B H ={}, non-motor vehicle detection box confidence set S V = {}, face detection frame set S H={}. All face detection frames in the human-vehicle detection frame pairs detected by the deep learning target detection model are stored in set B H In, get B H ={b1 H ,···,b N H}, all non-motor vehicle detection frames paired with the above face detection frames are stored in set B in sequence V In, get B V ={b1 V ,···,b N V}, all non-motor vehicle detection box confidences are stored in the set S V In the V ={s1 V ,···,s N V}, all face detection frame confidences are stored in the set S H In the H ={s1 H ,···,s N H}. Among them, the non-motor vehicle detection frame b1 V and face detection frame b1 H There is a natural coupling relationship, s1 V b1 is the non-motor vehicle detection frame V The confidence level, s1 H b1 is the non-motor vehicle detection frame H confidence level.

[0068] S24: Using the confidence of the face image and the confidence of the non-motor vehicle image, the face image frame and the non-motor vehicle image frame are combined to form at least one set of human-vehicle detection frame pairs.

[0069] Specifically, the confidence of each face image and the confidence of the non-motor vehicle image paired with the face image are weighted to obtain the confidence of each group of face images and non-motor vehicle images. The face image and non-motor vehicle image whose confidence scores are not less than the preset score are combined to determine the human-vehicle detection pair. In another optional embodiment, the face images and non-motor vehicle images corresponding to the confidence scores less than the preset scores are deleted; and the confidence scores not less than the preset scores are stored in a set order.

[0070] In a specific embodiment, the parameters used in the matching process are first preset, including the score weight λ of the face detection frame, the threshold Ω of the non-motor vehicle detection frame, and the score weight λ of the face detection frame. V , face detection box threshold Ω H , confidence score threshold Ω SAmong them, the above parameters can be adjusted according to different scenarios and actual needs.

[0071] The final confidence score of each person-vehicle detection pair among all person-vehicle detection pairs is calculated using formula (2).

[0072] s i =λs i H +(1-λ)s i V (2)

[0073] Where: s i The final confidence score of the human-vehicle detection pair; s i H is the confidence of the face detection frame; s i V is the confidence of the non-motor vehicle detection frame; λ is the score weight of the face detection frame.

[0074] If the final confidence score of the human-vehicle detection pair is s i Less than the confidence score threshold Ω S , then from the non-motor vehicle detection frame set B V Eliminate non-motor vehicle detection box b i V , the face detection frame set B H The face detection frame b paired with the non-motor vehicle i H Remove; if the final confidence score of the human-vehicle detection pair is s i Not less than the confidence score threshold Ω S , the final confidence score s i Put them into the final confidence score set S and sort them from large to small. Through this step, the face detection frame and non-motor vehicle detection frame with low confidence in the human-vehicle detection pair are removed to reduce the workload for subsequent steps.

[0075] S25: Determine face detection frame set B H Is it an empty set?

[0076] Specifically, it is determined whether the final confidence scores of all vehicle-vehicle detection pairs obtained by the deep learning target detection model are less than the confidence score threshold Ω S If the final confidence scores of all detected vehicle-car pairs are less than the confidence score threshold Ω S , then all vehicle detection pairs need to be removed, and the face detection frame set B H is an empty set; if the final confidence scores of some of the detected human-vehicle detection pairs are less than the confidence score threshold Ω S , then the part of the human-vehicle detection pairs is eliminated, and the face detection frame set BH Some face detection frames are retained in the face detection frame set B H Not an empty set.

[0077] If the face detection box set B H If the face detection frame set B is not an empty set, the process goes directly to step S26; if the face detection frame set B is H If it is an empty set, go directly to step S28.

[0078] S26: Calculate the intersection-and-union ratios of the face image frames of each group of vehicle-person detection pairs with the face image frames of other vehicle-person detection pairs, and determine the passengers corresponding to the face image frames with an intersection-and-union ratio higher than a first threshold as the same passenger.

[0079] Specifically, the face detection frame set B H If it is not an empty set, the final confidence score s of the person-vehicle detection pair with the highest score in the final confidence score set S is obtained. T Corresponding face detection frame-non-motor vehicle detection frame (b T V , b T H ), (b T V , b T H ) into set R, and from set B V Remove the detection box b T V , from set B H Remove b T H , remove s from the set S T . Calculate b separately T H With set B H Each other face detection frame b contained in i H The intersection-over-union (IOU) H , IOU H and face detection box threshold Ω H Make a comparison.

[0080] If IOU H Not less than the face detection frame threshold Ω H , indicating that the intersection and union ratio IOU H The corresponding face detection frame and the final confidence score s T The corresponding face detection frames detect the same person. If there are repeated face detection frames, the intersection and union (IOU) H The corresponding face detection frame b i H From set B HRemove the face detection frame b i H Paired b i V From set B V Then, set B H Other face detection frames in b T H The intersection-over-union ratio and the face detection box threshold Ω H Make a comparison.

[0081] If IOU H Less than the face detection frame threshold Ω H , it means that the intersection and union ratio IOU H The corresponding face detection frame b i H and the final confidence score s T The corresponding face detection frame b T H If it indicates a different face, the process directly jumps to step S27.

[0082] S27: Calculate the intersection-and-union ratios of each non-motor vehicle image frame and the non-motor vehicle image frames of other human-vehicle detection pairs, and determine that the passengers corresponding to the face image frames whose intersection-and-union ratios are higher than a second threshold are the same non-motor vehicle.

[0083] Specifically, if we calculate b T H With set B H A face detection frame b contained in i H The intersection-over-union (IOU) H Less than the face detection frame threshold Ω H , it means that the intersection and union ratio IOU H The corresponding face detection frame and the final confidence score s T The corresponding face detection frames indicate different faces. Then continue to calculate b T V With set B V and face detection frame b i H Paired non-motor vehicle detection boxes b i V The intersection-over-union (IOU) V , IOU V and face detection box threshold Ω V The second threshold is the face detection frame threshold Ω V .

[0084] If IOU V Not less than the non-motor vehicle detection frame threshold Ω V, indicating that the intersection and union ratio IOU V The corresponding non-motor vehicle detection frame b i V and the final confidence score s T The corresponding non-motor vehicle detection frame b T V The same non-motor vehicle is detected, and there is a non-motor vehicle detection frame b i V and the final confidence score s T The corresponding non-motor vehicle detection frame b T V In the same situation, that is, there are two face detection frames corresponding to one non-motor vehicle detection frame, the intersection and union (IOU) V The corresponding face detection frame b i H From set B H Remove the face detection frame b i H Paired b i V From set B V Establish a new mapping relationship (b T V , b T H , b i H ), put the new mapping relationship into set R, and then put set B V Other face detection frames in b T H The intersection-over-union ratio and the face detection box threshold Ω H Make a comparison.

[0085] If IOU V Less than the non-motor vehicle detection box threshold Ω V , indicating that the intersection and union ratio IOU V The corresponding non-motor vehicle detection frame b i V and the final confidence score s T The corresponding non-motor vehicle detection frame b T V The corresponding non-motor vehicles are different. Then set B V Other face detection frames b in i H With b T H The intersection-over-union ratio and the face detection box threshold Ω H Make a comparison.

[0086] Waiting for collection B H All face detection frames contained in set B are removed. VAll non-motor vehicle detection frames contained in are also removed. That is, set B H and set B V All are empty sets. Then jump directly to S25.

[0087] S28: Output set R.

[0088] Specifically, if the set B H If it is an empty set, it means that all the human-vehicle detection pairs have been detected one by one, and the non-motor vehicle detection frames in the set R and all the face detection frames matching the non-motor vehicle are output. In a specific embodiment, R = {(b T V , b T H , b i H )}.

[0089] S29: Determine whether the non-motor vehicle is overloaded according to the type of the non-motor vehicle and the corresponding number of passengers.

[0090] Specifically, see Figure 4 Determining whether a non-motor vehicle is overloaded according to the type of the non-motor vehicle and the corresponding number of passengers specifically includes the following sub-steps:

[0091] S291: Determine the maximum number of passengers on the non-motor vehicle according to the type of the non-motor vehicle.

[0092] Specifically, the type of non-motor vehicle in the image to be detected is detected by the deep learning target detection model. That is, (b T V , b T H , b i H ) T V The maximum number of passengers for different types of non-motor vehicles is different. For example, the maximum number of passengers for a motorcycle is 2.

[0093] S292: Determine whether the number of passengers corresponding to the non-motor vehicle exceeds the maximum number of passengers.

[0094] Specifically, determine (b T V , b T H , b i H ) Non-motor vehicle detection frame b T V There are two corresponding face detection frames in b T H and b i H, indicating that the number of passengers detected on the non-motor vehicle is 2. The number of passengers detected on the non-motor vehicle is compared with the maximum number of passengers of the non-motor vehicle of this type.

[0095] If the detected number of passengers of the non-motor vehicle is greater than the maximum number of passengers of this type of non-motor vehicle, jump directly to step S293; if the detected number of passengers of the non-motor vehicle is not greater than the maximum number of passengers of this type of non-motor vehicle, jump directly to step S294.

[0096] S293: Determine that a non-motor vehicle is illegally carrying passengers.

[0097] Specifically, when the number of passengers detected on the non-motor vehicle is greater than the maximum number of passengers of the non-motor vehicle of this type, the non-motor vehicle is deemed to be illegally carrying passengers. When the non-motor vehicle is determined to be illegally carrying passengers, the result is reported and a warning report is output, the output method includes but is not limited to outputting a label box and text description on the original image or intercepting abnormal non-motor vehicles, face images, etc.

[0098] S294: Determine whether a non-motor vehicle can legally carry passengers.

[0099] Specifically, when the detected number of passengers of the non-motor vehicle is not greater than the maximum number of passengers of the non-motor vehicle of this type, the non-motor vehicle is legally carrying passengers.

[0100] The vehicle detection method provided in this embodiment includes: acquiring an image to be detected; detecting a face image and its confidence level, a non-motor vehicle image and its confidence level, and the type of each non-motor vehicle in the image to be detected; using the confidence level of the face image and the confidence level of the non-motor vehicle image to form at least one group of human-vehicle detection pairs from the face image and the non-motor vehicle image; comparing the face images of each group of human-vehicle detection pairs with the face images in other human-vehicle detection pairs, and determining the face images with a degree of overlap higher than a first threshold as the same passenger; and comparing each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, and determining the non-motor vehicle images with a degree of overlap higher than a second threshold as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle. The vehicle detection method of the present invention detects paired facial images and non-motor vehicles, and eliminates human-vehicle detection pairs with repeated facial images by comparing the facial images of each group of human-vehicle detection pairs with the facial images of other human-vehicle detection pairs; and screens out human-vehicle detection pairs with repeated non-motor vehicle images by comparing each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, thereby effectively improving the judgment result of the non-motor vehicle carrying situation and improving the detection accuracy.

[0101] See also Figure 5 , Figure 5 This is a schematic block diagram of an implementation method of a terminal of the present application. Figure 5 As shown, the terminal 70 of this embodiment includes: a processor 71, a memory 72, and a computer program stored in the memory 72 and executable on the processor 71. When the computer program is executed by the processor 71, the above-mentioned vehicle detection method is implemented. To avoid repetition, it is not described one by one here.

[0102] See also Figure 6 , Figure 6 It is a schematic block diagram of an implementation of a computer-readable storage medium of the present application.

[0103] A computer-readable storage medium 80 is also provided in an embodiment of the present application. The computer-readable storage medium 80 stores a computer program 801. The computer program 801 includes program instructions. The processor executes the program instructions to implement any vehicle detection method provided in the embodiment of the present application.

[0104] The computer-readable storage medium 80 may be an internal storage unit of the computer device of the aforementioned embodiment, such as a hard disk or memory of the computer device. The computer-readable storage medium 80 may also be an external storage device of the computer device, such as a plug-in hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the computer device.

[0105] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A vehicle detection method, characterized in that: The vehicle detection method comprises: Obtain the image to be detected; Detecting a human face image and its confidence level, a non-motor vehicle image and its confidence level, and the type of each non-motor vehicle in the image to be detected; Using the confidence of the human face image and the confidence of the non-motor vehicle image, the human face image and the non-motor vehicle image form at least one group of human-vehicle detection pairs; Compare the facial images of each group of the human-vehicle detection pairs with the facial images of other human-vehicle detection pairs, and determine the facial images with a degree of overlap higher than a first threshold as the same passenger; and compare each non-motor vehicle image with the non-motor vehicle images of other human-vehicle detection pairs, and determine the non-motor vehicle images with a degree of overlap higher than a second threshold as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle; If the non-motor vehicle corresponding to the non-motor vehicle images in different vehicle-person detection pairs is the same non-motor vehicle, and the passengers corresponding to the face images are different passengers, it is determined that the different users are riding in the same non-motor vehicle; The step of using the confidence of the face image and the confidence of the non-motor vehicle image to form at least one person-vehicle detection pair with the face image and the non-motor vehicle image comprises: Performing weighted processing on the confidence scores of each of the face images and the non-motor vehicle images in turn to obtain confidence scores of each group of the face images and the non-motor vehicle images; The face image whose confidence score is not less than a preset score is combined with the non-motor vehicle image to be determined as the human-vehicle detection pair.

2. The vehicle detection method according to claim 1, characterized in that: The vehicle detection method further comprises: The face images and the non-motor vehicle images corresponding to the confidence scores less than the preset scores are deleted; and the confidence scores not less than the preset scores are stored in a set order.

3. The vehicle detection method according to claim 1, characterized in that: The step of detecting the face image and its confidence level, the non-motor vehicle image and its confidence level, and the type of each non-motor vehicle in the image to be detected includes: The detecting of the face image frame and its confidence level, the non-motor vehicle image frame and its confidence level, and the type of each non-motor vehicle in the image to be detected; The step of using the confidence of the human face image and the confidence of the non-motor vehicle image to form at least one human-vehicle detection pair with the human face image and the non-motor vehicle image includes: The confidence level of the human face image and the confidence level of the non-motor vehicle image are used to form at least one human-vehicle detection frame pair with the human face image frame and the non-motor vehicle image frame; The steps of comparing the facial images of each group of the vehicle-person detection pairs with the facial images of other vehicle-person detection pairs, and determining the facial images with a degree of overlap higher than a first threshold as the same passenger; and comparing each non-motor vehicle image with the non-motor vehicle images of other vehicle-person detection pairs, and determining the non-motor vehicle images with a degree of overlap higher than a second threshold as the same non-motor vehicle, so as to determine the number of passengers corresponding to each non-motor vehicle, include: The intersection-and-union ratios of the facial image frames of each group of the human-vehicle detection pairs and the facial image frames in other human-vehicle detection pairs are calculated respectively, and the passengers corresponding to the facial image frames whose intersection-and-union ratios are higher than the first threshold are determined to be the same passenger; and the intersection-and-union ratios of each non-motor vehicle image frame and the non-motor vehicle image frames in other human-vehicle detection pairs are calculated respectively, and the passengers corresponding to the facial image frames whose intersection-and-union ratios are higher than the second threshold are determined to be the same non-motor vehicle.

4. The vehicle detection method according to any one of claims 1 to 3, characterized in that: The vehicle detection method further comprises: Whether the non-motor vehicle is overloaded is determined according to the type of the non-motor vehicle and the corresponding number of passengers.

5. The vehicle detection method according to claim 4, characterized in that: The step of determining whether the non-motor vehicle is overloaded according to the type of the non-motor vehicle and the corresponding number of passengers specifically includes: Determining the maximum number of passengers on the non-motor vehicle according to the type of the non-motor vehicle; Determining whether the number of passengers corresponding to the non-motor vehicle exceeds the maximum number of passengers; If it exceeds, the non-motor vehicle is determined to be illegally carrying passengers.

6. The vehicle detection method according to any one of claims 1 to 3, characterized in that: The step of detecting the face image and its confidence level, the non-motor vehicle image and its confidence level, and the type of each non-motor vehicle in the image to be detected includes: The image to be detected is detected by a deep learning target detection model to obtain the face image and its confidence, the non-motor vehicle image corresponding to the face image and its confidence, and the type of the non-motor vehicle.

7. The vehicle detection method according to claim 6, characterized in that: Before the step of detecting the face image and its confidence level, the non-motor vehicle image and its confidence level, and the type of each non-motor vehicle in the image to be detected, the method further includes: Train the deep learning object detection model.

8. The vehicle detection method according to claim 7, characterized in that: The step of training the deep learning target detection model specifically includes: Acquire a training sample set, wherein the training sample set includes a plurality of image samples, wherein the image samples are images marked with a real face image region frame, a real non-motor vehicle region frame, and a real category of a non-motor vehicle; Inputting the image sample into an initial detection model to extract features and obtain a basic feature map; Detecting the basic feature map to obtain a face image prediction frame and the non-motor vehicle prediction frame paired with the face image; Mapping the paired face image prediction frame and the non-motor vehicle prediction frame onto the basic feature map to obtain a paired face image feature map and a non-motor vehicle feature map; The paired facial image feature maps and the paired non-motor vehicle feature maps are subjected to feature fusion to obtain paired facial image prediction region frames, non-motor vehicle prediction region frames, and non-motor vehicle prediction categories; Constructing a loss function by using the predicted face image region frame and the real face image region frame, the predicted non-motor vehicle region frame and the real non-motor vehicle region frame, and the predicted non-motor vehicle category and the real non-motor vehicle category; The initial detection model is iteratively trained using the loss function to obtain the deep learning target detection model.

9. A terminal comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: The processor is used to implement the steps in the vehicle detection method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the vehicle detection method according to any one of claims 1 to 8 are implemented.

Images