Pedestrian-vehicle association method and retrieval matching method based on video monitoring

Abstract

The invention discloses a human-vehicle association method based on video monitoring. The method comprises the following steps: detecting and tracking vehicles in a monitoring area based on a first camera on one side of a lane, a second camera on the other side of the lane and a third camera in the coming direction of a traffic flow; in vehicle tracking of the first camera and the second camera, detecting a door opening area # imgabs0 # belonging to the same vehicle target j in each frame of image i, identifying a human body target PersonArea appearing in the door opening area # imgabs1 #, judging whether the human body target has a get-off behavior or not, if so, marking a vehicle corresponding to the door opening area # imgabs2 # to which the human body target belongs as Carjopen, and if not, marking the vehicle corresponding to the door opening area # imgabs2 # to which the human body target belongs as Carjopen; detecting and tracking the human body target to obtain a face image with the best quality; performing license plate positioning identification on each vehicle target frame in the tracking process of the third camera; and carrying out proximity matching to obtain license plate information of the target Carjopen of the vehicle, thereby realizing human-vehicle binding. The invention further provides a human-vehicle retrieval matching method.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and more specifically, to a person-vehicle association method and a retrieval and matching method based on video surveillance. Background Art

[0002] At the customs inspection site, vehicles and passengers need to be inspected separately. The driver drives the vehicle into the inspection lane and is inspected by the lane information collection equipment on the lane. Other passengers except the driver need to get off at the inspection site and enter the inspection hall for inspection. However, when these other passengers who get off the vehicle encounter abnormalities during the inspection process in the inspection hall, it is necessary to deal with the passengers with abnormal inspections and their vehicles. Among them, the inspection abnormalities may include but are not limited to facial recognition failures, inconsistencies with the pre-declared information during the inspection process, or personnel designated by the inspection unit that need attention. However, at this time, only abnormal personnel enter the inspection hall, and the vehicle and abnormal personnel have been separated. It is difficult to know which vehicle the abnormal personnel in the inspection hall comes from. Therefore, it is necessary to monitor the vehicle when it enters the customs clearance site, which vehicles have which passengers, that is, to achieve human-vehicle association, so that it is necessary to associate with the relevant vehicles when the abnormality needs to be checked.

[0003] In some existing methods for associating people with cars, cameras in the inspection site capture the trajectories of people and cars, and judge whether a person and a car are associated based on the number of times a person and a car appear in the same camera and the number of places where they appear together in the same place. The more times a person appears in the same camera and the more places where they appear together in the same place, the greater the probability of association. This method associates people with cars by counting the number of times people and cars appear together and the number of places. However, there are many people and cars in the inspection site, and people getting off a car will also appear with other vehicles. This method cannot perform accurate association. Summary of the invention

[0004] In view of the prior art, the technical problem solved by the present application is to provide a person-vehicle association method and a retrieval and matching method based on video surveillance that improve the association accuracy.

[0005] In order to solve the above technical problems, in a first aspect, the present application provides a method for associating a person with a vehicle based on video surveillance, which comprises:

[0006] Performing vehicle detection and tracking based on a first video image captured by a first camera on one side of the lane and a second video image captured by a second camera on the other side of the lane;

[0007] In the vehicle detection and tracking process, the door opening area in each frame image i belonging to the same vehicle target j is detected. ;

[0008] Identify the door opening area The human target Person Area , determine the human target Person Area Whether there is a get-off behavior, if it occurs, the human target Person Area Door opening area The corresponding vehicle is marked as the open door vehicle Car jopen , and then for the human target Person Area Perform detection and tracking to obtain the best quality face image Face best ;

[0009] Perform vehicle detection and tracking based on a third video image captured by a third camera located in the direction of the vehicle flow, and perform license plate positioning and recognition on each vehicle target frame obtained by vehicle detection and tracking during the tracking process;

[0010] Get information about the vehicle target Car jopen The coordinates of the center point of the target frame at a time t are O 1 , obtain the center point coordinates O of the target frames of all vehicles in a frame image F corresponding to the third video image at time t m ;

[0011] The target box center point coordinate O 1 And all the center point coordinates O m Convert to the same reference coordinate system on the lane for proximity matching to obtain the vehicle target Car jopen license plate information; and

[0012] The face image Face best 、Vehicle target Car jopen and vehicle target Car jopen The license plate information is bound and stored.

[0013] In one possible implementation, the recognition occurs in the door opening area. The human target Person Area ,include:

[0014] Training a binary classification recognition model: Collect training data sets, which include images containing human targets in the door opening area and images not containing human targets in the door opening area. Human targets include human targets sitting in the car, bending over to get out of the car, standing upright, and walking. Input the data into the EfficientNet-B2 binary classification model for training to obtain a binary classification recognition model. The recognition results of the binary classification recognition model include true and false. A true result indicates that the door opening area contains a human body, and a false result indicates that the door opening area does not contain a human body.

[0015] Determine the door opening area Whether there is a human body: each frame of the video image V is sequentially divided into the door opening area in each frame according to the order of appearance. Input to the binary classification recognition model; when the door is opened at the kth frame When the recognition result of the input to the binary classification recognition model appears to be true for the first time, it indicates the door opening area The target person appears inside Area ;

[0016] Use the human body recognition model to identify the door opening area of ​​the kth frame Perform human target detection in the selected area to obtain the human target Person Area ;

[0017] Wherein, 1≤k≤i, the video image V represents the first video image or the second video image.

[0018] In one possible implementation, the target Person is determined Area Whether there is any disembarkation behavior, including:

[0019] Starting from the kth frame, obtain the door opening area of ​​each vehicle target j And the double door area The human target Person Area Tracking, obtaining human target Person Area The human target box in each frame;

[0020] When the following conditions are met, it indicates that the human target Person Area Get off at frame L+1:

[0021] In the adjacent L-th frame image and L+1-th frame image: and ;

[0022] Where L represents any frame image starting from the kth frame in the video image V, L ≥ k, Indicates the door opening area of ​​vehicle target j in the Lth frame image A collection of pixels, Indicates the door opening area of ​​vehicle target j in the L+1th frame image A set of pixels; is the human target Person in the Lth frame image Area The pixel point set of the human target box, is the human target Person in the L+1th frame image Area The pixel point set of the human target box.

[0023] In one possible implementation, for a human target PersonArea Perform detection and tracking to obtain the best quality face image Face best ,include:

[0024] When the human target Person Area After getting off the bus, obtain the human target Person in the next N frames of images Area The detection box set Box, set Box = {box n ,n=1,…N}, where box n Represented as the human target Person in any frame n of N frames during the target tracking process Area Detection box of

[0025] Detection box for each human target n Perform face detection in the selected area and obtain each detection box n All face areas in the area Face boxn ;

[0026] Remove all face areas Face boxn Person that is not a human target Area The faces of pedestrians in the surroundings are interfered with, and the faces of human targets in the subsequent N frames of images are obtained. Are A set of face regions;

[0027] Based on the clarity, brightness value and face deflection angle, the best quality face region image in the face region set is selected as the face image Face best .

[0028] In one possible implementation, all face regions Face are removed boxn Person that is not a human target Area The surrounding interfering pedestrians’ faces are:

[0029] The human target Person in any frame n of the subsequent N frames Area The detection frame is evenly divided into an upper half area and a lower half area in the vertical axis direction;

[0030] Take the face region face as all face regions Face boxn One of the face regions;

[0031] If the face region face is located in the lower half, the face region face is the face of the interfering pedestrian to be removed;

[0032] If the face region face is located in the upper half area and the distance between the center point of the face region face and the center point of the upper half area is greater than a set threshold, the face region face is the face of an interfering pedestrian that needs to be removed.

[0033] In a possible implementation, the target frame center point coordinate O 1 And all the center point coordinates O m Convert to the same reference coordinate system on the lane for proximity matching to obtain the vehicle target Car open License plate information, including:

[0034] Based on the reference objects set in the lane, the image coordinate systems of all cameras are converted to a physical reference coordinate system on the lane;

[0035] The center point coordinates O 1 Convert to the physical coordinate value W in the physical reference coordinate system 1 ;

[0036] The coordinates of each center point O m Convert to the physical coordinate value W in the physical reference coordinate system m ;

[0037] The physical coordinate value W 1 With each physical coordinate value W m Calculate Euclidean distance;

[0038] Sort all Euclidean distances and take the physical coordinate value W corresponding to the minimum Euclidean distance m The vehicle target is recorded as vehicle target Car min , with vehicle target Car min The license plate information is the vehicle target Car open license plate.

[0039] In a possible implementation, the image coordinate systems of all cameras are aligned to a physical reference coordinate system on the lane based on reference objects set in the lane, including:

[0040] The first reference point set in the lane is taken as the origin, and the traffic direction of the lane is taken as the vertical axis Y. 1 And the vertical and longitudinal axis direction is the horizontal axis X 1 Experience the physical reference coordinate system;

[0041] Measure the coordinate values ​​P of multiple second reference points set on the lane in the physical reference coordinate system C ;

[0042] Get the coordinate value P of each second reference point in the image field of view of the first camera 1, obtain the coordinate value P of each second reference point in the image field of view of the second camera 2 , obtain the coordinate value P of each second reference point in the image field of view of the third camera 4 ;

[0043] According to the coordinate values ​​P corresponding to at least four second reference points C and the coordinate values ​​P1 corresponding to the at least four second reference points, solving the first homography transformation matrix H describing the mapping relationship between the image coordinate system of the first camera and the physical reference coordinate system 1 ;

[0044] According to the coordinate values ​​P corresponding to at least four second reference points C and the coordinate values ​​P corresponding to the at least four second reference points 2 , solve the second homography transformation matrix H describing the mapping relationship between the image coordinate system of the second camera and the physical reference coordinate system 2 ;

[0045] According to the coordinate values ​​P corresponding to at least four second reference points C and the coordinate values ​​P corresponding to the at least four second reference points 3 , solve the second homography transformation matrix H describing the mapping relationship between the image coordinate system of the third camera and the physical reference coordinate system 3 .

[0046] In a second aspect, the present application provides a person-vehicle matching retrieval method, which includes:

[0047] Get the personnel P who have abnormal inspection during the inspection process curious face image face';

[0048] The face image face' is combined with the human-vehicle association method based on video surveillance to obtain all face images Face best to make a comparison; and,

[0049] When the match is successful, person P curious The vehicle for getting off is the vehicle obtained by the method for associating a person with a vehicle based on video surveillance and the face image Face best Bound vehicle.

[0050] The beneficial effects of the person-vehicle association method and retrieval matching method based on video surveillance provided by the present application are as follows: the person-vehicle association method detects the door opening of the passing vehicles on both sides of the lane through the first camera and the second camera, obtains the door opening area of ​​the door-opening vehicle, and then detects whether the pedestrian target Person appears in the door opening area. Area , then for the pedestrian target Person Area Tracking to determine whether the pedestrian has gotten off the bus.Area Get the best quality face image Face best ; Since the third camera is located on the side of the traffic flow, it can monitor the license plate information of the vehicles in the monitoring area, so the first camera and the second camera will obtain the vehicle with the door open and the person getting off the car. jopen Proximity matching with the vehicle in the third camera to obtain the vehicle Car jopen The license plate information of the pedestrian getting off the bus is thus realized. Area and pedestrian target Person Area Correlation matching of license plate information of the vehicle in which you are riding. The vehicle-person association method of the present application first detects the door opening area of ​​the vehicle and the pedestrian targets in the door opening area. By focusing only on the pedestrian targets in the door opening area, compared with the prior art vehicle-person association method based on the number and quantity of passing persons and vehicles, the interference of other pedestrian targets outside the vehicle can be eliminated, thereby improving the accuracy of vehicle-person association. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0052] Figure 1 A schematic diagram of the camera deployment and physical reference coordinate system within the monitoring area of ​​the inspection site according to an embodiment of the present application.

[0053] Figure 2 This is a flowchart of the steps of the human-vehicle association method based on video surveillance in an embodiment of the present application.

[0054] Figure 3 A schematic diagram of a door opening area of ​​a vehicle according to an embodiment of the present application. DETAILED DESCRIPTION

[0055] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0056] It should be noted that when an element is referred to as being "fixed to" or "disposed on" another element, it can be directly on the other element or indirectly on the other element. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element.

[0057] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the drawings, and are only for the convenience of describing the present application and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore should not be understood as a limitation on the present application.

[0058] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0059] The human-vehicle association method and retrieval and matching method based on video surveillance of the present application are now described in detail with reference to the accompanying drawings.

[0060] Please refer to Figure 1 and Figure 2 The embodiment of the present application provides a method for associating a person with a vehicle based on video surveillance, which is used to monitor a person getting off a vehicle and to bind the person getting off the vehicle, and includes the following steps S100 to S700:

[0061] Step S100: performing vehicle detection and tracking based on a first video image captured by a first camera on one side of the lane and a second video image captured by a second camera on the other side of the lane;

[0062] Step S200: During the vehicle detection and tracking process, detect the door opening area in each frame image i belonging to the same vehicle target j ;

[0063] Step S300: Identify the door opening area The human target Person Area , determine the human target Person Area Whether there is a get-off behavior, if it occurs, the human target Person Area Door opening area The corresponding vehicle is marked as the open door vehicle Car jopen , and then for the human target Person Area Perform detection and tracking to obtain the best quality face image Face best ;

[0064] Step S400: performing vehicle detection and tracking based on a third video image captured by a third camera located in the direction of the vehicle flow, and performing license plate positioning and recognition on each vehicle target frame obtained by vehicle detection and tracking during the tracking process;

[0065] Step S500: Get information about the vehicle target Car jopen The coordinates of the center point of the target frame at a time t are O 1 , obtain the center point coordinates O of the target frames of all vehicles in a frame image F corresponding to the third video image at time t m ;

[0066] Step S600: Set the target frame center point coordinate O 1 And all the center point coordinates O m Convert to the same reference coordinate system on the lane for proximity matching to obtain the vehicle target Car jopen License plate information;

[0067] Step S700: Transform the face image Face best 、Vehicle target Car jopen and vehicle target Car jopen The license plate information is bound and stored.

[0068] It is worth noting that if Figure 1 As shown, the first camera A is located on one side of the lane, and the second camera B is located on the other side of the lane. The lane can be a single lane or a multi-lane lane. The first camera A and the second camera B are located on opposite sides of the passing vehicle. Thus, for the passing vehicles on the lane, the first camera and the second camera are located on the side with the door of the passing vehicle. The third camera C is located on the side of the lane where the traffic is coming from, that is, when the vehicle passes on the lane, the vehicle drives towards the third camera. It can be understood that the third camera is located on the side of the front of the passing vehicle. The first camera, the second camera, the third camera, the first reference point and the multiple second reference points belong to the same monitoring area defined by the lane. The first reference point appears in the image field of the first camera, the second camera and the third camera at the same time. Similarly, the multiple reference points appear in the image field of the first camera, the second camera and the third camera at the same time. The first reference point and the second reference point may be, but are not limited to, circular reference objects, square reference objects, or cross reference objects set on the lane. Specifically, the first reference point and the second reference point may be the center of the circular reference object, the center of the square reference object, or the center of the cross. The first reference point and the second reference point may be reference points of different colors for easy distinction. Figure 1 In the coordinate system, the coordinate reference point is the first reference point and the calibration point is the second reference point.

[0069] In the above steps, the same steps performed in the first video image and the second video image, that is, the above steps S100 to S700 are involved in the first video image and the second video image. Wherein, for step S100, vehicle detection and tracking based on the first video image and vehicle tracking based on the second video image are performed by the same means, which is specifically implemented as follows: the vehicle target in each frame of the video image is detected using a first target detection model, the first target detection model can be a yolox model, the yolox model performs vehicle target detection on each frame image to obtain the vehicle target frame of each vehicle, and the ByteTrack tracking method is used to realize the tracking of the same vehicle target between the two adjacent frames. Since the vehicle target detection and tracking process involves vehicles in the open door state and vehicles in the closed door state, in order to make the yolox model detection more accurate, the data set for training the yolox model can include vehicles in the open door state and vehicles in the closed door state.

[0070] For step S200, the trained second target detection model is used to detect the door opening area in each frame image i belonging to the same vehicle target j. , specifically:

[0071] Collect a data set, which includes images of vehicles with open doors and images of vehicles with closed doors; perform data annotation: annotate each image of a vehicle with closed doors with a label of "closed door", annotate each image of a vehicle with open doors with a label of "open door" and select the open door area, such as Figure 3 As shown, Figure 3 The area enclosed by a rectangular frame is the door opening area; train to obtain the second target detection model: input the dataset of labeled information into the YOLOv9-S model, and use the existing model training method to train the YOLOv9-S model to obtain the second target detection model; detect the door opening area of ​​each vehicle target in each frame image: in the vehicle detection and tracking of the first video image, input the vehicle target frame area in each frame image into the second target detection model to obtain the door opening area; and in the vehicle detection and tracking of the second video image, input the vehicle target frame area in each frame image into the second target detection model to obtain the door opening area. Among them, the door opening area fed back by the second target detection model is as follows Figure 3 The rectangular area shown in the selection box.

[0072] It is worth noting that when a vehicle enters the monitoring area of ​​the first camera, the second camera, and the third camera, it is not known at what moment the vehicle stops and the door is opened to wait for people to get off the vehicle, so it is necessary to detect the door opening area of ​​each vehicle target in each frame image while performing vehicle detection and tracking. In order to improve detection efficiency and accuracy, when detecting the door opening area of ​​each vehicle, the entire image of each frame of the first video image or the second video image is not input into the second target detection model, but the vehicle target frame obtained by vehicle detection and tracking is input into the second target detection model, so that the interference area outside the vehicle area in the image can be excluded.

[0073] In step S300, the first video image or the second video image is represented by the video image V, and the image that appears in the door opening area is identified. The human target Person Area ,include:

[0074] Step S311: training a binary classification recognition model: collecting a training data set, the data set includes images containing human targets in the door opening area and images not containing human targets in the door opening area, wherein human targets include human targets in a state of sitting in a car, bending over to get out of a car, standing upright, and walking; inputting the images into an EfficientNet-B2 binary classification model for training to obtain a binary classification recognition model, the recognition results of the binary classification recognition model include true and false, a true result indicates that a human body is contained in the door opening area, and a false result indicates that a human body is not contained in the door opening area;

[0075] Step S312: Determine the door opening area Whether there is a human body: each frame of the video image V is sequentially divided into the door opening area in each frame according to the order of appearance. Input to the binary classification recognition model; when the door is opened at the kth frame When the recognition result of the input to the binary classification recognition model appears to be true for the first time, it indicates the door opening area The target person appears inside Area ;

[0076] Step S313: Using the human body recognition model to identify the door opening area of ​​the kth frame Perform human target detection in the selected area to obtain the human target Person Area ; where 1≤k≤i.

[0077] Furthermore, in the above step S313, human target detection is performed, specifically: each door opening area The input is sent to the human recognition model to detect whether there is a human target in the area selected by the door opening area. If there is a human target, the human target detection frame is output. The human recognition model can be but not limited to the yolox model. Since special attention is paid to the people in the door opening area, that is, the people who were originally sitting in the car and may get off the car later, the status of the people in the door opening area is diverse. In order to improve the detection accuracy of the human target detection model, human targets in the sitting state, bending state, upright state and walking state can be collected during the training of the yolox model.

[0078] In step S300, determine the human target Person Area Whether there is any getting off behavior, including the following steps:

[0079] Starting from the kth frame, obtain the door opening area of ​​each vehicle target j And the double door area The human target Person Area Tracking, obtaining human target Person Area The human target frame in each frame image, where the human target Person Area Tracking uses ByteTrack tracking method;

[0080] When the following conditions are met, it indicates that the human target Person Area Get off at frame L+1:

[0081] In the adjacent L-th frame image and L+1-th frame image: and ;

[0082] Where L represents any frame image starting from the kth frame in the video image V, L ≥ k, Indicates the door opening area of ​​vehicle target j in the Lth frame image A collection of pixels, Indicates the door opening area of ​​vehicle target j in the L+1th frame image A set of pixels; is the human target Person in the Lth frame image Area The pixel point set of the human target box, is the human target Person in the L+1th frame image Area The pixel point set of the human target box.

[0083] in, and is an operator that computes the relation between two sets. Indicates the collection For collection Subset. show Not for A subset of Can be used with Intersect or Not at all Inside.

[0084] In step S300, for the human target Person Area Perform detection and tracking to obtain the best quality face image Face best ,include:

[0085] Step S321: When the human target Person Area After getting off the bus, obtain the human target Person in the next N frames of images Area The detection box set Box, set Box = {box n ,n=1,…N}, where box n Represented as the human target Person in any frame n of N frames during the target tracking process Area Detection box of

[0086] Step S322: Detection box for each human target n Perform face detection in the selected area and obtain each detection box n All face areas in the area Face boxn ;

[0087] Step S323: Remove all face regions Face boxn Person that is not a human target Area The faces of pedestrians in the surroundings are interfered with, and the faces of human targets in the subsequent N frames of images are obtained. Are A set of face regions;

[0088] Step S324: Select the best quality face region image in the face region set based on the clarity, brightness value and face deflection angle as the face image Face best .

[0089] It is worth explaining that the purpose of this application is to obtain the person who gets off the vehicle and bind the person who gets off the vehicle to the vehicle, so that when the person who gets off the vehicle needs to enter the inspection hall for inspection and finds an abnormality, it can be associated with which vehicle the person who is inspected for abnormality is from. For this purpose, it is necessary to obtain the facial information of the person who gets off the vehicle. For a person who gets off the vehicle, after the vehicle door is opened, the vehicle presents a door opening area as described above. The person who may get off the vehicle later first appears in the door opening area as sitting in the car. At this time, the human target can be obtained, but because the first camera and the second camera take images on the side of the vehicle, although the human target that appears in the door opening area can be detected, the person's front face and high-quality facial information in terms of clarity are not easy to obtain. Therefore, it is necessary to first determine the human target Person that appears in the door opening area. Area Whether the person has gotten off the bus, starting from the frame of the image that determines whether the person has gotten off the bus, the human target Person Area Detection and tracking are performed, and subsequently in steps S321 to S324, high-quality facial images of the person getting off the bus are obtained during the human target detection and tracking process.

[0090] Furthermore, in the above step S322, the face area Face boxn Is the target Person in the human body Area Each detection box n Detected in, among which, the human target Person Area Each detection box n The face region can be detected by any existing target detection model, such as the Yolox detection model. Area After getting off the car, you will inevitably appear in the outdoor scene outside the car. In the outdoor scene, it is inevitable that other people who did not get off the car appear around the vehicle and happen to be close to the people who got off the car. In this case, the surrounding interfering people may appear in the detection box n At this time, the detection box box is extracted by step S322 n Face area within boxn When the face area Face is extracted boxn Will be greater than one, that is, the face area Face at this time boxn Including the person getting off the bus Area In view of this, the interference targets are removed through the above step S323.

[0091] Further, for step S323, all face regions Face are removed. boxn Person that is not a human target AreaThe surroundings interfere with the pedestrian's face. The specific implementation process is:

[0092] The human target Person in any frame n of the subsequent N frames Area The detection frame is evenly divided into an upper half area and a lower half area in the vertical axis direction;

[0093] Take the face region face as all face regions Face boxn One of the face regions;

[0094] If the face region face is located in the lower half, the face region face is the face of the interfering pedestrian to be removed;

[0095] If the face region face is located in the upper half area and the distance between the center point of the face region face and the center point of the upper half area is greater than a set threshold, the face region face is the face of an interfering pedestrian that needs to be removed.

[0096] Understandably, for a human target Person Area For the target Person, Area The face is located on the human body target Person Area The middle of the upper half of the detection frame, for the human target Person Area The face in the lower half of the detection frame is not the human target Person Area The face of the person. Area There are more than one face in the detection frame. The larger face that deviates from the middle of the upper half is not the human target Person. Area The face itself. ;

[0097] In step S324, based on the definition, brightness value and face deflection angle, the subsequent N frames of images obtained in step S323 are analyzed to identify the human target Person. Are The quality of each face region in the face region set is evaluated, and the face region image with the best quality is selected as the face image Face best , where the evaluation of the image quality S of each face area is: S=α*S angle +β*S clarity +γ* (-0.04 * (S brightness -50)²+100); S angle is the angle fraction, S clarity is the clarity score, S brightnessis the brightness score, α, β and γ are weighted coefficients, α+β+γ=1. The solution of the clarity score is any calculation method for evaluating the clarity of an image in the prior art, and the solution of the brightness score is any calculation method for evaluating the brightness of an image in the prior art. angle To evaluate the face posture score, the face posture includes the evaluation of the yaw angle direction, pitch angle direction and roll angle direction of the face, and the angle score S of the face involving the yaw angle, pitch angle and roll angle of the face is angle The calculation method is any method in the prior art and will not be described in detail here.

[0098] Furthermore, when a vehicle target opens its door, a door opening area appears. The size of the door opening area of ​​the same vehicle target in different frames may fluctuate. In step S100 and step S200, the same vehicle target is continuously tracked and detected in different frames, and the door opening area is detected in the vehicle target frame obtained by tracking. Then, whether the door opening area in different frames belongs to the same vehicle target is also known. Area Door opening area The corresponding vehicle is marked as Car jopen That is, for the same vehicle target j, when the door opening area can be detected in the target frame of the vehicle target j and the human target can be detected in the door opening area, the vehicle is marked as the door opening vehicle Car jopen ,Car jopen Indicates that vehicle target j is a vehicle with its door open.

[0099] It is worth noting that since it is necessary to detect whether a person has gotten off the vehicle, it is first necessary to detect whether the vehicle has opened the door. In order to more accurately detect the door opening area of ​​the vehicle, the first camera and the second camera located on both sides of the passing vehicle are used to collect vehicle images. However, since the first camera and the second camera are located on the side, it may not be so easy to detect the vehicle license plate information. The purpose of this application is to detect the person getting off the vehicle and associate the person getting off with the vehicle they are riding in, that is, it is necessary to know the vehicle license plate information. In view of this, a third camera located in the direction of the traffic flow is used to collect video images. The field of view of the third camera can conveniently monitor the license plate information of the front of the oncoming vehicle. Since in step S400, the vehicle is continuously detected and tracked based on the third camera, and the license plate is located and recognized for the target frame of each vehicle during the tracking process, wherein the license plate positioning and license plate recognition adopt any method of the prior art, even if the license plate of a vehicle is not captured at certain times, the license plate area will always be captured as the tracking process continues. However, since the first camera, the second camera and the third camera perform vehicle target detection and tracking respectively, it is necessary to associate and match the vehicle in the first camera with the vehicle in the third camera in order to know the door-opening vehicle Car monitored by the first camera. jopen The license plate information of the second camera is the same; this association matching process is implemented using steps S500 and S600.

[0100] Specifically, for step S500 and step S600, since the monitoring object is a vehicle with an open door and a human target in the door opening area, in order to exclude the association matching of irrelevant vehicles, that is, it is not necessary to perform association matching on all vehicles, then, it is necessary to obtain the vehicle target Car based on the first camera and the second camera in step S300. jopen As a benchmark, we can perform vehicle association matching under different camera fields of view. Therefore, when matching, we need to first obtain the vehicle target Car jopen The target frame at a time t and the target frames of all vehicles in a frame image F corresponding to the same time t, for different camera fields of view, select the vehicle information at the same time t to ensure the uniqueness of the vehicle position in the same reference coordinate system.

[0101] Specifically, for step S600, the target frame center point coordinate O 1 And all the center point coordinates O m Convert to the same reference coordinate system on the lane for proximity matching to obtain the vehicle target Car open License plate information, including:

[0102] Step S610: converting the image coordinate systems of all cameras to a physical reference coordinate system on the lane based on the reference objects set in the lane;

[0103] Step S620: Set the center point coordinate O 1 Convert to the physical coordinate value W in the physical reference coordinate system 1 ;

[0104] Step S630: Set the coordinates of each center point O m Convert to the physical coordinate value W in the physical reference coordinate system m ;

[0105] Step S640: The physical coordinate value W 1 With each physical coordinate value W m Calculate Euclidean distance;

[0106] Step S650: sort all the Euclidean distances and sort the physical coordinate value W corresponding to the minimum Euclidean distance. m The vehicle target is recorded as vehicle target Car min , with vehicle target Car min The license plate information is the vehicle target Car open license plate.

[0107] Further, for step S610, the image coordinate systems of all cameras are aligned to a physical reference coordinate system on the lane based on the reference objects set in the lane, including:

[0108] The first reference point set in the lane is taken as the origin, and the traffic direction of the lane is taken as the vertical axis Y. 1 And the vertical and longitudinal axis direction is the horizontal axis X 1 Experience the physical reference coordinate system;

[0109] Measure the coordinate values ​​P of multiple second reference points set on the lane in the physical reference coordinate system C ;

[0110] Get the coordinate value P of each second reference point in the image field of view of the first camera 1 , obtain the coordinate value P of each second reference point in the image field of view of the second camera 2 , obtain the coordinate value P of each second reference point in the image field of view of the third camera 4 ;

[0111] According to the coordinate values ​​P corresponding to at least four second reference points C and the coordinate values ​​P1 corresponding to the at least four second reference points, solving the first homography transformation matrix H describing the mapping relationship between the image coordinate system of the first camera and the physical reference coordinate system 1 ;

[0112] According to the coordinate values ​​P corresponding to at least four second reference points C and the coordinate values ​​P corresponding to the at least four second reference points2 , solve the second homography transformation matrix H describing the mapping relationship between the image coordinate system of the second camera and the physical reference coordinate system 2 ;

[0113] According to the coordinate values ​​P corresponding to at least four second reference points C and the coordinate values ​​P corresponding to the at least four second reference points 3 , solve the second homography transformation matrix H describing the mapping relationship between the image coordinate system of the third camera and the physical reference coordinate system 3 .

[0114] Among them, for the first homography transformation matrix H 1 , consisting of at least four pairs of coordinate values ​​P C And the coordinate value P1 solves the first homography transformation matrix H 1 The solution process can be obtained by using the opencv function opencv. findHomography(). For the matrix H 2 and the matrix H 3 The same is true.

[0115] The present application also provides a method for searching for a person-vehicle match, including the following steps:

[0116] Get the personnel P who have abnormal inspection during the inspection process curious face image face';

[0117] The face image face' is combined with the human-vehicle association method based on video surveillance to obtain all face images Face best Make a comparison;

[0118] When the match is successful, person P curious The vehicle for getting off is the vehicle obtained by the method for associating a person with a vehicle based on video surveillance and the face image Face. best Bound vehicle.

[0119] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A method for associating people and vehicles based on video surveillance, characterized in that: include: Performing vehicle detection and tracking based on a first video image captured by a first camera on one side of the lane and a second video image captured by a second camera on the other side of the lane; In the vehicle detection and tracking process, the door opening area in each frame image i belonging to the same vehicle target j is detected. ; Identify the door opening area The human target Person Area , determine the human target Person Area Whether there is a get-off behavior, if it occurs, the human target Person Area Door opening area The corresponding vehicle is marked as the open door vehicle Car jopen , and then for the human target Person Area Perform detection and tracking to obtain the best quality face image Face best ; Perform vehicle detection and tracking based on a third video image captured by a third camera located in the direction of the vehicle flow, and perform license plate positioning and recognition on each vehicle target frame obtained by vehicle detection and tracking during the tracking process; Get information about the vehicle target Car jopen The target frame center point coordinate O1 at a time t is obtained, and the target frame center point coordinate O1 of all vehicles in a frame image F corresponding to the third video image at the time t is obtained. m ; The target frame center point coordinate O1 and all the center point coordinates O m Convert to the same reference coordinate system on the lane for proximity matching to obtain the vehicle target Car jopen License plate information; as well as The face image Face best 、Vehicle target Car jopen and vehicle target Car jopen The license plate information is bound and stored.

2. The method for associating a person with a vehicle based on video surveillance according to claim 1, characterized in that: Identify the door opening area The human target Person Area ,include: Training a binary classification recognition model: Collect training data sets, which include images containing human targets in the door opening area and images not containing human targets in the door opening area. Human targets include human targets sitting in the car, bending over to get out of the car, standing upright, and walking. Input the data into the EfficientNet-B2 binary classification model for training to obtain a binary classification recognition model. The recognition results of the binary classification recognition model include true and false. A true result indicates that the door opening area contains a human body, and a false result indicates that the door opening area does not contain a human body. Determine the door opening area Whether there is a human body: each frame of the video image V is sequentially divided into the door opening area in each frame according to the order of appearance. Input to the binary classification recognition model; when the door is opened at the kth frame When the recognition result of the input to the binary classification recognition model appears to be true for the first time, it indicates the door opening area The target person appears inside Area ; Use the human body recognition model to identify the door opening area of ​​the kth frame Perform human target detection in the selected area to obtain the human target Person Area ; Wherein, 1≤k≤i, the video image V represents the first video image or the second video image.

3. The method for associating a person with a vehicle based on video surveillance as claimed in claim 2, characterized in that: Judge the target person Area Whether there is any disembarkation behavior, including: Starting from the kth frame, obtain the door opening area of ​​each vehicle target j And the double door area The human target Person Area Tracking, obtaining human target Person Area The human target box in each frame; When the following conditions are met, it indicates that the human target Person Area Get off at frame L+1: In the adjacent L-th frame image and L+1-th frame image: and ; Where L represents any frame image starting from the kth frame in the video image V, L ≥ k, Indicates the door opening area of ​​vehicle target j in the Lth frame image A collection of pixels, Indicates the door opening area of ​​vehicle target j in the L+1th frame image A set of pixels; is the human target Person in the Lth frame image Area The pixel point set of the human target box, is the human target Person in the L+1th frame image Area The pixel point set of the human target box.

4. The method for associating a person with a vehicle based on video surveillance as claimed in claim 3, characterized in that: For human target Person Area Perform detection and tracking to obtain the best quality face image Face best ,include: When the human target Person Area After getting off the bus, obtain the human target Person in the next N frames of images Area The detection box set Box, set Box = {box n ,n=1,…N}, where box n Represented as the human target Person in any frame n of N frames during the target tracking process Area Detection box of Detection box for each human target n Perform face detection in the selected area and obtain each detection box n All face areas in the area Face boxn ; Remove all face areas Face boxn Person that is not a human target Area The faces of pedestrians in the surroundings are interfered with, and the faces of human targets in the subsequent N frames of images are obtained. Are A set of face regions; Based on the clarity, brightness value and face deflection angle, the best quality face region image in the face region set is selected as the face image Face best .

5. The method for associating a person with a vehicle based on video surveillance as claimed in claim 4, characterized in that: Remove all face areas Face boxn Person that is not a human target Area The surrounding interfering pedestrians’ faces are: The human target Person in any frame n of the subsequent N frames Area The detection frame is evenly divided into an upper half area and a lower half area in the vertical axis direction; Take the face region face as all face regions Face boxn One of the face regions; If the face region face is located in the lower half, the face region face is the face of the interfering pedestrian to be removed; If the face region face is located in the upper half area and the distance between the center point of the face region face and the center point of the upper half area is greater than a set threshold, the face region face is the face of an interfering pedestrian that needs to be removed.

6. The method for associating a person with a vehicle based on video surveillance according to claim 1, characterized in that: The target frame center point coordinate O1 and all the center point coordinates O m Convert to the same reference coordinate system on the lane for proximity matching to obtain the vehicle target Car open License plate information, including: Based on the reference objects set in the lane, the image coordinate systems of all cameras are converted to a physical reference coordinate system on the lane; Convert the center point coordinate O1 to the physical coordinate value W1 in the physical reference coordinate system; The coordinates of each center point O m Convert to the physical coordinate value W in the physical reference coordinate system m ; The physical coordinate value W1 and each physical coordinate value W m Calculate Euclidean distance; Sort all Euclidean distances and take the physical coordinate value W corresponding to the minimum Euclidean distance m The vehicle target is recorded as vehicle target Car min , with vehicle target Car min The license plate information is the vehicle target Car open license plate.

7. The method for associating a person with a vehicle based on video surveillance as claimed in claim 6, characterized in that: Based on the reference objects set in the lane, the image coordinate systems of all cameras are converted to a physical reference coordinate system on the lane, including: A physical reference coordinate system is defined by taking a first reference point set in the lane as the origin, taking the traffic direction of the lane as the longitudinal axis Y1 and taking the direction perpendicular to the longitudinal axis as the transverse axis X1; Measure the coordinate values ​​P of multiple second reference points set on the lane in the physical reference coordinate system C ; Obtaining a coordinate value P1 of each second reference point in the image field of view of the first camera, obtaining a coordinate value P2 of each second reference point in the image field of view of the second camera, and obtaining a coordinate value P4 of each second reference point in the image field of view of the third camera; According to the coordinate values ​​P corresponding to at least four second reference points C and the coordinate values ​​P1 corresponding to the at least four second reference points, solving a first homography transformation matrix H1 describing a mapping relationship between an image coordinate system of the first camera and a physical reference coordinate system; According to the coordinate values ​​P corresponding to at least four second reference points C and the coordinate values ​​P2 corresponding to the at least four second reference points, solving a second homography transformation matrix H2 describing a mapping relationship between an image coordinate system of the second camera and a physical reference coordinate system; According to the coordinate values ​​P corresponding to at least four second reference points C The coordinate values ​​P3 corresponding to the at least four second reference points are used to solve a second homography transformation matrix H3 describing a mapping relationship between the image coordinate system of the third camera and the physical reference coordinate system.

8. A person-vehicle matching retrieval method, based on the person-vehicle association method based on video surveillance as claimed in any one of claims 1 to 7, characterized in that: include: Get the personnel P who have abnormal inspection during the inspection process curious face image face'; The face image face' is combined with the human-vehicle association method based on video surveillance to obtain all face images Face best Make a comparison; as well as When the match is successful, person P curious The vehicle for getting off is the vehicle obtained by the method for associating a person with a vehicle based on video surveillance and the face image Face best Bound vehicle.

Images