A system for measuring the number of people in vehicles at a gate based on three-dimensional imaging and radar technology

Abstract

The present invention discloses a system for counting the number of people in a vehicle at a gate based on three-dimensional image and radar technology. The system comprises a radar speed, distance and vehicle measurement type module, a camera photo feature extraction module, a candidate area selection and calculation module, a feature integration module and a target detection and calculation module. The radar speed, distance and vehicle measurement type module comprises: radar speed measurement, radar distance measurement and radar vehicle measurement type; the camera photo feature extraction module comprises: camera photo taking, extracting point cloud top view features, extracting point cloud front view features and extracting image features; the candidate area calculation module comprises: calculating a candidate area from top view features; the feature integration module comprises: integrating the candidate area with three features, projecting the top view candidate area onto the front view and integrating the uniformity; and the target detection and calculation module comprises: statistically calculating the number of people in the vehicle.

Description

Technical Field

[0001] The present invention relates to a vehicle-in-vehicle number counting system, in particular to a vehicle-in-vehicle number counting system at a gate based on three-dimensional image and radar technology. Background Art

[0002] Governments and industrial parks at all levels place particular emphasis on personnel safety management. For example, government agencies and businesses need to identify the number of people in vehicles entering and exiting for safety reasons. Traditional methods, where staff visually record the number of people in vehicles, present numerous challenges. First, this is time-consuming and inefficient, hindering the passage of following vehicles. Second, staff may make errors in recording the number of people, compromising the safety of the organization.

[0003] Currently, a technology that significantly improves efficiency is the use of infrared thermal imaging for detection and recording. This technology utilizes the difference in temperature between a person and the car to easily identify the number of people inside the vehicle and automatically record the number of people. However, for infrared thermal imaging to detect the number of people inside a vehicle, the windows must be open. Infrared thermal imaging uses a wavelength of 8-14 μm, which cannot penetrate glass and cannot detect the number of people inside. Therefore, this invisible light recognition technology can create a negative experience for car owners. Summary of the Invention

[0004] Purpose of the invention: The technical problem to be solved by the present invention is to provide a system for measuring the number of people in vehicles at a gate based on three-dimensional images and radar technology in response to the shortcomings of the existing technology.

[0005] To solve the above technical problems, the present invention discloses a system for counting the number of people in vehicles at a gate based on three-dimensional imaging and radar technology, comprising: a radar speed measurement, distance measurement, and vehicle type measurement module, a camera feature extraction module, a candidate area selection and calculation module, a feature integration module, and a target detection and calculation module;

[0006] Among them, the radar speed measurement, distance measurement and vehicle measurement type modules include: radar speed measurement, radar distance measurement and radar vehicle measurement type;

[0007] The camera feature extraction module includes: camera photography, point cloud top view feature extraction, point cloud front view feature extraction and image feature extraction;

[0008] The candidate region calculation module includes: calculating the candidate region from the top view features;

[0009] The feature integration module includes: integrating the candidate area with three features, projecting the candidate area of ​​the top view to the front view, and integrating the uniformity;

[0010] The target detection calculation module includes: counting and calculating the number of people in the car.

[0011] The method for calculating the number of people in a vehicle using the system includes:

[0012] Step 1: Use the radar speed, distance and vehicle type measurement modules to measure speed, distance and vehicle type;

[0013] Step 2: Extract image features through the camera feature extraction module;

[0014] Step 3, calculating the candidate area through the candidate area selection calculation module;

[0015] Step 4: Integrate the features through the feature integration module;

[0016] In step 5, the target detection calculation module is used to count the number of people in the target vehicle through the methods of cone generation, 3D instance segmentation and 3D boundary regression calculation.

[0017] Step 1 of the present invention comprises:

[0018] Step 1-1: When a vehicle enters the 100-meter range in front of the gate, the radar measures the vehicle speed. If the vehicle speed exceeds 30 km / h, a reminder will be displayed on the display above the gate; otherwise, it goes to step 1-2;

[0019] Steps 1-2: Calculate the vehicle's height through radar and determine whether it is a large or small vehicle. (The vehicle identification criteria are: 0.9m < front height < 1.1m, 1.4m < body height 1.6m for a small car; 1.1m < front height < 1.5m, 1.68m < body height 1.8m for an SUV; 1.8m < body height < 4m for a heavy or medium-sized truck; 4m < body height < 4.2m for a container truck; vehicles taller than that cannot be identified by the system.)

[0020] The present invention uses the camera feature extraction module to extract and process three-dimensional graphics features, that is, step 2 includes:

[0021] Step 2-1: Based on the vehicle type detected by the radar in step 1-2, the camera height and angle are dynamically adjusted when the vehicle enters the 10-meter range of the gate. At the same time, the fill light above the gate is turned on to ensure that the captured images and videos are clear and visible, ensuring that the camera can penetrate the car glass.

[0022] Step 2-2: Use a camera to quickly construct a three-dimensional model from the captured vehicle image to obtain a three-dimensional image, which includes a top view and a front view;

[0023] Step 2-3, based on the top view of the three-dimensional image obtained in step 2-2, extract features from the three dimensions of height, intensity, and density in the top view; the height is calculated by calculating the highest value in the unit cell of the point cloud in the three-dimensional image, dividing the point cloud into M blocks, and then calculating the corresponding height map for each block, thereby obtaining M height maps; the intensity is calculated by cyclically calculating the value of the point with the maximum height in the unit cell of the M height maps, which is the intensity value; the density is calculated by the data of the point in each cell of the M height maps, and is calculated by the following formula:

[0024]

[0025] Where N is the number of points in the cell; intensity and density are the point cloud feature values ​​of the top view in the entire 3D image, height is calculated from M slices, and the total top view is encoded as (M+2) channel features;

[0026] Step 2-4, based on the front view in the 3D image obtained in step 2-2, calculate and project it onto a cylindrical surface to form a dense front view to extract the vehicle front view features, including:

[0027] Assume that the three-dimensional coordinate p is: p = (x, y, z)

[0028] That is, the front view coordinate p is obtained fv For: p fv =(r,c)

[0029] The r-axis and c-axis are calculated using the following formula:

[0030]

[0031] c=atan2(y,x) / Δθ

[0032] Among them, atan2 is the inverse tangent function, Δφ is the diameter variable, and Δθ is the angle variable;

[0033] Steps 2-5 use the convolutional neural network model pre-trained with the Visual Geometry Group Network (VGG-16) for feature extraction, including:

[0034] The vehicle front view features obtained in steps 2-3 and 2-4 are used as input to perform feature extraction by adapting the VGG16.npy in the visual geometry group network.

[0035] Step 3 of the present invention comprises:

[0036] Step 3-1: Select the top view in the 3D graphics to participate in the candidate area calculation:

[0037] The top view was chosen as the candidate region based on three key insights from experiments: 1) When objects are projected onto the top view, their original physical dimensions are maintained, resulting in smaller size variance, a benefit not found in the front view. 2) The top view avoids occlusion issues because the car occupies different spaces. 3) In scenarios where cars enter a gate, the vertical position variance is smaller, providing a good foundation for obtaining an accurate 3D model.

[0038] Step 3-2: Use the Region Proposal Network (RPN) to extract candidate boxes.

[0039] Step 3-3, obtain the candidate area of ​​the top view.

[0040] Step 4 of the present invention comprises:

[0041] Step 4-1: Integrate the top view candidate regions obtained in the candidate region calculation module with the features obtained in the camera feature extraction module;

[0042] Step 4-2, projecting the top view candidate region into the front view and 3D image, includes:

[0043] The top view candidate region in each 3D image is parameterized as:

[0044] (x,y,z,l,w,h)

[0045] Where x, y, z are the three-dimensional coordinate axes, l, w, h are the length, width, and height; these anchor points are represented by (x, y, l, w); after discretization, the anchor points in the front view and image are obtained;

[0046] Step 4-3: Use the ROI pooling method to reduce the dimensionality of the three-dimensional image data to the same dimension.

[0047] The cone generation described in step 5 of the present invention refers to feature extraction based on the VGG-16 neural network, and then predicting anchor points as the cone using the Faster PCNN network; the three-dimensional instance segmentation refers to rotating each cone to obtain a central perspective, using this perspective as the point cloud data of the coordinate axis, and then performing instance segmentation on the point cloud data to determine whether each anchor point belongs to the target point.

[0048] The calculation method for three-dimensional boundary regression described in step 5 of the present invention includes: using the results of three-dimensional instance segmentation as a mark to obtain a point cloud set of the target; calculating the center of mass of the point cloud set of the target and using it as the origin of the new coordinate system; translating the view cone point cloud to the origin of the new coordinate system, and then regressing the center, size and direction of the three-dimensional boundary to output the result.

[0049] In step 3-2 of the present invention, a region generation network RPN is used to extract candidate frames, including:

[0050] The front view features of the vehicle obtained by VGG-16 convolution are then converted into a basic feature map through a series of convolution calculations;

[0051] After convolution calculation, a set of feature maps are obtained;

[0052] Combined with a set of anchor points, the anchor point is the center point selected for a candidate box during the convolution process, and the candidate box is obtained after post-processing.

[0053] The target detection calculation module of the present invention adopts the Faster PCNN network structure to perform calculations, including the following steps:

[0054] For the selected candidate box, after RPN correction, the selected candidate box is calculated and determined;

[0055] Map candidate boxes to a common feature map;

[0056] Reduce the dimension of the feature map to the same dimension;

[0057] By classifying and regressing the ROI features, we find the image features that correspond to the candidate boxes one by one, and add 1 to their counts.

[0058] Beneficial effects:

[0059] When a vehicle passes through a gate, the driver and passengers in the vehicle can count the number of people in the vehicle without being aware of the situation. This is achieved by using the radar's sensitive target capture characteristics to assist high-definition camera photography, three-dimensional modeling, AI image recognition and other technologies. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments, and the above and / or other advantages of the present invention will become more apparent.

[0061] Figure 1 It is a schematic diagram of the system module of the present invention.

[0062] Figure 2 It is a flow chart of the method adopted by the system of the present invention.

[0063] Figure 3 This is a diagram illustrating the process of identifying the number of people in a vehicle in the present invention. DETAILED DESCRIPTION

[0064] like Figure 1As shown, the system of the present invention includes five modules: radar speed, distance and vehicle measurement module, camera feature extraction module, candidate area selection calculation module, feature integration module, and target detection calculation module. These five functional modules are designed based on the technical architecture of artificial intelligence.

[0065] The radar speed and distance measurement module has three main functions. First, when a vehicle enters the lane 100 meters ahead, the radar can measure the vehicle's speed. If the speed exceeds 30 km / h, the gate display will remind the driver to control the speed. Second, when the vehicle enters within 100 meters, the radar measures the vehicle's height to determine whether it is a large or small vehicle. Finally, when the vehicle enters the 10-meter zone, the radar will turn on the auxiliary fill light and adjust the height and angle of the camera according to the type of vehicle.

[0066] The camera feature extraction module is used to take a set of photos and videos by adjusting the camera height and shooting angle with the help of radar, and quickly synthesize them into a three-dimensional model image, and then perform image feature extraction based on the top view and front view taken;

[0067] The candidate region selection calculation module is used to select a top view to participate in the calculation of the view network, which involves data calculation;

[0068] The feature integration module is used to combine the three steps of the point cloud top view feature extraction, point cloud front view feature extraction, and image feature extraction in the camera feature extraction module to form a fused 3D graphic feature involved in the calculation, then project the top view candidate area into the front view, and finally perform the same latitude adjustment to find the optimal feature for integration;

[0069] The target detection calculation is used to identify the cumulative process of the number of people in the car using neural network calculation, which mainly uses the Faster PCNN network structure for calculation.

[0070] like Figure 2 As shown, the radar speed and distance measurement vehicle type specifically performs the following steps:

[0071] Step a1: When a vehicle enters the 100-meter range in front of the gate, the radar will test the vehicle's speed. If the vehicle's speed exceeds 30 km / h, the display above the gate will remind it to control the speed within 30 km / h. If it does not exceed 30 km / h, it will proceed to step a2.

[0072] Step a2: When a car enters the 100-meter range of the gate, the radar will calculate the height of the car and determine whether it is a large car or a small car;

[0073] The three-dimensional graphic feature extraction specifically performs the following steps:

[0074] Step b1: Dynamically adjust the camera height and angle based on the vehicle type detected by the radar, and simultaneously turn on the fill light above the gate to ensure that the captured images and videos are clear and visible, while also ensuring that the camera can penetrate the car glass;

[0075] Step b2: Using the camera's built-in image processing function to quickly construct the captured vehicle image into a 3D model and output a 3D image;

[0076] Step b3: Based on the top view of the 3D image formed in step b2, extract features from the three dimensions of height, intensity, and density in the top view. The height is calculated by calculating the highest value in the unit cell of the point cloud in the 3D image. Here, the point cloud is divided into M blocks, and then the corresponding height map is calculated for each block, thereby obtaining M height maps. The intensity is calculated by cyclically finding the value of the point with the maximum height in the unit cell of the M height maps, which is the intensity value. The density is the data of the point in each cell of the M height maps, which is calculated as follows:

[0077] Where N is the number of points in the cell. Here, intensity and density are calculated for the entire point cloud feature value of the entire 3D top view, while height is calculated for M slices. In this way, the total top view is encoded as (M+2) channel features.

[0078] Step b4: Based on the front view of the 3D image formed in step b2, calculate and project it onto a cylindrical surface to form a dense front view to extract vehicle features. The purpose of front view feature extraction is to prevent the 3D image point cloud from being very sparse, which will be very sparse when projected onto a 2D image. Therefore, it is necessary to project the captured front view onto a cylindrical surface to produce a new and dense front view. The specific calculation is as follows:

[0079] Assume the three-dimensional coordinates are: p = (x, y, z)

[0080] That is, the front view is: p fv =(r,c)

[0081] Where r and c can be calculated as follows:

[0082]

[0083] c=atan 2(y,x) / Δθ

[0084] Among them, atan2 is the inverse tangent function, Δφ is the diameter variable, and Δθ is the angle variable;

[0085] Step b5: Based on step b4, the VGG-16 (reference: Ge Huilin et al. Application of Multiscale Facial Feature Manifold Learning Based on VGG-16 [J]. Journal of Sensors, 2021, 2021) pre-trained convolutional neural network model is used for feature extraction. The VGG-16 pre-trained model is a third-party model; the VGG-16 pre-trained model is a complete VGG16.npy pre-trained model download provided by a third party. Here, only steps b3 and b4 in claim 3 are required as input, and then VGG16.npy is simply adjusted as follows to perform feature extraction:

[0086]

[0087] The network for determining candidate regions primarily selects the top view for candidate region calculation. The top view is chosen as a candidate region based on three key considerations: 1) When projecting objects onto the top view, the original physical dimensions are preserved, resulting in smaller size variance, a benefit not found in the front view; 2) The top view avoids occlusion issues because the car occupies different spaces; and 3) In scenarios where a car enters a gate, the variance in the car's vertical position is smaller, providing a good foundation for obtaining an accurate 3D model.

[0088] The feature integration specifically performs the following steps:

[0089] Step c1: Integrate the candidate region with the features obtained from the output of steps b3, b4, and b5 in the 3D graphic feature extraction;

[0090] Step c2: Project the top view candidate region into the front view and image; the approach is to parameterize each 3D candidate box as:

[0091] (x,y,z,l,w,h), where x, y, z are the three-dimensional space coordinate axes, l, w, h are the length, width, and height

[0092] These anchor points are represented by (x,y,l,w)

[0093] In this way, after discretization, the anchor points in the front view and image can be obtained;

[0094] Step c3: Reduce the dimensionality of the 3D image data to the same dimension. This is done using the ROI pooling method (reference: Yi Hou et al. Efficient ConvNet Feature Extraction with Multiple RoIPooling for Landmark-Based Visual Localization of Autonomous Vehicles [J]. Mobile Information Systems, 2017, 2017: 1-14). ROI pooling is a mature third-party feature map frame algorithm.

[0095] Step c2: Project the candidate area of ​​the top view to the front view and image. The specific method is to parameterize each 3D candidate box as follows:

[0096] (x, y, z, l, w, h), where x, y, z are the three-dimensional space coordinate axes, l, w, h are the length, width, and height; these anchor points are represented by (x, y, l, w); after discretization, the anchor points in the front view and image can be obtained;

[0097] The target detection calculation is an accumulation process of identifying the number of people in the car through the neural network calculation of claims 8 and 9, which mainly adopts the Faster PCNN network structure for calculation (reference: Zhili Chen, Zhimin Zhao, Adamu Abubakar Abba. Detection of Microcalcifications in Mammograms Based on HyperFaster R-CNN[C] / / . Conference proceedings of 2021 4th International Conferenceon Algorithms, Computing and Artificial Intelligence(ACAi 2021).,2021:416-423.DOI:10.26914 / c.cnkihy.2021.055265.), and its recognition process, such as Figure 3 As shown:

[0098] The selected candidate box is corrected by RPN and the selected candidate box is calculated and determined;

[0099] Map candidate boxes to a common feature map;

[0100] Reduce the dimension of the feature map to the same dimension;

[0101] By classifying and regressing the ROI features, we find the image features that correspond to the candidate boxes one by one, and add 1 to their counts.

[0102] The present invention provides a system for counting the number of people in vehicles at a gate based on three-dimensional imaging and radar technology. While there are numerous methods and approaches for implementing this technical solution, the foregoing merely represents a preferred embodiment of the present invention. It should be noted that those skilled in the art may make improvements and modifications without departing from the principles of the present invention, and such improvements and modifications are also within the scope of protection of the present invention. Any components not specified in this embodiment may be implemented using existing technologies.

Claims

1. A system for counting the number of people in vehicles at a gate based on three-dimensional imaging and radar technology, characterized in that: It includes radar speed measurement, ranging measurement and vehicle type module, camera photo feature extraction module, candidate area selection calculation module, feature integration module and target detection calculation module; Among them, the radar speed measurement, distance measurement and vehicle measurement type modules include: radar speed measurement, radar distance measurement and radar vehicle measurement type; The camera feature extraction module includes: camera photography, point cloud top view feature extraction, point cloud front view feature extraction and image feature extraction; The candidate region calculation module includes: calculating the candidate region from the top view features; The feature integration module includes: integrating the candidate area with three features, projecting the candidate area of ​​the top view to the front view, and integrating the uniformity; The target detection calculation module includes: counting and calculating the number of people in the car; Among them, the method of counting the number of people in the car includes: Step 1: Use the radar speed, distance and vehicle type measurement modules to measure speed, distance and vehicle type; Step 2: Extract image features through the camera feature extraction module; Step 3, calculating the candidate area through the candidate area selection calculation module; Step 4: Integrate the features through the feature integration module; Step 5: Use the target detection calculation module to count the number of people in the target car through the methods of cone generation, 3D instance segmentation and 3D boundary regression calculation; Step 1 includes: Step 1-1: When a vehicle enters the 100-meter range in front of the gate, the radar measures the vehicle speed. If the vehicle speed exceeds 30 km / h, a reminder will be displayed on the display above the gate; otherwise, it goes to step 1-2; Step 1-2: Calculate the vehicle's height using radar to determine whether it is a large vehicle or a small vehicle. Step 2 includes: Step 2-1: Based on the vehicle type detected by the radar in step 1-2, the camera height and angle are dynamically adjusted when the vehicle enters the 10-meter range of the gate. At the same time, the fill light above the gate is turned on to ensure that the captured images and videos are clear and visible, ensuring that the camera can penetrate the car glass. Step 2-2: Use a camera to quickly construct a three-dimensional model from the captured vehicle image to obtain a three-dimensional image, which includes a top view and a front view; Step 2-3, based on the top view of the three-dimensional image obtained in step 2-2, extract features from the three dimensions of height, intensity, and density in the top view; the height is calculated by calculating the highest value in the unit cell of the point cloud in the three-dimensional image, dividing the point cloud into M blocks, and then calculating the corresponding height map for each block, thereby obtaining M height maps; the intensity is calculated by cyclically calculating the value of the point with the maximum height in the unit cell of the M height maps, which is the intensity value; the density is calculated by the data of the point in each cell of the M height maps, and is calculated by the following formula: Where N is the number of points in the cell; intensity and density are the point cloud feature values ​​of the top view in the entire 3D image, height is calculated from M slices, and the total top view is encoded as (M+2) channel features; Step 2-4, based on the front view in the 3D image obtained in step 2-2, calculate and project it onto a cylindrical surface to form a dense front view to extract the vehicle front view features, including: Assume that the three-dimensional coordinate p is: p = (x, y, z) That is, the front view coordinate p is obtained fv For: p fv =(r,c) The r-axis and c-axis are calculated using the following formula: c=atan2(y,x) / Δθ Among them, atan2 is the inverse tangent function, Δφ is the diameter variable, and Δθ is the angle variable; Steps 2-5 use the convolutional neural network model pre-trained with the Visual Geometry Group Network (VGG-16) for feature extraction, including: The vehicle front view features obtained in steps 2-3 and 2-4 are used as input to perform feature extraction by adapting the VGG16.npy in the visual geometry group network.

2. The system for counting the number of people in vehicles at a gate based on three-dimensional imaging and radar technology according to claim 1, characterized in that: Step 3 includes: Step 3-1, select the top view in the 3D graphics to participate in the candidate area calculation; Step 3-2: Use the region generation network RPN to extract candidate boxes; Step 3-3, obtain the candidate area of ​​the top view.

3. The system for counting the number of people in vehicles at a gate based on three-dimensional imaging and radar technology according to claim 2, characterized in that: Step 4 includes: Step 4-1: Integrate the top view candidate regions obtained in the candidate region calculation module with the features obtained in the camera feature extraction module; Step 4-2, projecting the top view candidate region into the front view and 3D image, includes: The top view candidate region in each 3D image is parameterized as: (x,y,z,l,w,h) Where x, y, z are the three-dimensional coordinate axes, l, w, h are the length, width, and height; these anchor points are represented by (x, y, l, w); after discretization, the anchor points in the front view and image are obtained; Step 4-3: Use the ROIpooling method to reduce the dimensionality of the three-dimensional image data to the same dimension.

4. The system for counting the number of people in vehicles at a gate based on three-dimensional imaging and radar technology according to claim 3 is characterized in that: The cone generation described in step 5 refers to feature extraction based on the VGG-16 neural network, and then the anchor points are predicted as the cones using the FasterPCNN network; the three-dimensional instance segmentation refers to rotating each cone to obtain the central perspective, using this perspective as the point cloud data of the coordinate axis, and then performing instance segmentation on the point cloud data to determine whether each anchor point belongs to the target point.

5. The system for counting the number of people in a vehicle at a gate based on three-dimensional imaging and radar technology according to claim 4 is characterized in that: The calculation method for three-dimensional boundary regression described in step 5 includes: using the result of three-dimensional instance segmentation as a mark to obtain a point cloud set of the target; calculating the center of mass of the point cloud set of the target and using it as the origin of the new coordinate system; translating the view cone point cloud to the origin of the new coordinate system, and then regressing the center, size and direction of the three-dimensional boundary to output the result.

6. The system for counting the number of people in a vehicle at a gate based on three-dimensional imaging and radar technology according to claim 5, characterized in that: In step 3-2, the region generation network RPN is used to extract candidate boxes, including: The front view features of the vehicle obtained by VGG-16 convolution are then converted into a basic feature map through a series of convolution calculations; After convolution calculation, a set of feature maps are obtained; Combined with a set of anchor points, the anchor point is the center point selected for a candidate box during the convolution process, and the candidate box is obtained after post-processing.

7. The system for counting the number of people in a vehicle at a gate based on three-dimensional imaging and radar technology according to claim 6, characterized in that: The target detection calculation module uses the Faster PCNN network structure to perform calculations, which includes: For the selected candidate box, after RPN correction, the selected candidate box is calculated and determined; Map candidate boxes to a common feature map; Reduce the dimension of the feature map to the same dimension; By classifying and regressing the ROI features, we find the image features that correspond to the candidate boxes one by one, and add 1 to their counts.

Images