AI barrier gate monitoring system based on radar and image dual analysis

Abstract

The invention relates to the technical field of intelligent traffic, in particular to an AI barrier gate monitoring system based on radar and image dual analysis, which comprises an intelligent license plate feature extraction module, a multi-sensor data fusion module and a self-adaptive visual adjustment module, wherein the intelligent license plate feature extraction module adopts an edge detection algorithm to extract multiple pieces of edge information in a vehicle two-dimensional image, distinguishes license plate area edge information, and adopts a projection method to extract license plate characters in the license plate area edge information; the multi-sensor data fusion module fuses the vehicle two-dimensional image and the vehicle three-dimensional image by adopting a data fusion technology, and defines the two-dimensional image and the three-dimensional image as a fused image; and the self-adaptive visual adjustment module senses whether the intelligent license plate feature extraction module can successfully identify the edge information of the license plate region in real time, analyzes the contour coordinate points of the vehicle by using the fused image when the edge information cannot be identified, calculates the angle to be adjusted of the camera by using a vehicle positioning method, and adjusts the focal length according to the distance between the vehicle and the camera.

Description

Technical Field

[0001] The present invention relates to the field of intelligent transportation technology, and in particular to an AI barrier gate monitoring system combining radar and image dual analysis. Background Art

[0002] In the field of intelligent transportation, barrier systems are key equipment for vehicle access management. Existing barrier systems typically install cameras and lidar sensors in front of the barrier. The lidar creates a three-dimensional image of the vehicle to determine its distance from the barrier, while the camera captures a two-dimensional image of the vehicle for license plate recognition. When the lidar detects a vehicle entering the camera's monitoring range, the camera begins continuously capturing a two-dimensional image of the vehicle, determining whether the vehicle's license plate information is present in the barrier system and allowing the vehicle to pass.

[0003] However, due to low-light environments (such as at night and in underground garages), the two-dimensional vehicle images captured by the camera are prone to problems such as high noise and low contrast, resulting in blurred images and difficulty in clearly presenting license plate features. Strong light interference (such as direct sunlight and vehicle high beams) will cause the image to be overexposed or underexposed, and it is also impossible to accurately identify the edge information of the license plate area and extract characters. At the same time, the vehicle driving path in actual scenarios is highly uncertain. Some vehicles drive in at an angle, approach from the side, or deviate from the normal route due to avoidance, causing the license plate position to shift in the two-dimensional image captured by the camera with a fixed angle of view, resulting in license plate information distortion and edge loss, which seriously affects the accuracy of license plate recognition. In view of this, we propose an AI barrier gate monitoring system that combines radar and image dual analysis. Summary of the Invention

[0004] The purpose of this invention is to solve the problems of difficulty in displaying the license plate features in the two-dimensional images of vehicles collected by cameras in low-light and strong-light environments, difficulty in extracting the edge information and characters of the license plate area, and the problem of license plate offset, information distortion and loss in the two-dimensional images under fixed viewing angles with variable vehicle driving paths, and inaccurate license plate recognition.

[0005] To achieve the above objectives, the present invention provides an AI barrier gate monitoring system that combines radar and image dual analysis, including an intelligent license plate feature extraction module, a multi-sensor data fusion module, and an adaptive visual adjustment module, wherein:

[0006] The intelligent license plate feature extraction module uses an edge detection algorithm to extract multiple edge information in the vehicle two-dimensional image, distinguishes the edge information of the license plate area, and uses a projection method to extract the license plate characters in the edge information of the license plate area; the multi-sensor data fusion module uses data fusion technology to fuse the vehicle two-dimensional image and the vehicle three-dimensional image, and defines it as a fused image; the adaptive visual adjustment module perceives that the intelligent license plate feature extraction module cannot recognize the edge information signal of the license plate area, and uses the vehicle positioning method to analyze whether the camera needs to adjust the angle, and the vehicle positioning method is analyzed through fused image analysis.

[0007] As a further improvement of this technical solution, the working principle of the edge detection algorithm in the intelligent license plate feature extraction module is as follows: the two-dimensional image of the vehicle collected by the perception camera is smoothed by a Gaussian filter. The template of the Gaussian filter is a two-dimensional Gaussian function, and its formula is: (x, y) is the coordinate in the template, σ is the standard deviation of the Gaussian distribution, and the smoothed image is obtained by convolving the vehicle's two-dimensional image with the Gaussian template;

[0008] The Sobel operator is used to calculate the horizontal and vertical gradients of the smoothed vehicle two-dimensional image, where The vehicle 2D image is respectively x and S y Convolution operation to obtain the horizontal gradient G x and vertical gradient G y , and then calculate the gradient amplitude direction

[0009] For each pixel in the vehicle 2D image, the gradient magnitude is compared with the gradient magnitude of the adjacent pixel along the gradient direction. If the gradient magnitude of the pixel is not a local maximum, it is set to 0.

[0010] Set the threshold interval [T l ,T h ], if the pixel in the gradient magnitude image is >T h , then the pixel point is defined as a strong edge point;

[0011] If the pixel point in the gradient magnitude image is less than T l , then the pixel point is suppressed; between T l and T h If the pixels between them are connected to strong edge points, they are retained, otherwise they are suppressed.

[0012] The beneficial effect of the above further scheme is that compared with processing all the pixel information of the entire two-dimensional image of the vehicle, the edge information only contains the part of the image where the pixels change dramatically, and the amount of data is greatly reduced. Therefore, by extracting the edge information, the license plate area can be analyzed more quickly without processing a large amount of irrelevant background pixel data, which helps to improve the real-time performance of license plate recognition and meet the needs of the gate system to quickly determine whether the vehicle can pass.

[0013] On the basis of the above technical solution, the present invention can also be improved as follows.

[0014] As a further improvement of this technical solution, the intelligent license plate feature extraction module calculates multiple edge information sizes and sets a license plate size threshold area. If the minimum license plate size threshold is less than the edge information size and less than the maximum license plate size threshold, the edge information is the license plate area edge information.

[0015] As a further improvement of this technical solution, the projection method in the intelligent license plate feature extraction module identifies the peak area corresponding to the character and the trough area corresponding to the blank interval by analyzing the pixel accumulation value curve in the horizontal and vertical directions. Specifically: perceive each pixel point in the two-dimensional image of the vehicle, accumulate the pixel values ​​by row and by column, and obtain the horizontal projection curve respectively. Vertical projection curve Where I(x,y) is the pixel value of the vehicle's two-dimensional image at coordinate (x,y), H and W are the height and width of the vehicle's two-dimensional image respectively;

[0016] Set the threshold T and projection value, the projection value represents H(y) and V(x); if the continuous interval of projection value P(i)≥T [i start ,i end ] is the peak area, that is, the license plate characters; the continuous interval of projection value P(i)<T is the trough area, that is, the blank space between characters.

[0017] The beneficial effect of this further solution is that by analyzing the license plate area from multiple edge information, it avoids blindly searching the entire two-dimensional image of the vehicle, significantly reducing the computational workload of subsequent processing and improving response speed. For example, in scenes containing multiple vehicles or complex backgrounds, quickly locating the license plate area allows the system to focus on the key target;

[0018] In addition, the accurately extracted license plate character recognition results can be directly compared with the barrier system database to achieve automatic release or interception without human intervention, improving traffic efficiency and reducing labor costs.

[0019] On the basis of the above technical solution, the present invention can also be improved as follows.

[0020] As a further improvement of the present technical solution, the working principle of the data fusion technology in the multi-sensor data fusion module is as follows: perceiving the three-dimensional image of the vehicle constructed by the lidar, establishing a connection between the coordinates of the three-dimensional image of the vehicle and the coordinates of the two-dimensional image of the vehicle collected by the camera, and fusing the two-dimensional image of the vehicle and the coordinates of the three-dimensional image of the vehicle: perceiving external parameters and internal parameters;

[0021] The extrinsic parameters describe the relative position and attitude relationship between the LiDAR coordinate system and the camera coordinate system, including rotation and translation. The extrinsic parameter matrix is ​​used to transform the 3D points in the LiDAR coordinate system to the camera coordinate system.

[0022] The intrinsic parameters describe the imaging geometry of the camera. After converting the 3D points to the camera coordinate system, the intrinsic parameters are used to project the 3D points onto the vehicle's 2D image to obtain the corresponding pixel coordinates, thereby realizing the fusion of the vehicle's 2D image and 3D image coordinates.

[0023] The beneficial effect of the above further solution is that at night or in underground garages, the camera image may be blurred, but the lidar can still accurately detect the vehicle outline. By fusing the image, even when the quality of the vehicle's two-dimensional image is poor, the intelligent license plate feature extraction module can continuously and stably complete the license plate character extraction even in environments that affect the camera acquisition quality, avoiding system failures due to environmental factors and improving the reliability and stability of the intelligent license plate feature extraction module.

[0024] On the basis of the above technical solution, the present invention can also be improved as follows.

[0025] As a further improvement of this technical solution, the corresponding calculation formula of the data fusion technology in the multi-sensor data fusion module is: There is a point P in the sensing laser radar coordinate system. L =(x L ,y L ,z L ), there is a point P in the camera coordinate system C =(x L ,y L ,z L ), the pixel coordinates on the vehicle two-dimensional image are p = (u, v, 1); the perception external parameter matrix H ext And the internal parameter matrix K, where the external parameter matrix H ext It consists of a rotation matrix R and a translation vector T;

[0026] External parameter conversion: transform point P in the laser radar coordinate system L Convert to camera coordinate system P C , the calculation formula is P C =H ext P L ,in Specifically expanded as follows:

[0027] Intrinsic parameter projection: point P in the camera coordinate system C Projected onto the vehicle's two-dimensional image to obtain the pixel coordinate p, the calculation formula is: The internal parameter matrix f x 、f y are the focal lengths in the x and y directions respectively, and (u0, v0) is the coordinate of the optical center.

[0028] As a further improvement of this technical solution, the working steps of the vehicle positioning method in the adaptive visual adjustment module are as follows:

[0029] Step 1: Detect the coordinate points of the vehicle outline in the fused image Where N is the number of contour points, (xi,yi) is the horizontal and vertical coordinates of the i-th contour point; determine the center coordinates of the vehicle in the fused image: the horizontal center coordinates of the vehicle in the fused image are The vertical center coordinate is

[0030] Step 2: Set the vehicle to be in a horizontal state in the 2D image, which is the ideal state of the vehicle: The horizontal coordinate of the vehicle in the fused image in the ideal state is: x ideal =W / 2; The vertical coordinate of the vehicle in the ideal state in the fused image is: y ideal =H / 2;

[0031] Step 3: Calculate the deviation between the center coordinates of the vehicle and the coordinates of the vehicle in the fused image under ideal conditions: Δx = x c -x ideal , Δy=y c -y ideal ;

[0032] Step 4: Calculate the horizontal rotation angle, vertical pitch angle, and twist angle that need to be adjusted for the camera: the installation height of the perception camera is h, the horizontal distance between the vehicle and the camera in the fused image is L, and the ratio of the image pixels to the actual distance is s;

[0033] The camera needs to adjust the horizontal rotation angle to The camera needs to be adjusted to a vertical pitch angle of The camera needs to be adjusted to a torsion angle of γ = -θ, where θ is the angular deviation of the vehicle center coordinate in the fused image.

[0034] As a further improvement of this technical solution, the calculation formula of the angle deviation θ is: respectively call out the coordinates of the two feature points (x1, y1) and (x2, y2) of the vehicle outline in the fused image, and calculate the slope of the two coordinates Then the angular deviation is θ=arctan(k).

[0035] As a further improvement of the present technical solution, the angle deviation θ represents the degree of deviation of the vehicle from the horizontal direction. When the vehicle is restored to the horizontal direction, the camera needs to be twisted in the opposite direction by the same angle to offset the tilt of the vehicle. Therefore, the twisting angle is γ = -θ.

[0036] As a further improvement of this technical solution, the adaptive visual adjustment module further adjusts the focal length f according to the distance L between the vehicle and the camera: Where f0 is the initial focal length and L0 is the reference distance.

[0037] The beneficial effect of the above further solution is that, based on the fused image and according to the vehicle distance measured by the lidar, the camera focal length is dynamically adjusted to ensure the size stability of the license plate in the two-dimensional image of the vehicle and improve the accuracy of license plate character extraction, thereby forming a complete closed loop of "detection-analysis-adjustment-redetection", automatically adapting to different vehicle models, driving paths and ambient lighting, improving the gradient calculation accuracy of the Sobel operator in the intelligent license plate feature extraction module, reducing the misjudgment of non-maximum suppression, and using the three-dimensional point cloud information fused by the multi-sensor data fusion module to accurately calculate the true posture of the vehicle, guide the angle adjustment of the adaptive visual adjustment module, and form an efficient link of "data fusion-posture analysis-parameter adjustment";

[0038] Through real-time perception, dynamic adjustment and intelligent optimization of the adaptive visual adjustment module, the key problem of license plate recognition in complex scenarios is solved, and the adaptability, reliability and intelligence of the intelligent license plate feature extraction module are significantly improved, providing core technical support for the field of intelligent transportation.

[0039] In addition to the above-described objects, features and advantages, the present invention has other objects, features and advantages. The present invention will be further described in detail below with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] Figure 1 This is the overall module principle diagram of the present invention;

[0041] Figure 2 This is a flow chart of the working principle of the multi-sensor data fusion module of the present invention;

[0042] Figure 3 This is a flow chart of the working principle of the adaptive visual adjustment module of the present invention.

[0043] The meaning of each number in the figure is:

[0044] 100. Intelligent license plate feature extraction module; 200. Multi-sensor data fusion module; 300. Adaptive visual adjustment module. DETAILED DESCRIPTION

[0045] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0046] Example 1, reference Figure 1-Figure 3 As shown, an AI barrier gate monitoring system combining radar and image dual analysis includes an intelligent license plate feature extraction module 100, a multi-sensor data fusion module 200, and an adaptive visual adjustment module 300. The intelligent license plate feature extraction module 100 senses a two-dimensional vehicle image captured by a camera, uses an edge detection algorithm to extract multiple edge information in the two-dimensional vehicle image, calculates multiple edge information sizes, and sets a license plate size threshold area. If the minimum license plate size threshold is less than the edge information size and less than the maximum license plate size threshold, the edge information is the edge information of the license plate area. A projection method is then used to extract the license plate characters from the edge information of the license plate area.

[0047] The working principle of the edge detection algorithm is as follows: a Gaussian filter is used to smooth the two-dimensional image of the vehicle to reduce the influence of noise. The template of the Gaussian filter is a two-dimensional Gaussian function, and its formula is: (x, y) is the coordinate in the template, σ is the standard deviation of the Gaussian distribution, and the smoothed image is obtained by convolving the vehicle's two-dimensional image with the Gaussian template;

[0048] The Sobel operator is used to calculate the horizontal and vertical gradients of the smoothed vehicle two-dimensional image, where The vehicle 2D image is respectively x and S y Convolution operation to obtain the horizontal gradient G x and vertical gradient G y , and then calculate the gradient amplitude direction

[0049] For each pixel in the vehicle 2D image, the gradient magnitude is compared with the gradient magnitude of the adjacent pixel along the gradient direction. If the gradient magnitude of the pixel is not a local maximum, it is set to 0.

[0050] Set the threshold interval [T l ,T h ], if the pixel in the gradient magnitude image is >T h , then the pixel point is defined as a strong edge point;

[0051] If the pixel point in the gradient magnitude image is less than T l , then the pixel point is suppressed; between T l and T h If the pixels between them are connected to strong edge points, they are retained, otherwise they are suppressed.

[0052] The projection method in the intelligent license plate feature extraction module 100 calculates the distribution of pixel values ​​in the horizontal and vertical directions of the image, extracts the license plate characters and outputs them to the barrier gate recognition system. After the barrier gate recognition system receives the license plate characters, it compares them with the information in the barrier gate recognition system. If the match is successful, the barrier gate recognition system will automatically control the barrier gate to open, realizing fast and automatic passage of vehicles without human intervention, thereby improving passage efficiency and saving time and labor costs.

[0053] The projection method analyzes the pixel accumulation curves in the horizontal and vertical directions (i.e., projection curves) to identify the peak areas corresponding to the characters and the valley areas corresponding to the blank intervals. Specifically, it senses each pixel point in the two-dimensional image of the vehicle, accumulates the pixel values ​​by row and by column, and obtains the horizontal projection curves respectively. Vertical projection curve Where I(x,y) is the pixel value of the vehicle's two-dimensional image at coordinate (x,y), H and W are the height and width of the vehicle's two-dimensional image respectively;

[0054] Set the threshold T and projection value, the projection value represents H(y) and V(x); if the continuous interval of projection value P(i)≥T [i start ,i end ] is the peak area, that is, the license plate characters; the continuous interval of projection value P(i)<T is the trough area, that is, the blank space between characters.

[0055] Since license plate characters are continuous and spaced in the horizontal and vertical directions, the accumulated pixel values ​​of the license plate character part are high, forming peaks, and the accumulated pixel values ​​of the blank spaces between the characters are low, forming troughs. According to the distribution of peaks and troughs, the position range of the characters in the horizontal and vertical directions is determined, and the continuous peak areas are divided into license plate characters.

[0056] In the second embodiment, based on the first embodiment, in order to avoid the situation in which the two-dimensional image of the vehicle collected in the low-light environment cannot clearly present the license plate and vehicle features, resulting in the intelligent license plate feature extraction module 100 being unable to recognize the edge information of the license plate area and extract the license plate characters, reference is made to Figure 2 and Figure 3As shown, this embodiment differs from the first embodiment in that the multi-sensor data fusion module 200 senses the three-dimensional vehicle image constructed by the laser radar, uses data fusion technology to fuse the two-dimensional vehicle image and the three-dimensional vehicle image, and defines it as a fused image. This ensures that license plate character extraction can be completed continuously and stably even in environments that affect the camera acquisition quality, avoids system failure due to environmental factors, and improves the reliability and stability of the intelligent license plate feature extraction module 100.

[0057] The working principle of the data fusion technology in the multi-sensor data fusion module 200 is as follows: establish a connection between the coordinates of the vehicle's 3D image and the coordinates of the vehicle's 2D image captured by the camera, fuse the vehicle's 2D image and the vehicle's 3D image coordinates; perceive external parameters and internal parameters, and perform coordinate conversion based on the external parameters and internal parameters. The specific working principle is as follows:

[0058] The extrinsic parameters describe the relative position and attitude relationship between the LiDAR coordinate system and the camera coordinate system, including rotation and translation. The extrinsic parameter matrix is ​​used to transform the 3D points in the LiDAR coordinate system to the camera coordinate system.

[0059] Intrinsic parameters describe the camera's imaging geometry, such as focal length and optical center position. After converting a 3D point to the camera coordinate system, the intrinsic parameters are used to project the 3D point onto the vehicle's 2D image to obtain the corresponding pixel coordinates, thereby achieving the fusion of the vehicle's 2D image and 3D image coordinates.

[0060] The corresponding calculation formula of data fusion technology is:

[0061] There is a point P in the laser radar coordinate system L =(x L ,y L ,z L )(homogeneous coordinate representation), there is a point P in the camera coordinate system C =(x L ,y L ,z L ), the pixel coordinates on the vehicle two-dimensional image are p = (u, v, 1); the perception external parameter matrix H ext And the internal parameter matrix K, where the external parameter matrix H ext It consists of a rotation matrix R and a translation vector T;

[0062] External parameter conversion: transform point P in the laser radar coordinate system L Convert to camera coordinate system P C , the calculation formula is P C =H ext P L ,in Specifically expanded as follows:

[0063] Intrinsic parameter projection: point P in the camera coordinate system C Projected onto the vehicle's two-dimensional image to obtain the pixel coordinate p, the calculation formula is: The internal parameter matrix f x 、f y are the focal lengths in the x and y directions respectively, and (u0, v0) is the coordinate of the optical center.

[0064] In a third embodiment, in a barrier gate system, cameras and lidar are typically installed in locations that clearly cover vehicle entrances and exits. The lidar constructs a sector-shaped measurement area centered on itself to accurately measure the distance to vehicles approaching the barrier gate. Once the lidar detects a vehicle entering its monitoring range, the camera begins continuously capturing a two-dimensional image of the vehicle.

[0065] However, in actual scenarios, the path of vehicles before entering the gate is highly uncertain. Some vehicles will drive in diagonally, approach from the side, or deviate from the normal route to avoid other vehicles. When the driving path changes, the license plate position in the two-dimensional image captured by the camera will be offset due to the fixed viewing angle of the camera, causing it to deviate from the center area of ​​the vehicle's two-dimensional image, resulting in distortion of the license plate information and missing edges in the vehicle's two-dimensional image. Figure 3 As shown, the present embodiment differs from the above-mentioned embodiment 1 in that: the adaptive visual adjustment module 300 senses that the intelligent license plate feature extraction module 100 cannot recognize the edge information signal of the license plate area; in order to prevent the two-dimensional vehicle image captured by the camera from problems such as license plate position offset and image distortion caused by the variable vehicle driving path, the vehicle positioning method is used to analyze the camera angle that needs to be adjusted;

[0066] Vehicle localization method: Perceive the coordinate points of the vehicle outline in the fused image Where N is the number of contour points, (x i ,y i ) are the horizontal and vertical coordinates of the i-th contour point; determine the center coordinates of the vehicle in the fused image: the horizontal center coordinates of the vehicle in the fused image are The vertical center coordinate is The vehicle outline is composed of multiple points. The calculated center coordinates can represent the center of the vehicle in the fused image, providing a benchmark for subsequent analysis of vehicle position deviation.

[0067] Assume that the vehicle is in a horizontal state captured by the vehicle 2D image camera, which is the ideal state of the vehicle: The horizontal coordinate of the vehicle in the fusion image in the ideal state is: ideal =W / 2 (W is the width of the vehicle's two-dimensional image); the vertical coordinate of the vehicle in the fused image under ideal conditions is: ideal=H / 2 (H is the height of the vehicle two-dimensional image). By defining the ideal state, it is convenient to compare the actual vehicle coordinates and quantify the position deviation, so as to adjust the camera parameters in a targeted manner, so that the vehicle is in a horizontal state in the vehicle two-dimensional image. When in a horizontal state, the license plate characters in the vehicle two-dimensional image present a standard rectangular arrangement with clear outlines and regular shapes. Therefore, when the multi-sensor data fusion module 200 fuses the vehicle two-dimensional image and the vehicle three-dimensional image, it can avoid the interference caused by the deformation of the license plate characters, and further enable the intelligent license plate feature extraction module 100 to extract the license plate characters in the fused image, thereby reducing the recognition error caused by the deformation of the characters.

[0068] Further calculate the center coordinates of the vehicle and the deviation of the coordinates of the vehicle in the fused image under ideal conditions: Δx = x c -x ideal , Δy=y c -y ideal ;

[0069] Based on the coordinate deviation and geometric relationship, calculate the horizontal rotation angle, vertical pitch angle, and torsion angle that the camera needs to adjust to make the vehicle reach the ideal state in the fused image:

[0070] The installation height of the perception camera is h, the horizontal distance between the vehicle and the camera in the fused image is L, and the ratio of image pixels to actual distance is s;

[0071] The camera needs to be adjusted to a horizontal rotation angle of The camera needs to be adjusted to a vertical pitch angle of The camera needs to be adjusted to a torsion angle of γ = -θ, where θ is the angular deviation of the vehicle's center coordinates in the fused image. The horizontal rotation angle, vertical pitch angle, and torsion angle are calculated based on the coordinate deviation and geometric relationship, which can guide the camera to adjust its posture so that the vehicle reaches the ideal horizontal and vertical position without tilt in the fused image.

[0072] The calculation formula of the angle deviation θ is as follows: call out the coordinates of the two feature points (x1, y1) and (x2, y2) of the vehicle outline in the fusion image ((x1, y1) and (x2, y2) are the coordinates of the leftmost and rightmost sides of the vehicle body respectively), and calculate the slope of the two coordinates. The angle deviation is then θ = arctan(k). Since the angle deviation θ represents the degree of deviation of the vehicle from the horizontal direction, to restore the vehicle to level, the camera needs to be twisted in the opposite direction by the same angle to offset the vehicle's tilt. Therefore, the twisting angle is γ = -θ.

[0073] In order to keep the size of the vehicle in the two-dimensional image constant, the adaptive visual adjustment module 300 adjusts the focal length f according to the distance L between the vehicle and the camera: Where f0 is the initial focal length and L0 is the reference distance.

[0074] The above shows and describes the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The above embodiments and descriptions are merely preferred examples of the present invention and are not intended to limit the present invention. Various changes and improvements may be made to the present invention without departing from the spirit and scope of the present invention. Such changes and improvements fall within the scope of the present invention. The scope of protection claimed in the present invention is defined by the appended claims and their equivalents.

Claims

1. An AI barrier gate monitoring system combining radar and image dual analysis, characterized by: The system comprises an intelligent license plate feature extraction module (100), a multi-sensor data fusion module (200) and an adaptive visual adjustment module (300), wherein: The intelligent license plate feature extraction module (100) uses an edge detection algorithm to extract multiple edge information in the two-dimensional vehicle image, distinguishes the edge information of the license plate area, and uses a projection method to extract the license plate characters in the edge information of the license plate area; the multi-sensor data fusion module (200) uses a data fusion technology to fuse the two-dimensional vehicle image and the three-dimensional vehicle image, and defines the fused image; the adaptive visual adjustment module (300) senses that the intelligent license plate feature extraction module (100) cannot recognize the edge information signal of the license plate area, and uses a vehicle positioning method to analyze whether the camera needs to adjust the angle, and the vehicle positioning method is analyzed through the fused image.

2. The AI ​​barrier gate monitoring system with radar and image dual analysis according to claim 1 is characterized by: The working principle of the edge detection algorithm in the intelligent license plate feature extraction module (100) is as follows: The two-dimensional image of the vehicle collected by the perception camera is smoothed by a Gaussian filter. The template of the Gaussian filter is a two-dimensional Gaussian function, and its formula is: (x, y) is the coordinate in the template, σ is the standard deviation of the Gaussian distribution, and the smoothed image is obtained by convolving the vehicle's two-dimensional image with the Gaussian template; The Sobel operator is used to calculate the horizontal and vertical gradients of the smoothed vehicle two-dimensional image, where The vehicle 2D image is respectively x and S y Convolution operation to obtain the horizontal gradient G x and vertical gradient G y , and then calculate the gradient amplitude direction For each pixel in the vehicle 2D image, the gradient magnitude is compared with the gradient magnitude of the adjacent pixel along the gradient direction. If the gradient magnitude of the pixel is not a local maximum, it is set to 0. Set the threshold interval [T l ,T h ], if the pixel in the gradient magnitude image is >T h , then the pixel point is defined as a strong edge point; If the pixel point in the gradient magnitude image is less than T l , then the pixel point is suppressed; between T l and T h If the pixels between them are connected to strong edge points, they are retained, otherwise they are suppressed.

3. The AI ​​barrier gate monitoring system with radar and image dual analysis according to claim 2 is characterized by: The intelligent license plate feature extraction module (100) calculates multiple edge information sizes and sets a license plate size threshold area. If the minimum license plate size threshold is less than the edge information size and less than the maximum license plate size threshold, the edge information is the license plate area edge information.

4. The AI ​​barrier gate monitoring system with radar and image dual analysis according to claim 3 is characterized by: The projection method in the intelligent license plate feature extraction module (100) identifies the peak area corresponding to the character and the trough area corresponding to the blank interval by analyzing the pixel accumulation value curves in the horizontal and vertical directions. Specifically, each pixel point in the two-dimensional image of the vehicle is sensed, and the pixel values ​​are accumulated by row and by column to obtain the horizontal projection curves. Vertical projection curve Where I(x,y) is the pixel value of the vehicle's two-dimensional image at coordinate (x,y), H and W are the height and width of the vehicle's two-dimensional image respectively; Set the threshold T and projection value, the projection value represents H(y) and V(x); if the continuous interval of projection value P(i)≥T [i start ,i end ] is the peak area, that is, the license plate characters; the continuous interval of projection value P(i)<T is the trough area, that is, the blank space between characters.

5. The AI ​​barrier gate monitoring system with radar combined with image dual analysis according to claim 1 is characterized by: The working principle of the data fusion technology in the multi-sensor data fusion module (200) is as follows: sensing the three-dimensional image of the vehicle constructed by the laser radar, establishing a connection between the coordinates of the three-dimensional image of the vehicle and the coordinates of the two-dimensional image of the vehicle collected by the camera, fusing the two-dimensional image of the vehicle and the coordinates of the three-dimensional image of the vehicle: sensing external parameters and internal parameters; The external parameters describe the relative position and attitude relationship between the lidar coordinate system and the camera coordinate system, including rotation and translation; The 3D points in the LiDAR coordinate system are converted to the camera coordinate system through the external parameter matrix. The intrinsic parameters describe the imaging geometry of the camera. After converting the 3D points to the camera coordinate system, the intrinsic parameters are used to project the 3D points onto the vehicle's 2D image to obtain the corresponding pixel coordinates, thereby realizing the fusion of the vehicle's 2D image and 3D image coordinates.

6. The AI ​​barrier gate monitoring system with radar combined with image dual analysis according to claim 5 is characterized by: The corresponding calculation formula of the data fusion technology in the multi-sensor data fusion module (200) is: There is a point P in the sensing laser radar coordinate system L =(x L ,y L ,z L ), there is a point P in the camera coordinate system C =(x L ,y L ,z L ), the pixel coordinates on the vehicle two-dimensional image are p = (u, v, 1); the perception external parameter matrix H ext And the internal parameter matrix K, where the external parameter matrix H ext It consists of a rotation matrix R and a translation vector T; External parameter conversion: transform point P in the laser radar coordinate system L Convert to camera coordinate system P C , the calculation formula is P C =H ext P L ,in Specifically expanded as follows: Intrinsic parameter projection: point P in the camera coordinate system C Projected onto the vehicle's two-dimensional image to obtain the pixel coordinate p, the calculation formula is: The internal parameter matrix f x 、f y are the focal lengths in the x and y directions respectively, and (u0, v0) is the coordinate of the optical center.

7. The AI ​​barrier gate monitoring system with radar combined with image dual analysis according to claim 6 is characterized by: The working steps of the vehicle positioning method in the adaptive visual adjustment module (300) are as follows: Step 1: Detect the coordinate points of the vehicle outline in the fused image Where N is the number of contour points, (x i ,y i ) are the horizontal and vertical coordinates of the i-th contour point; Determine the center coordinates of the vehicle in the fused image: The horizontal center coordinates of the vehicle in the fused image are The vertical center coordinate is Step 2: Set the vehicle to be in a horizontal state in the 2D image, which is the ideal state of the vehicle: The horizontal coordinate of the vehicle in the fused image in the ideal state is: x ideal =W / 2; The vertical coordinate of the vehicle in the ideal state in the fused image is: y ideal =H / 2; Step 3: Calculate the deviation between the center coordinates of the vehicle and the coordinates of the vehicle in the fused image under ideal conditions: Δx = x c -x ideal , Δy=y c -y ideal ; Step 4: Calculate the horizontal rotation angle, vertical pitch angle, and twist angle that need to be adjusted for the camera: the installation height of the perception camera is h, the horizontal distance between the vehicle and the camera in the fused image is L, and the ratio of the image pixels to the actual distance is s; The camera needs to adjust the horizontal rotation angle to The camera needs to be adjusted to a vertical pitch angle of The camera needs to be adjusted to a torsion angle of γ = -θ, where θ is the angular deviation of the vehicle center coordinate in the fused image.

8. The AI ​​barrier gate monitoring system with radar combined with image dual analysis according to claim 5 is characterized by: The calculation formula of the angle deviation θ is: respectively call out the coordinates of the two feature points (x1, y1) and (x2, y2) of the vehicle outline in the fusion image, and calculate the slope of the two coordinates Then the angular deviation is θ=arctan(k).

9. The AI ​​barrier gate monitoring system with radar combined with image dual analysis according to claim 8 is characterized by: The angle deviation θ represents the degree of deviation of the vehicle from the horizontal direction. When the vehicle is restored to the horizontal direction, the camera needs to be twisted in the opposite direction by the same angle to offset the tilt of the vehicle. Therefore, the twisting angle is γ=-θ.

10. The AI ​​barrier gate monitoring system with radar combined with image dual analysis according to claim 1 is characterized by: The adaptive visual adjustment module (300) further adjusts the focal length f according to the distance L between the vehicle and the camera: Where f0 is the initial focal length and L0 is the reference distance.

Images