Vehicle detection method and system based on monocular vision and deep learning

Abstract

The present invention provides a vehicle detection method and system based on monocular vision and deep learning, which obtains highway video frame image data, optimizes camera parameters of back-projection from two-dimensional space to three-dimensional space by using an evolutionary algorithm, performs camera calibration, and obtains calibrated camera parameters; performs vehicle detection by using a neural network to obtain vehicle type and a two-dimensional bounding box, and uses the calibrated camera parameters to trace the foot points of the two-dimensional bounding box of the vehicle back to three-dimensional space to obtain vehicle coordinate points in a world coordinate system; calculates vehicle distance by using Euclidean distance according to the vehicle coordinate points in the world coordinate system and the zero point coordinates established by the camera, calculates vehicle speed according to the vehicle distance based on a sliding time window algorithm, and determines whether the speed variance is less than a given threshold value, and determines that a vehicle with too small a speed is a stopped vehicle; realizes rapid detection of vehicle distance and vehicle speed, and improves the real-time performance of detection.

Description

Technical Field

[0001] The present invention relates to the field of vehicle detection technology, and in particular to a vehicle detection method and system based on monocular vision and deep learning. Background Art

[0002] The statements in this section merely provide background art related to the present invention and do not necessarily constitute prior art.

[0003] The safety of vehicles on highways has always been a focus of social attention. The initial highway monitoring system required personnel to view videos in the monitoring background. After a traffic accident occurred, personnel reported it to the monitoring background to learn about the accident and make emergency responses. It was impossible to predict and warn of primary and secondary accidents, and it was impossible to avoid primary and secondary accidents. With the rise and development of new technologies such as big data and artificial intelligence, intelligent traffic management systems have become an important means of traffic management in my country.

[0004] The intelligent transportation system of highways can provide assistance for the safe driving of vehicles, monitor road information and evaluate the road environment. In recent years, some highways have begun to install intelligent management systems, which provide traffic departments with road commuting information and vehicle speed, distance, etc. through sensor detection. Although a single sensor or multi-sensor fusion vehicle detection system can detect vehicles and meet people's requirements for diversified system detection parameters, the cost increases significantly with the increase in the number of sensors, which is not the best choice under the condition of cost control. Computer vision is a branch of artificial intelligence, which uses machines to replace human eyes to make measurements and judgments. It is usually used in assisted driving of vehicle-road collaboration, detection of vehicles on the road, distance and speed identification, using computer vision technology, identifying the vehicle in front through the on-board computer and detection system, and obtaining the accurate status of the target vehicle, which can be used as assisted driving information of the vehicle to ensure the safe driving of the vehicle. Assisted driving, that is, intelligent driving technology, only provides auxiliary functions. Through environmental perception technology, it can reliably detect vehicles, pedestrians and obstacles around the driving vehicle, and provide the driver with driving environment and warning information to achieve the purpose of safe driving.

[0005] The inventors found that as the quality of video images increases, the existing 5G network infrastructure does not have comprehensive coverage, and the 4G network bandwidth cannot meet the truly real-time transmission of high-quality video images. The computer receives the data sent back from the scene and then performs further computer vision processing to measure the vehicle speed and distance, which makes it impossible to guarantee the real-time performance of vehicle speed and distance measurement. Summary of the invention

[0006] In order to address the deficiencies in the prior art, the present invention provides a vehicle detection method and system based on monocular vision and deep learning, which achieves rapid detection of vehicle distance and vehicle speed and improves the real-time performance of detection.

[0007] In order to achieve the above object, the present invention adopts the following technical solution:

[0008] A first aspect of the present invention provides a vehicle detection method based on monocular vision and deep learning.

[0009] A vehicle detection method based on monocular vision and deep learning includes the following processes:

[0010] Obtain highway video frame image data, use evolutionary algorithms to optimize the camera parameters of back-projection from two-dimensional space to three-dimensional space, perform camera calibration, and obtain calibrated camera parameters;

[0011] A neural network is used to detect vehicles, obtain the vehicle type and two-dimensional bounding box, and the calibrated camera parameters are used to trace the vehicle's two-dimensional bounding box footpoints back to three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0012] The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera.

[0013] A second aspect of the present invention provides a vehicle detection method based on monocular vision and deep learning.

[0014] A vehicle detection method based on monocular vision and deep learning includes the following processes:

[0015] Obtain highway video frame image data, use evolutionary algorithms to optimize the camera parameters of back-projection from two-dimensional space to three-dimensional space, perform camera calibration, and obtain calibrated camera parameters;

[0016] A neural network is used to detect vehicles, obtain the vehicle type and two-dimensional bounding box, and the calibrated camera parameters are used to trace the vehicle's two-dimensional bounding box footpoints back to three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0017] The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0018] Based on the obtained vehicle distance, the long-term vehicle appearance changes are learned through a histogram-based adaptive appearance model to obtain the trajectory of the vehicle to be detected, and the clustering method is used to associate the vehicle trajectory to a longer trajectory.

[0019] A third aspect of the present invention provides a vehicle detection method based on monocular vision and deep learning.

[0020] A vehicle detection method based on monocular vision and deep learning includes the following processes:

[0021] Obtain highway video frame image data, use evolutionary algorithms to optimize the camera parameters of back-projection from two-dimensional space to three-dimensional space, perform camera calibration, and obtain calibrated camera parameters;

[0022] A neural network is used to detect vehicles, obtain the vehicle type and two-dimensional bounding box, and the calibrated camera parameters are used to trace the vehicle's two-dimensional bounding box footpoints back to three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0023] The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0024] The vehicle speed is calculated according to the vehicle distance based on the sliding time window algorithm, and it is determined whether the speed variance is less than a given threshold. If the speed is too small, the vehicle is judged to be a stopped vehicle.

[0025] A fourth aspect of the present invention provides a vehicle detection system based on monocular vision and deep learning.

[0026] A vehicle detection system based on monocular vision and deep learning, comprising:

[0027] The data acquisition module is configured to: acquire highway video frame image data, optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space using an evolutionary algorithm, perform camera calibration, and obtain calibrated camera parameters;

[0028] The vehicle detection module is configured to: use a neural network to detect the vehicle, obtain the vehicle type and the two-dimensional bounding box, and use the calibrated camera parameters to trace the foot points of the vehicle's two-dimensional bounding box back to the three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0029] The distance detection module is configured to calculate the vehicle distance using the Euclidean distance according to the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera.

[0030] A fifth aspect of the present invention provides a vehicle detection system based on monocular vision and deep learning.

[0031] A vehicle detection system based on monocular vision and deep learning, comprising:

[0032] The data acquisition module is configured to: acquire highway video frame image data, optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space using an evolutionary algorithm, perform camera calibration, and obtain calibrated camera parameters;

[0033] The vehicle detection module is configured to: use a neural network to detect the vehicle, obtain the vehicle type and the two-dimensional bounding box, and use the calibrated camera parameters to trace the foot points of the vehicle's two-dimensional bounding box back to the three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0034] The distance detection module is configured to: calculate the vehicle distance using the Euclidean distance according to the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0035] The trajectory generation module is configured to: based on the obtained vehicle distance, learn the long-term vehicle appearance change through a histogram-based adaptive appearance model to obtain the trajectory of the vehicle to be detected, and use a clustering method to associate the vehicle trajectory to a longer trajectory.

[0036] A sixth aspect of the present invention provides a vehicle detection system based on monocular vision and deep learning.

[0037] A vehicle detection system based on monocular vision and deep learning, comprising:

[0038] The data acquisition module is configured to: acquire highway video frame image data, optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space using an evolutionary algorithm, perform camera calibration, and obtain calibrated camera parameters;

[0039] The vehicle detection module is configured to: use a neural network to detect the vehicle, obtain the vehicle type and the two-dimensional bounding box, and use the calibrated camera parameters to trace the foot points of the vehicle's two-dimensional bounding box back to the three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0040] The distance detection module is configured to: calculate the vehicle distance using the Euclidean distance according to the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0041] The speed detection module is configured to: calculate the vehicle speed according to the vehicle distance based on the sliding time window algorithm, determine whether the speed variance is less than a given threshold, and if the vehicle has a speed that is too small, determine it to be a stopped vehicle.

[0042] The seventh aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the vehicle detection method based on monocular vision and deep learning as described in the first aspect, the second aspect or the third aspect of the present invention.

[0043] An eighth aspect of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the vehicle detection method based on monocular vision and deep learning as described in the first aspect, the second aspect, or the third aspect of the present invention are implemented.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] 1. The vehicle detection method and system based on monocular vision and deep learning described in the present invention calculate the vehicle distance using Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera, thereby realizing rapid detection of vehicle distance and vehicle speed and improving the real-time performance of detection.

[0046] 2. The vehicle detection method and system based on monocular vision and deep learning described in the present invention use an optimization algorithm to minimize the reprojection error during camera calibration, thereby improving the accuracy of camera calibration and further improving the efficiency and accuracy of video image acquisition.

[0047] 3. The vehicle detection method and system based on monocular vision and deep learning described in the present invention adopts a histogram-based adaptive appearance model to calculate the appearance change loss, thereby achieving more accurate real-time tracking of vehicles.

[0048] Advantages of additional aspects of the present invention will be given in part in the following description, and in part will become obvious from the following description, or will be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings in the specification, which constitute a part of the present invention, are used to provide a further understanding of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute improper limitations on the present invention.

[0050] Figure 1 A flowchart of a vehicle detection method based on monocular vision and deep learning provided in Example 1 of the present invention.

[0051] Figure 2 A flowchart of a vehicle detection method based on monocular vision and deep learning provided in Example 2 of the present invention.

[0052] Figure 3 A flowchart of a vehicle detection method based on monocular vision and deep learning provided in Example 3 of the present invention. DETAILED DESCRIPTION

[0053] The present invention will be further described below in conjunction with the accompanying drawings and embodiments.

[0054] It should be noted that the following detailed descriptions are exemplary and are intended to provide further explanation of the present invention. Unless otherwise specified, all technical and scientific terms used herein have the same meanings as those commonly understood by those skilled in the art to which the present invention belongs.

[0055] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit exemplary embodiments according to the present invention. As used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form. In addition, it should be understood that when the terms "comprising" and / or "including" are used in this specification, it indicates the presence of features, steps, operations, devices, components and / or combinations thereof.

[0056] In the absence of conflict, the embodiments of the present invention and the features of the embodiments may be combined with each other.

[0057] Embodiment 1:

[0058] like Figure 1 As shown, Embodiment 1 of the present invention provides a vehicle detection method based on monocular vision and deep learning, including the following process:

[0059] Obtain highway video monitoring data, use evolutionary algorithms to optimize the camera parameters of back-projection from two-dimensional space to three-dimensional space, calibrate the highway video monitoring camera, and obtain the calibrated camera parameters;

[0060] A neural network is used to detect vehicles, obtain the vehicle type and two-dimensional bounding box, and the calibrated camera parameters are used to trace the vehicle's two-dimensional bounding box footpoints back to three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0061] The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera.

[0062] The camera calibration method is performed in the central processing unit of the front-end device AI chip (the front-end detection device is installed at the location where the vehicle needs to be detected, such as the entrance of a highway bridge or tunnel, an intersection, a road construction site, etc. The front-end device includes an intelligent AI chip, a camera, and a warning device (a flashing light, a warning screen, etc.)):

[0063] For the camera to be calibrated, two pairs of vanishing lines manually marked in the highway video monitoring image and the camera's height range are received. The two pairs of vanishing lines are parallel lines that are orthogonal to each other on the ground plane of three-dimensional space. Usually, a pair of parallel lane lines and a pair of parallel lines orthogonal to them are manually marked.

[0064] According to the two pairs of marked vanishing lines, two vanishing points on the ground plane are derived. The first vanishing point V x and the second vanishing point V y ;

[0065] According to the two vanishing points V x and V yThe camera parameters are calculated based on the camera's height range, and the optimization algorithm is used to optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space to calibrate the highway video monitoring camera.

[0066] For the output camera parameters, including matrix K, matrix R, matrix t and matrix P (matrix P is a 3×4 projection matrix), they represent the intrinsic camera matrix, rotation matrix, translation matrix and projection matrix respectively. In the 3D coordinate system, the ground plane is equivalent to the XY plane, and the Z axis points upward and passes through the camera position.

[0067] In order to calculate the camera parameters more accurately, an optimization algorithm is used to minimize the reprojection error.

[0068] A set of N located on the ground plane ls Line segments, each of which is defined by two endpoints, are marked in 3D as The length of the line segments in 3D is measured in the map, and their Euclidean distance represents the estimated 3D length of the line segment;

[0069] Use the calculated camera parameters to move the 2D endpoints of the line segment (labeled as {p k} and {q k}) Back-project to 3D;

[0070] The absolute difference between the estimate and the ground truth is the reprojection error.

[0071] The optimization objective is defined as minimizing the reprojection error as follows:

[0072]

[0073] Among them, P k and Q l are the estimated endpoints of the selected line segment of the 3D ground plane, and Represents the estimated endpoint of the selected line segment back-projected onto the 3D ground plane, Rng P is the initial range of each camera parameter to be optimized.

[0074] The nonlinear optimization problem in formula (1) is iteratively solved by the distribution estimation algorithm (EDA) until the average value of the estimated probability density function (pdf) is less than the specified threshold ∈ P The iterative process stops, optimizing the camera parameters in the projection matrix P.

[0075] In the first iteration of EDA, the initial population of camera parameters (denoted as M) is generated by uniform distribution. Among M, the selected population with the lowest estimated reprojection error, denoted as N, is selected to fit the probability density function with 12 variables normal distribution, because there are 12 camera parameters to be optimized in P. Then, the next iteration is continued to generate a new initial population and select the best subset to estimate the probability density function again until the mean of the estimated pdf is less than the specified threshold and the iterative process stops.

[0076] Vehicle detection using neural networks in a multi-module collaborative object recognition framework based on deep learning, including:

[0077] The multi-unit collaborative object recognition system based on deep learning includes a video input unit, a video processing subsystem unit, an intelligent video engine unit, a neural network acceleration engine unit, a video graphics subsystem unit and a video output unit that are integrated and work together;

[0078] A video input unit, for receiving real-time video data and storing it in a designated memory area;

[0079] A video processing subsystem unit, used to retrieve the original video data in the memory area and decompose it into basic video data and extended video data; wherein the basic video data maintains the resolution of the original video data; and the resolution of the extended video data matches the neural network model in the neural network acceleration engine unit;

[0080] An intelligent video engine unit, used for converting image frame data in the current extended video data into frame data in an image format that matches the neural network model;

[0081] The neural network acceleration engine unit is used to obtain the frame data after format conversion, and obtain the category and contour four-point coordinate position information of the object through neural network model recognition; the neural network acceleration engine unit includes loading a trained neural network model in a specific format during the initialization operation;

[0082] The video graphics subsystem unit is used to obtain basic video data, and then draw a contour frame of the identified object in the basic video data based on the category of the object and the coordinate position information of the four points of the contour;

[0083] The video output unit is used to output video image data with a contour frame of a recognized object.

[0084] Embodiment 2:

[0085] like Figure 2 As shown, Embodiment 2 of the present invention provides a vehicle detection method based on monocular vision and deep learning, including the following process:

[0086] Obtain highway video monitoring data, use evolutionary algorithms to optimize the camera parameters of back-projection from two-dimensional space to three-dimensional space, calibrate the highway video monitoring camera, and obtain the calibrated camera parameters;

[0087] A neural network is used to detect vehicles, obtain the vehicle type and two-dimensional bounding box, and the calibrated camera parameters are used to trace the vehicle's two-dimensional bounding box footpoints back to three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0088] The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0089] The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0090] According to the fusion of visual and semantic features, based on the obtained vehicle distance, the long-term vehicle appearance changes are learned through a histogram-based adaptive appearance model to obtain the trajectory of the detected vehicle, and the clustering method is used to associate the trajectory to a longer trajectory.

[0091] The specific camera calibration method and the vehicle detection method based on the neural network are shown in Example 1 and will not be described in detail here.

[0092] The clustering method is executed in the central processing unit of the AI ​​chip of the front-end device, and includes:

[0093] The bottom-up clustering of track associations is performed through the loss function, so that the tracks are associated into longer tracks to achieve vehicle tracking. All tracks are iteratively clustered into tracks until convergence. Convergence is guaranteed because the loss decreases or remains unchanged after each selection operation. The loss function consists of motion attributes, time attributes, and appearance attributes.

[0094]

[0095] Among them, n v is the number of vehicles detected in the image, l i is the cluster loss of the ith trajectory,

[0096] The loss functions include trajectory smoothness loss, speed change loss, time interval loss, and appearance change loss;

[0097] l i =λ sm l i，sm +λ vc l i，vc +λ ti l i，ti +λ ac l i，ac

[0098] Among them, n v is the number of vehicles detected in the image, l i is the cluster loss of the ith trajectory, l i，sm is the trajectory smoothness loss, li ，vc is the speed change loss, l i，ti is the time interval loss between adjacent tracks, l i，ac is the appearance change loss, λ sm represents the regularization parameter of the trajectory smoothness loss function, set to 0.2, λ vc represents the regularization parameter of the speed change loss function, set to 8, λ ti The regularization parameter of the time interval loss function between adjacent tracks is set to 25, λ ac represents the regularization parameter of the appearance change loss function and is set to 0.5.

[0099] Smoothness loss is used to measure the smoothness of points belonging to the same trajectory. The points in each trajectory are sorted according to the input timestamp. For each pair of adjacent points in the ith trajectory, the distance between the adjacent points in the connected region is calculated. The smoothness loss of the ith trajectory is the sum of the distances between all points and their adjacent points in the connected region.

[0100] Velocity change loss is a measure of the velocity change between two adjacent tracks through acceleration. If high acceleration is detected in the area of ​​connection, it is unlikely that the two tracks will maintain the same identity. Velocity change loss is calculated as the sum of the maximum accelerations around each end point of the track.

[0101] The time interval loss is the difference between the timestamps of the two endpoints of a trajectory. If there is a long time interval between a pair of adjacent trajectories, they are unlikely to share the same trajectory.

[0102] Reliable vehicle appearance description is key to correctly track vehicle trajectories, especially for nearby vehicles with similar appearance. A histogram-based adaptive appearance model is used to compute the appearance variation loss. In particular, a histogram-based adaptive appearance model is designed to encode long-term appearance variations and enhance robustness. The variation loss is combined with a bottom-up trajectory association clustering strategy.

[0103] The method of calculating the appearance change loss by using the histogram-based adaptive appearance model includes:

[0104] Construct a histogram for each cropped vehicle region within the detected bounding box;

[0105] When the vehicle is occluded by other objects, the occluded area is removed from the target region before constructing the concatenated histogram of visual features;

[0106] Gaussian (kernel) distribution is used to spatially weight the pixel values ​​constructed from the histogram;

[0107] By controlling the pixel position in the visible object area away from the centroid of the visible area, the spatial weight is reduced, thereby suppressing the influence of the background area on vehicle recognition;

[0108] The appearance change loss is the ratio of the histogram vectors of two mismatched appearance models, and the threshold of the histogram distance is 0.3. The histogram distance is measured using the Bhattacharyya distance. The vehicles in the trajectory merge their appearance models together according to the appearance change loss based on the depth information, and the merged appearance model describes the appearance changes of the vehicle along the entire trajectory.

[0109] Embodiment 3:

[0110] like Figure 3 As shown, Embodiment 3 of the present invention provides a vehicle detection method based on monocular vision and deep learning, including the following process:

[0111] Obtain highway video monitoring data, use evolutionary algorithms to optimize the camera parameters of back-projection from two-dimensional space to three-dimensional space, calibrate the highway video monitoring camera, and obtain the calibrated camera parameters;

[0112] A neural network is used to detect vehicles, obtain the vehicle type and two-dimensional bounding box, and the calibrated camera parameters are used to trace the vehicle's two-dimensional bounding box footpoints back to three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0113] The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0114] The vehicle speed is calculated according to the vehicle distance based on the sliding time window algorithm, and it is determined whether the speed variance is less than a given threshold. If the speed is too small, the vehicle is judged to be a stopped vehicle.

[0115] The specific camera calibration method and the vehicle detection method based on the neural network are shown in Example 1 and will not be described in detail here.

[0116] In the vehicle speed calculation, the expected travel time is estimated given the travel distance between two cameras, and for false negatives with too low detection scores, the speed of the closest true positive is applied to them.

[0117] Furthermore, the speeds of all vehicles detected in all frames of all highway videos are estimated. The ground truth speed data of vehicles is collected by in-car tracking of a subset of cars in each video. The ground truth speed data of vehicles is used to form training, validation and test sets.

[0118] Input the training set and validation set into the neural network for training and cross-validation;

[0119] The neural network is optimized through the test set to obtain the trained neural network.

[0120] Furthermore, the estimated speed of the vehicles is evaluated based on the ability to locate these vehicles and predict their speeds. The performance evaluation score of a particular vehicle is calculated as:

[0121] E s =R d ×(1-NRMSE)

[0122] Among them, R d is the detection rate, which is the ratio of the identified highway vehicles to the total number of real highway vehicles. NRMSE is the normalized root mean square error of speed. E s Between 0 and 1, E s The higher the score, the better. In the velocity estimation, the normalized root mean square error of the velocity is small (4.0963mi / h) because the proposed camera calibration scheme can minimize the reprojection error on the ground, making the velocity estimation robust.

[0123] Embodiment 4:

[0124] Embodiment 4 of the present invention provides a vehicle detection system based on monocular vision and deep learning, including:

[0125] The data acquisition module is configured to: acquire highway video frame image data, optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space using an evolutionary algorithm, perform camera calibration, and obtain calibrated camera parameters;

[0126] The vehicle detection module is configured to: use a neural network to detect the vehicle, obtain the vehicle type and the two-dimensional bounding box, and use the calibrated camera parameters to trace the foot points of the vehicle's two-dimensional bounding box back to the three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0127] The distance detection module is configured to calculate the vehicle distance using the Euclidean distance according to the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera.

[0128] The working method of the system is the same as the vehicle detection method based on monocular vision and deep learning provided in Example 1, and will not be repeated here.

[0129] Embodiment 5:

[0130] Embodiment 5 of the present invention provides a vehicle detection system based on monocular vision and deep learning, including:

[0131] The data acquisition module is configured to: acquire highway video frame image data, optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space using an evolutionary algorithm, perform camera calibration, and obtain calibrated camera parameters;

[0132] The vehicle detection module is configured to: use a neural network to detect the vehicle, obtain the vehicle type and the two-dimensional bounding box, and use the calibrated camera parameters to trace the foot points of the vehicle's two-dimensional bounding box back to the three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0133] The distance detection module is configured to: calculate the vehicle distance using the Euclidean distance according to the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0134] The trajectory generation module is configured to: based on the obtained vehicle distance, learn the long-term vehicle appearance change through a histogram-based adaptive appearance model to obtain the trajectory of the vehicle to be detected, and use a clustering method to associate the vehicle trajectory to a longer trajectory.

[0135] The working method of the system is the same as the vehicle detection method based on monocular vision and deep learning provided in Example 2, and will not be repeated here.

[0136] Embodiment 6:

[0137] Embodiment 6 of the present invention provides a vehicle detection system based on monocular vision and deep learning, including:

[0138] The data acquisition module is configured to: acquire highway video frame image data, optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space using an evolutionary algorithm, perform camera calibration, and obtain calibrated camera parameters;

[0139] The vehicle detection module is configured to: use a neural network to detect the vehicle, obtain the vehicle type and the two-dimensional bounding box, and use the calibrated camera parameters to trace the foot points of the vehicle's two-dimensional bounding box back to the three-dimensional space to obtain the vehicle coordinate points in the world coordinate system;

[0140] The distance detection module is configured to: calculate the vehicle distance using the Euclidean distance according to the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera;

[0141] The speed detection module is configured to: calculate the vehicle speed according to the vehicle distance based on the sliding time window algorithm, determine whether the speed variance is less than a given threshold, and if the vehicle has a speed that is too small, determine it to be a stopped vehicle.

[0142] The working method of the system is the same as the vehicle detection method based on monocular vision and deep learning provided in Example 3, and will not be repeated here.

[0143] Embodiment 7:

[0144] Embodiment 7 of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps in the vehicle detection method based on monocular vision and deep learning as described in Embodiment 1, 2 or 3 of the present invention.

[0145] Embodiment 8:

[0146] Embodiment 8 of the present invention provides an electronic device, comprising a memory, a processor, and a program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the vehicle detection method based on monocular vision and deep learning as described in Embodiment 1, 2 or 3 of the present invention are implemented.

[0147] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware. Furthermore, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0148] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0149] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0150] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0151] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0152] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A vehicle detection method based on monocular vision and deep learning, characterized in that: The process includes: Obtain highway video frame image data, use evolutionary algorithms to optimize the camera parameters of back-projection from two-dimensional space to three-dimensional space, perform camera calibration, and obtain calibrated camera parameters; A neural network is used to detect vehicles, obtain the vehicle type and two-dimensional bounding box, and the calibrated camera parameters are used to trace the vehicle's two-dimensional bounding box footpoints back to three-dimensional space to obtain the vehicle coordinate points in the world coordinate system; The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera; Based on the obtained vehicle distance, the long-term vehicle appearance changes are learned through a histogram-based adaptive appearance model to obtain the trajectory of the vehicle to be detected, and the clustering method is used to associate the vehicle trajectory to a longer trajectory; A clustering method is used to associate the vehicle's trajectory into a longer trajectory, including: Bottom-up clustering of tracking trajectories is performed through the loss function to associate trajectories into longer trajectories to achieve vehicle tracking. All trajectories are iteratively clustered into trajectories until convergence; The loss functions include trajectory smoothness loss, speed change loss, time interval loss, and appearance change loss; The appearance change loss is calculated using a histogram-based adaptive appearance model, including: Construct a histogram for each cropped vehicle region within the detected bounding box; When the vehicle is occluded by other objects, the occluded area is removed from the target area before constructing the concatenated histogram of visual features, and the pixel values ​​constructed by the histogram are spatially weighted using Gaussian distribution; The appearance change loss is the ratio of the histogram vectors of two mismatched appearance models. The vehicles in the trajectory merge their appearance models together according to the depth information according to the appearance change loss, and the merged appearance model is used to describe the appearance changes of the vehicle along the entire trajectory.

2. A vehicle detection method based on monocular vision and deep learning, characterized in that: The process includes: Obtain highway video frame image data, use evolutionary algorithms to optimize the camera parameters of back-projection from two-dimensional space to three-dimensional space, perform camera calibration, and obtain calibrated camera parameters; A neural network is used to detect vehicles, obtain the vehicle type and two-dimensional bounding box, and the calibrated camera parameters are used to trace the vehicle's two-dimensional bounding box footpoints back to three-dimensional space to obtain the vehicle coordinate points in the world coordinate system; The vehicle distance is calculated using the Euclidean distance based on the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera; Based on the obtained vehicle distance, the long-term vehicle appearance changes are learned through a histogram-based adaptive appearance model to obtain the trajectory of the vehicle to be detected, and the clustering method is used to associate the vehicle trajectory to a longer trajectory; A clustering method is used to associate the vehicle's trajectory into a longer trajectory, including: Bottom-up clustering of tracking trajectories is performed through the loss function to associate trajectories into longer trajectories to achieve vehicle tracking. All trajectories are iteratively clustered into trajectories until convergence; The loss functions include trajectory smoothness loss, speed change loss, time interval loss, and appearance change loss; The appearance change loss is calculated using a histogram-based adaptive appearance model, including: Construct a histogram for each cropped vehicle region within the detected bounding box; When the vehicle is occluded by other objects, the occluded area is removed from the target area before constructing the concatenated histogram of visual features, and the pixel values ​​constructed by the histogram are spatially weighted using Gaussian distribution; The appearance change loss is the ratio of the histogram vectors of two mismatched appearance models. The vehicles in the trajectory merge their appearance models together according to the depth information according to the appearance change loss, and the merged appearance model is used to describe the appearance changes of the vehicle along the entire trajectory; The vehicle speed is calculated according to the vehicle distance based on the sliding time window algorithm, and it is determined whether the speed variance is less than a given threshold. If the speed is too small, the vehicle is judged to be a stopped vehicle.

3. The vehicle detection method based on monocular vision and deep learning as claimed in claim 2, characterized in that: Camera calibration, including: receiving two pairs of vanishing lines and a camera height range manually marked in a highway video monitoring image; Derive two vanishing points on the ground plane based on the two pairs of marked vanishing lines; The camera parameters are calculated based on the two vanishing points and the camera's height range, and the optimization algorithm is used to optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space to calibrate the highway video monitoring camera. An optimization algorithm is used to minimize the reprojection error.

4. The vehicle detection method based on monocular vision and deep learning as claimed in claim 2, characterized in that: Vehicle detection using neural networks in a multi-module collaborative object recognition model based on deep learning, including: A video input unit, for receiving real-time video data and storing it in a designated memory area; A video processing subsystem unit, used to retrieve the original video data in the memory area and decompose it into basic video data and extended video data; wherein the basic video data maintains the resolution of the original video data; and the resolution of the extended video data matches the neural network model in the neural network acceleration engine unit; An intelligent video engine unit, used for converting image frame data in the current extended video data into frame data in an image format that matches the neural network model; A neural network acceleration engine unit is used to obtain the frame data after format conversion, and obtain the category and contour four-point coordinate position information of the object through neural network model recognition. The neural network acceleration engine unit includes loading a trained neural network model in a specific format during the initialization operation; The video graphics subsystem unit is used to obtain basic video data, and then draw a contour frame of the identified object in the basic video data based on the category of the object and the coordinate position information of the four points of the contour; The video output unit is used to output video image data with a contour frame of a recognized object.

5. A vehicle detection system based on monocular vision and deep learning, characterized in that: include: The data acquisition module is configured to: acquire highway video frame image data, optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space using an evolutionary algorithm, perform camera calibration, and obtain calibrated camera parameters; The vehicle detection module is configured to: use a neural network to detect the vehicle, obtain the vehicle type and the two-dimensional bounding box, and use the calibrated camera parameters to trace the foot points of the two-dimensional bounding box of the vehicle back to the three-dimensional space to obtain the vehicle coordinate points in the world coordinate system; The distance detection module is configured to: calculate the vehicle distance using the Euclidean distance according to the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera; The trajectory generation module is configured to: based on the obtained vehicle distance, learn the long-term vehicle appearance change through a histogram-based adaptive appearance model to obtain the trajectory of the vehicle to be detected, and use a clustering method to associate the vehicle trajectory to a longer trajectory; A clustering method is used to associate the vehicle's trajectory into a longer trajectory, including: Bottom-up clustering of tracking trajectories is performed through the loss function to associate trajectories into longer trajectories to achieve vehicle tracking. All trajectories are iteratively clustered into trajectories until convergence; The loss functions include trajectory smoothness loss, speed change loss, time interval loss, and appearance change loss; The appearance change loss is calculated using a histogram-based adaptive appearance model, including: Construct a histogram for each cropped vehicle region within the detected bounding box; When the vehicle is occluded by other objects, the occluded area is removed from the target area before constructing the concatenated histogram of visual features, and the pixel values ​​constructed by the histogram are spatially weighted using Gaussian distribution; The appearance change loss is the ratio of the histogram vectors of two mismatched appearance models. The vehicles in the trajectory merge their appearance models together according to the depth information according to the appearance change loss, and the merged appearance model is used to describe the appearance changes of the vehicle along the entire trajectory.

6. A vehicle detection system based on monocular vision and deep learning, characterized in that: include: The data acquisition module is configured to: acquire highway video frame image data, optimize the camera parameters of the back-projection from two-dimensional space to three-dimensional space using an evolutionary algorithm, perform camera calibration, and obtain calibrated camera parameters; The vehicle detection module is configured to: use a neural network to detect the vehicle, obtain the vehicle type and the two-dimensional bounding box, and use the calibrated camera parameters to trace the foot points of the two-dimensional bounding box of the vehicle back to the three-dimensional space to obtain the vehicle coordinate points in the world coordinate system; The distance detection module is configured to: calculate the vehicle distance using the Euclidean distance according to the vehicle coordinate point in the world coordinate system and the zero point coordinate established by the camera; The trajectory generation module is configured to: based on the obtained vehicle distance, learn the long-term vehicle appearance change through a histogram-based adaptive appearance model to obtain the trajectory of the vehicle to be detected, and use a clustering method to associate the vehicle trajectory to a longer trajectory; A clustering method is used to associate the vehicle's trajectory into a longer trajectory, including: Bottom-up clustering of tracking trajectories is performed through the loss function to associate trajectories into longer trajectories to achieve vehicle tracking. All trajectories are iteratively clustered into trajectories until convergence; The loss functions include trajectory smoothness loss, speed change loss, time interval loss, and appearance change loss; The appearance change loss is calculated using a histogram-based adaptive appearance model, including: Construct a histogram for each cropped vehicle region within the detected bounding box; When the vehicle is occluded by other objects, the occluded area is removed from the target area before constructing the concatenated histogram of visual features, and the pixel values ​​constructed by the histogram are spatially weighted using Gaussian distribution; The appearance change loss is the ratio of the histogram vectors of two mismatched appearance models. The vehicles in the trajectory merge their appearance models together according to the depth information according to the appearance change loss, and the merged appearance model is used to describe the appearance changes of the vehicle along the entire trajectory; The speed detection module is configured to: calculate the vehicle speed according to the vehicle distance based on the sliding time window algorithm, determine whether the speed variance is less than a given threshold, and if the vehicle has a speed that is too small, determine it to be a stopped vehicle.

7. A computer-readable storage medium having a program stored thereon, characterized in that: When the program is executed by a processor, the steps in the vehicle detection method based on monocular vision and deep learning as described in any one of claims 1 to 4 are implemented.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps in the vehicle detection method based on monocular vision and deep learning as described in any one of claims 1 to 4 are implemented.

Images