Vehicle tracking method and system for hundred million-level pixel video

By using matrix cameras and deep learning technology, combined with improved tracking algorithms and feature extraction networks, the problem of insufficient resolution and robustness of ordinary cameras is solved, and stable, efficient, accurate detection and tracking of vehicle behavior is achieved, and automatic extraction is achieved. License license plate information of the illegal vehicle.

CN119027852APending Publication Date: 2024-11-26CHENGDU TIANQU ZHICHENG TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411098378.X
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

In the prior art, the resolution and field of view of ordinary cameras are limited, making it difficult to clearly identify long-distance vehicles, and are not robust in occlusion conditions and lighting changes, resulting in poor vehicle detection and tracking effects, high algorithm complexity, and real-time performance. It's hard to guarantee.

Method used

The matrix camera is used to obtain ultra-high resolution video images, and the convolutional neural network is combined for vehicle detection and tracking, and the improved GiaSort optimization tracking algorithm and CarNet network are used to extract vehicle features, and the deep learning model is used for cross-frame tracking and violation judgment. Automatically extract license plate information.

Benefits of technology

It realizes stable, efficient and accurate detection and tracking of vehicle behavior in complex traffic scenarios, improves the performance of the vehicle tracking system, can judge violations and automatically extract license plate information, and solves the lack of resolution and robustness of traditional cameras. question.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027852A_ABST
    Figure CN119027852A_ABST
Patent Text Reader

Abstract

The invention discloses a vehicle tracking method and system for a hundred million-level pixel video. The method specifically comprises the following steps: acquiring an ultrahigh-resolution video image through a matrix camera; carrying out vehicle detection and identification on the frames of the video image by adopting a convolutional neural network; performing cross-frame tracking and positioning on the detected vehicle; analyzing the vehicle track, and judging the illegal behavior type; and finally, extracting the license plate information of the illegal vehicle. The system comprises a matrix camera shooting module, a vehicle detection and recognition module, a target vehicle tracking module, an illegal behavior judgment module and a license plate information extraction module which are in communication connection in sequence. According to the invention, the problems of limited resolution, narrow visual field and blind area of a traditional camera are solved, long continuous tracking and analysis judgment of vehicle behaviors in a complex traffic scene are realized, and the performance of a vehicle tracking system is greatly improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of video synthesis processing, and in particular relates to a vehicle tracking method and system for billion-pixel videos. Background Art

[0002] With the development of social economy, the number of private cars has increased sharply, and urban traffic order needs to be regulated urgently. Traditional traffic police patrols are difficult to fully monitor the traffic conditions of the entire city. In order to realize smart cities, it is necessary to establish an intelligent traffic monitoring system that can monitor complex and changeable road traffic scenes all the time and in all areas. Video image target detection and tracking technology has been widely used in the field of intelligent monitoring. Vehicle detection and tracking for traffic scenes has also become a research hotspot. Vehicle detection and tracking systems based on ordinary cameras have made certain progress, and typical ones include tracking systems based on background difference method and tracking systems based on optical flow method.

[0003] However, existing methods have obvious shortcomings: the resolution and field of view of ordinary cameras are limited, making it difficult to clearly identify long-distance vehicles, resulting in a high rate of missed vehicle detection; the effect is not good in occlusion situations, and vehicles are easily missed when they are blocked by other vehicles; the external environment such as lighting changes has a great impact, and the robustness is weak; the algorithm is highly complex, and real-time performance is difficult to guarantee.

[0004] In recent years, deep learning-based target detection and tracking technology has made great progress, especially the outstanding performance of convolutional neural networks in image feature extraction, which has brought new opportunities for intelligent traffic monitoring. However, it is difficult to obtain clear and reliable video images by directly using ordinary cameras, which has become a bottleneck limiting the effectiveness of the system.

[0005] How to obtain higher quality traffic surveillance videos to achieve stable, efficient and accurate detection and tracking of vehicle behaviors in road scenes is a technical problem that needs to be solved at present. Summary of the invention

[0006] In order to overcome the defects of the prior art, the present invention provides a vehicle tracking method and system for billion-pixel videos.

[0007] A vehicle tracking method for billion-pixel video of the present invention comprises the following steps:

[0008] Step 1: Obtain ultra-high resolution video images through a matrix camera.

[0009] Step 2: Use a convolutional neural network to detect and identify vehicles in frames of video images.

[0010] Step 3: Track and locate the detected vehicle across frames.

[0011] Step 4: Analyze vehicle trajectories and determine the type of violation.

[0012] Among them, the types of violations include but are not limited to: running red lights, violating parking bans and failing to give way to pedestrians.

[0013] Step 5: Extract the license plate information of the illegal vehicle.

[0014] Firstly, the warped plane detection network WPOD-NET is used to detect the license plate area and locate the potential license plate target area; then, image preprocessing is performed to improve the clarity and contrast of the license plate; finally, a convolutional neural network is used to perform character recognition on the extracted license plate image and output the license plate number information.

[0015] Furthermore, in step 2, YOLOv5 is used as the detection model and pre-trained on the COCO dataset.

[0016] Furthermore, in step 2, CSPDarknet53 is used as the backbone feature extraction network to improve the feature extraction capability through the residual structure.

[0017] Furthermore, in step 2, the PANet path set structure is used to fuse semantic information of different scales and detect objects of different sizes.

[0018] Furthermore, in step 3, the GiaSort optimization tracking algorithm based on DeepSort is used, and a deep learning model is introduced. In the real-time target tracking process, the appearance features of the target are extracted for nearest neighbor matching, so that the output single-frame target frame can be tracked across frames, and then the trajectory information is output.

[0019] Furthermore, the GiaSort optimization tracking algorithm uses Kalman filtering for target state prediction and the Hungarian algorithm for matching allocation, and adaptively adjusts the threshold of appearance feature matching according to the motion state and appearance characteristics of different vehicles.

[0020] Furthermore, the GiaSort optimized tracking algorithm extracts more discriminative vehicle feature descriptors based on the residual network's vehicle appearance feature extraction network CarNet; the CarNet models the target more robustly through residual structure and multi-scale feature aggregation.

[0021] The CarNet model contains multiple residual blocks, each of which contains a convolutional feature stream composed of multiple convolutional layers. The number of convolutional feature streams and the number of convolutional layers they contain are set according to the depth of the network. Each convolutional feature stream is used to process feature information at a certain scale of the image pyramid.

[0022] In each residual block, convolutional feature streams of different scales first extract features in parallel, and then fuse the features through a unified aggregation gate structure; the aggregation gate structure includes a global average pooling layer, a fully connected layer and an affine transformation layer; among them, the global average pooling layer obtains the vector of each feature stream, the fully connected layer and the affine transformation layer learn the weight of each feature stream, and act as gating units to perform weighted fusion on the description vectors; the output of the residual block is the weighted aggregation result of features of different scales.

[0023] A vehicle tracking system for billion-pixel video of the present invention uses the above-mentioned method to track vehicles in billion-pixel video; the system includes a matrix camera shooting module, a vehicle detection and identification module, a target vehicle tracking module, a violation judgment module and a license plate information extraction module which are sequentially communicatively connected.

[0024] The matrix camera shooting module obtains ultra-high resolution video images through the matrix camera.

[0025] The vehicle detection and recognition module uses a convolutional neural network to perform vehicle detection and recognition on frames of video images.

[0026] The target vehicle tracking module tracks and locates the detected vehicle across frames.

[0027] The violation judgment module analyzes the vehicle trajectory and determines the type of violation.

[0028] The license plate information extraction module extracts the license plate information of the illegal vehicle. The license plate information extraction module includes a license plate area detection submodule, an image preprocessing submodule and a license plate recognition submodule; among them, the license plate area detection submodule uses the warped plane detection network WPOD-NET to detect the license plate area, the image preprocessing submodule performs image preprocessing, and the license plate recognition submodule uses a convolutional neural network to perform character recognition on the extracted license plate image.

[0029] Furthermore, the matrix camera includes at least 4 sub-lenses arranged in a 2×2 matrix shape; the ultra-high-resolution video image is an image obtained by super-pixel fusion of the sub-lenses of the matrix camera, which is a seamless high-definition wide-angle field of view.

[0030] Furthermore, the resolution of each sub-lens is 3840×2160 pixels; the pixels of the ultra-high-resolution video image are 300 million.

[0031] The beneficial technical effects of the present invention are:

[0032] 1. The present invention adopts a matrix camera to obtain ultra-high-definition video, uses a convolutional neural network for target detection and tracking, and combines vehicle motion trajectory analysis and scene analysis to determine the type of violation behavior. It solves the problems of limited resolution, narrow field of view, and blind spots of traditional cameras, realizes long-term continuous tracking and analysis and judgment of vehicle behavior in complex traffic scenes, and greatly improves the performance of the vehicle tracking system.

[0033] 2. The present invention can obtain ultra-high-resolution traffic monitoring videos, use deep learning algorithms to achieve stable detection and precise tracking of vehicles, and can determine the violations of vehicles and automatically extract the license plate information of the violating vehicles.

[0034] 3. The present invention obtains higher quality traffic monitoring videos and realizes stable, efficient and accurate detection and tracking of vehicle behaviors in road scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 The present invention is a flowchart of the steps of the vehicle tracking method using billion-pixel video.

[0036] Figure 2 This is a diagram showing the vehicle tracking effect of the present invention in a 300 million pixel video.

[0037] Figure 3 This is a model structure diagram of CarNet in the present invention.

[0038] Figure 4 It is a structural schematic diagram of the vehicle tracking system of the present invention with billion-pixel video.

[0039] Figure 4 In the figure: 100: matrix camera shooting module, 200: vehicle detection and recognition module, 300: target vehicle tracking module, 400: violation judgment module, 500: license plate information extraction module. DETAILED DESCRIPTION

[0040] The present invention is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0041] The vehicle tracking method of the present invention using billion-pixel video is as follows: Figure 1 As shown, specifically:

[0042] Step 1: Obtain ultra-high resolution video images through a matrix camera.

[0043] The matrix camera includes at least 4 sub-lenses arranged in a 2×2 matrix shape, but is not limited to this; the ultra-high-resolution video image is the image after super-pixel fusion of the sub-lenses of the matrix camera, which is a seamless high-definition wide-angle field of view. The resolution of the sub-lenses is 3840×2160 pixels, that is, 4K resolution; the pixels of the ultra-high-resolution video image are about 300 million, please refer to Figure 2 (a) and (b) show the vehicle tracking effect of the present invention in a 300 million pixel video. Compared with a single ordinary camera, the matrix camera can provide a larger field of view, higher resolution, and no blind spot images, ensuring that the vehicle target is clearly identifiable.

[0044] Step 2: Use a convolutional neural network to detect and identify vehicles in frames of video images.

[0045] Taking into account the balance between model size and detection accuracy and speed, the present invention adopts YOLOv5 as the detection model and pre-trains it on the COCO dataset. It can detect multiple categories of targets in traffic scenes and has the characteristics of fast detection speed and high accuracy.

[0046] The present invention uses CSPDarknet53 as the backbone feature extraction network and improves the feature extraction capability through the residual structure.

[0047] The present invention adopts the PANet path set structure, which fully integrates semantic information of different scales and is conducive to detecting targets of different sizes.

[0048] Step 3: Track and locate the detected vehicle across frames.

[0049] The present invention adopts the GiaSort optimization tracking algorithm improved based on DeepSort; wherein, DeepSort is an improvement based on Sort target tracking, which introduces a deep learning model, and in the real-time target tracking process, extracts the appearance features of the target for nearest neighbor matching, realizes cross-frame tracking of the output single-frame target frame, and then outputs trajectory information; thereby, the path drift problem caused by vehicle tracking can be effectively prevented.

[0050] The GiaSort optimization tracking algorithm uses Kalman filtering for target state prediction and the Hungarian algorithm for matching allocation, and combines deep appearance features to improve matching accuracy. The GiaSort optimization tracking algorithm also uses a cascade matching strategy and a dedicated trajectory management mechanism to efficiently handle difficult problems such as target occlusion. Compared with the traditional DeepSort inter-frame matching tracking method, the GiaSort optimization tracking algorithm can output a more stable and clear target trajectory.

[0051] The GiaSort optimization tracking algorithm adaptively adjusts the threshold of appearance feature matching according to the motion state and appearance characteristics of different vehicles. It lowers the threshold for vehicles with occlusions and increases the threshold for vehicles with faster speeds. This can effectively balance the matching accuracy and recall.

[0052] The GiaSort optimized tracking algorithm constructs a vehicle appearance feature extraction network CarNet based on the residual network, which can extract more discriminative vehicle feature descriptors to improve the accuracy of the GiaSort optimized tracking algorithm; CarNet can model the target more robustly through residual structure and multi-scale feature aggregation.

[0053] like Figure 3 As shown, it is a model structure diagram of CarNet in the present invention. The CarNet model includes multiple residual blocks, each residual block includes a convolutional feature stream composed of multiple convolutional layers; the number of convolutional feature streams and the number of convolutional layers they contain are set according to the depth of the network, and each convolutional feature stream is used to process feature information at a certain scale of the image pyramid.

[0054] In each residual block, convolutional feature streams of different scales first extract features in parallel, and then fuse the features through a unified aggregation gate structure; the aggregation gate structure includes a global average pooling layer, a fully connected layer, and an affine transformation layer. Among them, the global average pooling layer can obtain the vector of each feature stream, and the fully connected layer and the affine transformation layer can learn the weight of each feature stream and perform weighted fusion of the description vector as a gate control unit; finally, the output of the residual block is the weighted aggregation result of features of different scales.

[0055] Compared with the ordinary residual block structure, CarNet's aggregation gate mechanism realizes the dynamic fusion of multi-scale features, allowing the network to flexibly fuse information of different granularities; this is very important for maintaining the identifiability of multi-targets and multi-scales in narrow and long areas.

[0056] In the CarNet model, the first residual block contains two convolutional feature streams, the middle residual block contains three convolutional feature streams, and the last residual block contains four convolutional feature streams; each feature stream contains several groups of convolutional layers, usually a Bottleneck structure, that is, 1×1 and 3×3 convolutions. As the network depth increases, the number of convolutional feature streams and the number of channels also gradually increase to extract richer feature representations.

[0057] In the present invention, the GiaSort optimization tracking algorithm uses a Gaussian regression module and combines the motion information of the target vehicle to predict the vehicle position; thereby, the position prediction is more accurate and the drift phenomenon can be effectively reduced.

[0058] In the present invention, the GiaSort optimized tracking algorithm adopts an enhanced tracking initialization and management mechanism to achieve more efficient and stable long-term tracking; specifically:

[0059] 1) For a newly appeared vehicle, the GiaSort optimized tracking algorithm does not initialize tracking immediately, but sets up a buffer to temporarily store the detection information; only when the new vehicle is detected in multiple consecutive frames will a tracking trajectory be formally initialized for it, effectively filtering out false detections of jumps.

[0060] 2) If the number of consecutive successful matching frames of a tracking trajectory exceeds the threshold, the tracking trajectory will be confirmed; the confirmed tracking will receive a higher matching weight.

[0061] 3) If a vehicle is mismatched in a certain frame, the GiaSort optimization tracking algorithm will temporarily store it and will not delete the track immediately; if the target is redetected within a certain number of frames later, the tracking relationship of the vehicle will be restored.

[0062] 4) If a vehicle’s tracking trajectory is not matched or detected for multiple consecutive frames, the vehicle will eventually be deleted; at the same time, those trajectories that have not been matched since being confirmed will also be deleted.

[0063] Step 4: Analyze vehicle trajectories and determine the type of violation.

[0064] Among them, the types of violations include but are not limited to: running red lights, violating parking bans and failing to give way to pedestrians.

[0065] 1) Red light running judgment: First, the traffic light target in the frame needs to be detected and identified, classified into horizontal and vertical situations according to the direction of the traffic light, and the three areas of the traffic light are segmented; then the pixel value of the traffic light image is analyzed to determine the current lighting status of the traffic light; at the same time, the vehicle's trajectory information is obtained to analyze whether the vehicle continues to move forward when the red light is on, so as to determine whether a red light running behavior occurs.

[0066] 2) Judgment of violation of no-parking regulations: It is necessary to first identify the no-parking traffic signs in the video frame and locate the no-parking area; then analyze the path of the vehicles entering the area and the length of time they stay there. If the length of time exceeds the specified time, it is judged as illegal parking.

[0067] 3) Judgment of failure to give way to pedestrians: First, identify the pedestrian crossing sign in the video. Once the sign is identified, analyze whether there are pedestrian targets around the sign; at the same time, detect the vehicle trajectory to determine whether the vehicle slows down and stops at the sign to let pedestrians pass. If no courtesy behavior is made, it is judged as a violation.

[0068] Step 5: Extract the license plate information of the illegal vehicle.

[0069] Firstly, the warped plane detection network WPOD-NET is used to detect the license plate area and locate the potential license plate target area; then, image preprocessing is performed to improve the clarity and contrast of the license plate; finally, a convolutional neural network is used to perform character recognition on the extracted license plate image and output the license plate number information.

[0070] The vehicle tracking system of the present invention uses the above method to track the vehicle in the billion-pixel video. Figure 4 As shown, the system includes a matrix camera shooting module 100, a vehicle detection and identification module 200, a target vehicle tracking module 300, a violation judgment module 400 and a license plate information extraction module 500 which are sequentially communicatively connected.

[0071] The matrix camera shooting module 100 acquires ultra-high resolution video images through a matrix camera.

[0072] The vehicle detection and recognition module 200 uses a convolutional neural network to perform vehicle detection and recognition on frames of video images.

[0073] The target vehicle tracking module 300 performs cross-frame tracking and positioning on the detected vehicle.

[0074] The violation determination module 400 analyzes the vehicle trajectory and determines the type of violation.

[0075] The license plate information extraction module 500 extracts the license plate information of the illegal vehicle. The license plate information extraction module includes a license plate area detection submodule, an image preprocessing submodule and a license plate recognition submodule; wherein the license plate area detection submodule uses a warped plane detection network WPOD-NET to perform license plate area detection, the image preprocessing submodule performs image preprocessing, and the license plate recognition submodule uses a convolutional neural network to perform character recognition on the extracted license plate image.

[0076] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A vehicle tracking method for billion-pixel video, characterized in that: The following steps are involved: Step 1: Obtain ultra-high resolution video images through a matrix camera; Step 2: Use a convolutional neural network to detect and identify vehicles in frames of video images; Step 3: Track and locate the detected vehicle across frames; Step 4: Analyze vehicle trajectories and determine the type of violation; Among them, the types of violations include but are not limited to: running red lights, violating parking prohibitions and failing to give way to pedestrians; Step 5: Extract the license plate information of the illegal vehicle; Firstly, the warped plane detection network WPOD-NET is used to detect the license plate area and locate the potential license plate target area; then, image preprocessing is performed to improve the clarity and contrast of the license plate; finally, a convolutional neural network is used to perform character recognition on the extracted license plate image and output the license plate number information.

2. The vehicle tracking method of billion-pixel video according to claim 1, characterized in that: In step 2, YOLOv5 is used as the detection model and pre-trained on the COCO dataset; Use CSPDarknet53 as the backbone feature extraction network and improve feature extraction capabilities through residual structure; The PANet path collection structure is adopted to fuse semantic information of different scales and detect objects of different sizes.

3. The vehicle tracking method of the billion-pixel video according to claim 1, characterized in that: In step 3, an improved GiaSort optimization tracking algorithm based on DeepSort is used, and a deep learning model is introduced. In the real-time target tracking process, the appearance features of the target are extracted for nearest neighbor matching, so that the output single-frame target frame can be tracked across frames, and then the trajectory information is output.

4. The vehicle tracking method of billion-pixel video according to claim 3, characterized in that: The GiaSort optimization tracking algorithm adopts Kalman filtering to predict the target state and the Hungarian algorithm to perform matching allocation, and adaptively adjusts the threshold of appearance feature matching according to the motion state and appearance characteristics of different vehicles.

5. The vehicle tracking method of billion-pixel video according to claim 3, characterized in that: The GiaSort optimization tracking algorithm extracts more discriminative vehicle feature descriptors based on the residual network vehicle appearance feature extraction network CarNet; the CarNet models the target more robustly through residual structure and multi-scale feature aggregation; The CarNet model includes multiple residual blocks, each of which includes a convolutional feature stream composed of multiple convolutional layers; the number of convolutional feature streams and the number of convolutional layers they contain are set according to the depth of the network, and each convolutional feature stream is used to process feature information at a certain scale of the image pyramid; In each residual block, convolutional feature streams of different scales are first extracted in parallel, and then fused through a unified aggregation gate structure; the aggregation gate structure includes a global average pooling layer, a fully connected layer and an affine transformation layer; among them, the global average pooling layer obtains the vector of each feature stream, the fully connected layer and the affine transformation layer learn the weight of each feature stream, and act as gating units to perform weighted fusion on the description vectors; the output of the residual block is the weighted aggregation result of features of different scales.

6. A vehicle tracking system with billion-pixel video, characterized in that: The system uses the method described in any one of claims 1 to 5 to track vehicles using billion-pixel videos; the system includes a matrix camera shooting module, a vehicle detection and recognition module, a target vehicle tracking module, a violation judgment module, and a license plate information extraction module that are sequentially connected in communication; The matrix camera shooting module obtains ultra-high resolution video images through a matrix camera; The vehicle detection and recognition module uses a convolutional neural network to perform vehicle detection and recognition on frames of video images; The target vehicle tracking module performs cross-frame tracking and positioning on the detected vehicle; The violation judgment module analyzes the vehicle trajectory and judges the type of violation; The license plate information extraction module extracts the license plate information of the illegal vehicle; the license plate information extraction module includes a license plate area detection submodule, an image preprocessing submodule and a license plate recognition submodule; wherein the license plate area detection submodule uses a warped plane detection network WPOD-NET to perform license plate area detection, the image preprocessing submodule performs image preprocessing, and the license plate recognition submodule uses a convolutional neural network to perform character recognition on the extracted license plate image.

7. The vehicle tracking system of billion-pixel video according to claim 6, characterized in that: The matrix camera includes at least 4 sub-lenses arranged in a 2×2 matrix shape; the ultra-high-resolution video image is an image obtained by super-pixel fusion of the sub-lenses of the matrix camera, which is a seamless high-definition wide-angle field of view.

8. The vehicle tracking system of billion-pixel video according to claim 7, characterized in that: The resolution of the sub-lenses is 3840×2160 pixels; the pixels of the ultra-high resolution video image are 300 million.