Vehicle track recovery method based on computer vision

Abstract

The invention discloses a vehicle track recovery method based on computer vision, and the method specifically comprises the following steps: S1, obtaining a vehicle: collecting a video stream, and obtaining the vehicle in an image; s2, constructing an LPRNet network: constructing the LPRNet network, and training a weight file of an LPRNet model by using the data set; s3, identifying a license plate in the image: identifying license plate information of the vehicle in the image through a weight file of the trained LPRNet model, and checking whether the license plate information is correct or not; storing the verified license plate information in a database; s4, information interaction: interactively combining the geographic position of the image acquisition device with the license plate information and storing the information in a database; and S5, vehicle track recovery: reading the database to obtain the license plate information, associating the point location information corresponding to the vehicle, and recovering the track corresponding to the vehicle through a multi-point location path planning algorithm. According to the method, vehicle track recovery can be carried out under the condition that the GPS is lost.

Description

Technical Field

[0001] The invention belongs to the technical field of computer vision, and in particular relates to a vehicle trajectory recovery method based on computer vision. Background Art

[0002] Vehicle trajectory recovery plays an important role in smart cities and transportation. With the increase in vehicles, how to predict the trajectory of special vehicles and other vehicles, and return the corresponding trajectory information to the background for recording and analysis has become an important aspect of smart city applications. During the construction of new urban areas, the trajectory of engineering vehicles is closely monitored. Although engineering vehicles are equipped with GPS positioning devices, they can transmit the location information of the vehicles back to the background in real time to monitor the vehicles. This type of GPS information has the advantages of high accuracy and accurate time, but at the same time, these GPS data exist in the terminal of the vehicle equipment and its background. It is difficult for third users to obtain real-time GPS data, and there is also the possibility of GPS data loss.

[0003] At present, the vehicle trajectory is mainly restored with the help of GPS positioning data. The VDR (Vehicle Data Recovery) vehicle trajectory estimation algorithm fuses GPS data and IMU data to predict the vehicle's trajectory. The solution of analyzing the vehicle's motion trajectory through multiple sensor fusion data can well restore the vehicle's driving trajectory, and the data can still be fused and analyzed when the GPS positioning data is lost. Select multiple fine-grained trajectory data from the GPS data as auxiliary data to construct a dense vector of trajectory data, embed the spatiotemporal features of each group of trajectories into the corresponding time series, and then use the attention mechanism and average sampling strategy to fuse the spatiotemporal features and trajectory features to generate trajectory data for each intersection.

[0004] The existing technical solutions mainly use GPS data and other data to predict the trajectory. When GPS data is lost, there will be a certain error in the prediction of the trajectory. In addition, high-precision IMU equipment is expensive, and it is difficult for third-party users to obtain real-time IMU data. Considering that GPS data and IMU data are difficult to obtain in real time, this paper obtains the license plate information through video stream to recover the vehicle's trajectory.

[0005] Chinese patent document CN 118587678 A discloses a method for tracking indoor vehicles in a parking lot, including: performing target detection on the real-time image of the detection area to be detected, obtaining the target detection frame of the tracking target; for each tracking target image to be processed, in the first frame of the video, initializing all detection frames as track segments; for each tracking target image to be processed, in the subsequent frames of the video, associating and matching the target detection frame with the prediction frame of the track segment in the corresponding frame; updating the track information according to the matching result, and realizing the continued tracking of the tracking target. It mainly realizes the recovery of the video screen in the case of loss of the video target tracking detection frame due to occlusion, etc., and does not focus on track recovery.

[0006] Therefore, it is necessary to provide a vehicle trajectory recovery method based on computer vision, which can also recover the vehicle trajectory in the event of GPS loss, while improving management efficiency and safety, optimizing vehicle scheduling and route planning, and preventing potential safety risks. Summary of the invention

[0007] The technical problem to be solved by the present invention is to provide a method for vehicle trajectory recovery based on computer vision. In the case of GPS loss, the vehicle trajectory on the map can be restored with the help of vehicle type, license plate information, location information, road network planning, monitoring information, etc. in target recognition, thereby improving management efficiency and safety, optimizing vehicle scheduling and route planning, and preventing potential safety risks.

[0008] In order to solve the above technical problems, the technical solution adopted by the present invention is: the method for recovering vehicle trajectory based on computer vision specifically comprises the following steps:

[0009] S1 obtains the vehicle: collects the video stream and obtains the vehicle in the image;

[0010] S2 builds the LPRNet network: builds the LPRNet network and uses the dataset to train the weight file of the LPRNet model;

[0011] S3 recognizes the license plate in the image: recognizes the license plate information of the vehicle in the image through the weight file of the trained LPRNet model, and verifies whether the license plate information is correct; stores the verified license plate information in the database;

[0012] S4 information interaction: interactively combining the geographical location of the image acquisition device with the license plate information and storing them in the database;

[0013] S5 Vehicle trajectory recovery: read the database, obtain the license plate information, and associate the point information corresponding to the vehicle, then restore the corresponding trajectory of the vehicle through the multi-point path planning algorithm and display it on the map.

[0014] By adopting the above technical solution, the acquired license plate information is bound to the geographical location of the image acquisition device, and then the trajectory is restored, the position of the detected vehicle is fed back in real time on the map, and finally the intersections where the vehicle has traveled are queried through the background algorithm and the results are displayed on the front-end map; in the case of GPS loss, the vehicle trajectory on the map is restored with the help of the vehicle type, license plate information, location information, road network planning, monitoring information, etc. in the target identification. At the same time, it has a certain warning effect on illegal construction vehicles, improves management efficiency and safety, optimizes vehicle scheduling and route planning, and prevents potential safety risks, playing a certain role in urban governance.

[0015] Preferably, the specific steps of step S1 are:

[0016] S11: collecting a corresponding intersection video stream by an image acquisition device arranged at the intersection;

[0017] S12: Acquire an image through the collected video stream, and assign the geographical location information of the image acquisition device to the image;

[0018] S13: Acquire vehicle data from the image to form a data set; and then divide the data set into a training set and a validation set.

[0019] The purpose of adopting the above technical solution is to extract vehicle information from the video stream and provide necessary data support for subsequent trajectory recovery. The image acquisition device in the present invention is preferably a traffic monitoring camera, which should be able to clearly capture vehicle features and license plate information. Therefore, the license plate information of the vehicle is obtained by acquiring the corresponding intersection video stream through the monitoring camera; when the vehicle appears in the area that the camera can capture, its license plate is recorded in the database; and at the same time, the geographical location of the camera is written into the camera information.

[0020] Preferably, the LPRNet network constructed in step S2 includes a positioning network and a recognition network, the positioning network is used to locate the license plate area, and the recognition network is used to recognize the license plate characters; the structure of the LPRNet network specifically includes an input layer, a convolutional layer, a maximum pooling layer, a small basic block and a random deactivation function;

[0021] The specific steps of using the data set to train the LPRNet model in step S2 are:

[0022] S21 data preprocessing: first perform image enhancement on the image, and then use one-hot encoding or character index encoding; image enhancement: in order to increase the robustness of the model, the license plate image can be rotated, translated, scaled, cropped, etc.; label encoding: the license plate number usually needs to be encoded into a sequence of numbers or characters, using one-hot encoding or character index encoding;

[0023] S22 defines the loss function: The LPRNet model uses the CTC loss function, because the order of license plate characters does not completely correspond to the features of the image, and the CTC loss function can handle this problem well; the formula is:

[0024]

[0025] Where: Π is the set of all possible character sequences; P(π|x) is the probability of the sequence π given an input x; the core idea of ​​the CTC loss function is to let the model learn how to decode characters without requiring strict alignment of characters with time steps in the image;

[0026] S23 Network training: When training the LPRNet model, the network's goal is to minimize the CTC loss function; the training steps are:

[0027] S231 forward propagation: The input image is forward propagated through the network and the probability distribution of the license plate characters is output;

[0028] S232 Loss calculation: Calculate the difference between the model output and the true label according to the CTC loss function;

[0029] S233 Back Propagation: Calculate the gradient through back propagation and update the network weights;

[0030] S234 Optimization: Use an optimization algorithm to update model parameters to minimize the loss function; the formula is:

[0031]

[0032] Where: θ t is the current parameter; η is the learning rate; is the gradient of the loss function;

[0033] S24 Validation and tuning: During the training process, the validation set is used for validation, the accuracy of the model is evaluated, and tuning is performed based on the validation results. Common tuning methods include adjusting the learning rate, optimizer parameters, etc.

[0034] The network used in the vehicle recognition process is the LRPNet network released by Microsoft, which recognizes the specific number of the license plate appearing in the video. LPRNet is a license plate recognition algorithm based on a deep neural network. It is an end-to-end training method that can recognize license plates directly from raw images. The LPRNet network structure consists of two parts: the positioning network and the recognition network. The positioning network is used to locate the license plate area, and the recognition network is used to recognize the license plate characters. The advantage of LPRNet is that it does not require pre-segmentation of characters, so it can handle different types of license plates, including Chinese license plates.

[0035] The LPRNet (License Plate Recognition Network) network structure is an efficient license plate recognition network, which consists of two main parts: positioning network and recognition network;

[0036] (1) Positioning network

[0037] The localization network is used to locate the license plate area in the image, that is, to extract the location of the license plate from the entire image. This process is usually implemented through the target detection algorithm;

[0038] Input layer: The input layer accepts an image, usually a preprocessed license plate area image, with a size of W×H×C (W is width, H is height, and C is the number of channels, usually 1 or 3, depending on whether it is a grayscale image);

[0039] Convolutional layer: extracts image features through a series of convolution operations. The output of each convolution operation generates a new feature map through a convolution kernel filter, usually with different step sizes and numbers of filters;

[0040] Max Pooling Layer: The pooling layer is used to reduce the amount of calculation and extract important features in the image by reducing the dimension. The most commonly used one is Max Pooling, which selects the maximum value in the pooling window.

[0041] Small basic blocks and activation functions (ReLu): Network modules composed of multiple convolutional layers to extract more complex features. Small basic blocks usually consist of convolutional layers, batch normalization layers, and activation functions (such as ReLU);

[0042] The output of the localization network is a location coordinate (Bounding Box), which is the location of the license plate in the image; (2) Recognition network

[0043] The recognition network is used to perform character recognition on the license plate area and map the located license plate area into a sequence of license plate number characters;

[0044] Input layer: The image of the license plate area (extracted by the localization network) is used as input. The image size is usually cropped and standardized and scaled to a fixed size, such as 32×100 pixels;

[0045] Convolutional layers and small basic blocks: Similar to the localization network, the recognition network also extracts image features through convolutional layers and small basic blocks. High-dimensional features are extracted through multiple convolutional layers;

[0046] LSTM layer (Long Short-Term Memory Network): LPRNet uses LSTM network to process the sequence relationship of characters. Since license plate characters are sequence data, LSTM layer can capture the dependency relationship between characters; Output layer: The output layer usually uses CTC (Connectionist Temporal Classification) loss function, and the output of this layer is the probability distribution of characters at each time step;

[0047] The specific structure of the LPRNet network is as follows:

[0048] Input layer: an image of size H×W×C (usually an image of the license plate area);

[0049] Convolutional layer and pooling layer: extract image features through multiple convolutional layers and pooling layers;

[0050] Basic Block: These modules consist of multiple convolutional layers and activation functions to further extract high-level features;

[0051] LSTM layer: After extracting the features, LSTM is used to process the character sequence information and capture the temporal relationship between characters;

[0052] CTC output layer: Outputs the probability distribution of each character and is trained using CTC loss.

[0053] Preferably, in step S3, an image is obtained from the video stream collected from the area to be tested, the image with vehicle data is input into the trained LPRNet model, the license plate content of the vehicle is identified, and the license plate information is obtained; and the identified license plate information is modified into a string form for storage, and the time information of the vehicle appearing in the image is stored at the same time.

[0054] Preferably, the specific steps of step S4 are:

[0055] S41: information of the image acquisition device defined by the user and obtaining geographical location information of the image acquisition device;

[0056] S42: interacting the license plate information identified in step S3 with the geographical location information of the image acquisition device to generate an interactive data content of the license plate information and the geographical location information of the image acquisition device;

[0057] S43: storing the interactive data content in a database, that is, binding each piece of license plate information with the geographical location of the crossroads generating the data of the license plate information in the database.

[0058] In the entire closed environment, the geographical location of each camera is fixed. Therefore, the geographical location of the camera is written into the database as a field, and each piece of data in the database is bound to the geographical location of the camera. In this way, the recognized license plate can be combined with the geographical location of the camera, and then the geographical location can be integrated into the license plate information. In the database, since each piece of data is geographically bound to the intersection where the data is generated, it can be determined that the position of the vehicle is certain at that moment. When the vehicle passes the second intersection with a camera, a second set of vehicle-location related records is generated, and the record contains the geographical location coordinates. In the database, the coordinate information corresponding to the camera where the vehicle appears is associated with the license plate number and stored, and the time factor is taken to perform a temporal association operation, so the data collection work is completed.

[0059] Preferably, the image acquisition device is a camera; the geographic location information of the camera in step S41 includes the longitude, latitude and geographic location description of the camera; the database is stored in the form of a data table, and the data table includes the vehicle number, corresponding time, corresponding geographic location description, longitude and latitude and license plate number, and storage path.

[0060] Preferably, the specific steps of step S5 are:

[0061] S51: Read the data table in the database, present the database on the web, obtain the license plate information, and associate the point information corresponding to the vehicle;

[0062] S52: Restoring the trajectory corresponding to the vehicle through a multi-point path planning algorithm;

[0063] S53: Use an open source development library to write a web page to load and display the corresponding points, so that the points are displayed on the map.

[0064] The algorithm for path planning through fixed points is similar to the path planning algorithm for business travel problems. Therefore, a multi-point path planning algorithm is used to restore the corresponding trajectory of the vehicle. As far as the starting point to the end point is concerned, the way of thinking is to plan the path from the starting point to the end point through the path planning algorithm provided by navigation software such as Amap and Baidu Map. In terms of this feature, the selected solution is similar to the path planning algorithm in the map.

[0065] Preferably, the specific steps of step S52 are:

[0066] S521: First, the path planning area is constructed using the road data. The entire road data covers all road network data in the area, and the driving direction and lane line are also distinguished for the corresponding road data;

[0067] S522: In the background, the data information corresponding to the same vehicle is queried through the backend, the point data where the vehicle appears are aggregated, and this series of data is mapped with the road data;

[0068] S523: Using the Dijkstra algorithm to recover the trajectory of the points where the vehicle appears in the area.

[0069] Adopting the above technical solution, firstly, the path planning is established according to the existing data, and the existing data includes the road data and monitoring point data of the new district. Firstly, the path planning area is constructed using the road data. The entire road data covers all the road network data of the new district, and the driving direction and lane line are also distinguished for the corresponding road data. Such road data more accurately reflects the driving trajectory of the vehicle, and can better simulate the corresponding running trajectory of the vehicle in the later trajectory prediction. In the background, the data information corresponding to the same vehicle is queried through the backend, and the point data of the vehicle is aggregated together, and this series of data is mapped with the road data. At this point, the work of mapping the vehicle data with the road network data (road data) is completed, and the next work is to restore the path according to the corresponding points. As far as the path recovery problem is concerned, in a relatively complex road network, the situation of the intersection of the road network is taken into account, and the path planning algorithm of the commercial map is used to use Dijkstra to restore the trajectory of the points where the vehicle appears. Since there are certain errors in some road networks, some paths can be preprocessed. Dijkstra's algorithm is an algorithm for finding the shortest path in a weighted graph. It is one of the classic algorithms for solving the single-source shortest path problem. The goal of Dijkstra's algorithm is to find the shortest path from a vertex to all other vertices in the graph. The algorithm uses a greedy strategy to gradually determine the shortest path to each vertex until the shortest path to the target vertex is found.

[0070] Preferably, the specific steps of step S523 are:

[0071] S5231 Initialization: Mark the starting vertex as having a distance of 0 and other vertices as infinity; create an empty priority queue (usually implemented using a minimum heap) to select the next vertex to explore;

[0072] S5232 Select the shortest path vertex: select the vertex with the shortest distance from the priority queue in step S531 as the current vertex;

[0073] S5233 Update the distance of adjacent vertices: For each adjacent vertex of the current vertex, calculate the distance from the starting vertex through the current vertex to the adjacent vertex; if the path through the current vertex is shorter than the known shortest path, update the distance of the adjacent vertex;

[0074] S5234 marks the current vertex: marks the current vertex as visited; repeats steps S532 and S533 until all vertices are traversed or the shortest path to the target vertex is found;

[0075] S5235 Final result: When the traversal is completed, the shortest path length of each vertex is calculated; by backtracking from the target vertex to the starting vertex, the specific path of the shortest path is obtained.

[0076] Preferably, in both step S51 and step S53, the corresponding points are displayed through the open source webgis library of Openlayer; in step S53, the corresponding arrow direction of the restored path is displayed on the map side at the same time.

[0077] Compared with the prior art, the beneficial effects of the present invention are as follows: at a location with a higher monitoring coverage area, the geographic location information of the vehicle is obtained according to the point information of the video stream, and at the same time, linked with the geographic location information, the position of the detected vehicle is fed back on the map in real time, and finally the intersections where the vehicle has traveled are queried through the background algorithm and the results are finally displayed on the front-end map; when the GPS signal is lost or the error is large, it can also assist in restoring the vehicle's trajectory. At the same time, it has a certain warning effect on illegally driving engineering vehicles, improves management efficiency and safety, optimizes vehicle scheduling and route planning, prevents potential safety risks, and plays a certain role in urban governance. BRIEF DESCRIPTION OF THE DRAWINGS

[0078] Figure 1 A schematic diagram of a vehicle trajectory recovery method based on computer vision of the present invention;

[0079] Figure 2 Schematic diagram of special vehicle types in the method for restoring vehicle trajectory based on computer vision of the present invention, wherein Figure 2 (a) in the figure is a trailer; Figure 2 (b) in the figure is a crane; Figure 2 (c) in the figure is a cement mixer truck; Figure 2 (d) in the figure is a muck truck; Figure 2 (e) in the figure is a sanitation vehicle;

[0080] Figure 3 A schematic diagram of a map of monitoring points simulated in the method for restoring vehicle trajectories based on computer vision of the present invention;

[0081] Figure 4 A schematic diagram of the process of LPRNet network identification in the method for vehicle trajectory recovery based on computer vision of the present invention;

[0082] Figure 5A schematic diagram of a table constructed in a database in the method for recovering vehicle trajectory based on computer vision of the present invention;

[0083] Figure 6 A schematic diagram of the flow of the interaction between the recognized license plate and the geographic location in the method for restoring the vehicle trajectory based on computer vision of the present invention;

[0084] Figure 7 A schematic diagram of a process of restoring a vehicle trajectory in a method for restoring a vehicle trajectory based on computer vision of the present invention;

[0085] Figure 8 Schematic diagram of the predicted results in the method for vehicle trajectory recovery based on computer vision of the present invention Figure 1 ;

[0086] Fig. 9 Schematic diagram of the predicted results in the method for vehicle trajectory recovery based on computer vision of the present invention Figure 2 . DETAILED DESCRIPTION

[0087] The embodiments of the present invention are described in detail below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solutions of the present invention and are not intended to limit the protection scope of the present invention.

[0088] Example: Figure 1 As shown, the method for locating road damage specifically includes the following steps:

[0089] S1 Get vehicle: Collect video stream and get the vehicle in the image; special vehicles such as Figure 2 As shown in the figure, there are five types of vehicles, including muck trucks, trailer trucks, cement mixer trucks, sanitation trucks and cranes. These five types of vehicles are more common in urban construction. Figure 2 (a) in the figure is a trailer; Figure 2 (b) in the figure is a crane; Figure 2 (c) in the figure is a cement mixer truck; Figure 2 (d) in the figure is a muck truck; Figure 2 (e) in the figure is a sanitation vehicle;

[0090] The vehicle's license plate information is obtained by using surveillance camera test points in the urban area to restore the vehicle's trajectory. During urban expansion and high-tech development zone construction, there are many special vehicles and engineering vehicles (dump trucks, trailer trucks, cement mixer trucks, sanitation trucks, cranes, etc.). In order to determine the running trajectory of such vehicles, the specific steps of step S1 are as follows:

[0091] S11: Collect the corresponding intersection video stream through the image acquisition device (such as a traffic camera) installed at the intersection. The camera should be able to clearly capture vehicle features and license plate information;

[0092] S12: Then obtain an image through the collected video stream, and assign the geographical location information of the image acquisition device to the image; the information of the simulated monitoring point is shown in the figure below. Since it involves the specific location information of the camera, it is necessary to cover the monitoring information of each intersection, such as Figure 3 In the map diagram shown, there are some problems at the intersections. For example, some areas do not have cameras due to construction problems, so some intersections do not have camera points. At the same time, the camera is offline, so the camera points in the above figure are roughly consistent with the intersections on the entire island. It can also be found in the above figure that the road network density is relatively high near the northeast corner. At this stage, this area is mainly densely populated and has a large flow of people;

[0093] S13: Acquire vehicle data from the image to form a data set; and then divide the data set into a training set and a validation set;

[0094] S2 builds the LPRNet network: builds the LPRNet network and uses the dataset to train the weight file of the LPRNet model;

[0095] The LPRNet network constructed in step S2 includes a positioning network and a recognition network, the positioning network is used to locate the license plate area, and the recognition network is used to recognize the license plate characters; the structure of the LPRNet network specifically includes an input layer, a convolutional layer, a maximum pooling layer, a small basic block and a random deactivation function;

[0096] The specific steps of using the data set to train the LPRNet model in step S2 are:

[0097] S21 Data preprocessing: Data preprocessing is an important step in training deep learning models, mainly including:

[0098] Image enhancement: To increase the robustness of the model, the license plate image can be rotated, translated, scaled, cropped, etc.

[0099] Label encoding: License plate numbers usually need to be encoded into numbers or character sequences, using one-hot encoding or character index encoding;

[0100] S22 defines the loss function: LPRNet usually uses the CTC loss function (Connectionist Temporal Classification), because the order of license plate characters does not completely correspond to the features of the image, and the CTC loss function can handle this problem well;

[0101] The formula is:

[0102]

[0103] Where: Π is the set of all possible character sequences; P(π|x) is the probability of sequence π given input x;

[0104] The core idea of ​​the CTC loss function is to let the model learn how to decode characters without requiring strict alignment of characters with the time steps in the image;

[0105] S23 Network training: When training LPRNet, the network goal is to minimize the CTC loss function; the training steps are:

[0106] S231 forward propagation: The input image is forward propagated through the network and the probability distribution of the license plate characters is output;

[0107] S232 Loss calculation: Calculate the difference between the model output and the true label according to the CTC loss function;

[0108] S233 Back Propagation: Calculate the gradient through back propagation and update the network weights;

[0109] S234 Optimization: Use an optimization algorithm (such as Adam optimizer) to update the model parameters to minimize the loss function; the formula is:

[0110]

[0111] Where: θt is the current parameter; η is the learning rate; is the gradient of the loss function;

[0112] S24 Validation and tuning: During the training process, the validation set is used for validation, the accuracy of the model is evaluated, and tuning is performed based on the validation results. Common tuning methods include adjusting the learning rate, optimizer parameters, etc.

[0113] The LPRNet (License Plate Recognition Network) network structure is an efficient license plate recognition network, which consists of two main parts: positioning network and recognition network;

[0114] (1) Positioning network

[0115] The localization network is used to locate the license plate area in the image, that is, to extract the location of the license plate from the entire image. This process is usually implemented through the target detection algorithm;

[0116] Input layer: The input layer accepts an image, usually a preprocessed license plate area image, with a size of W×H×C (W is width, H is height, and C is the number of channels, usually 1 or 3, depending on whether it is a grayscale image);

[0117] Convolutional layer: extracts image features through a series of convolution operations. The output of each convolution operation generates a new feature map through a convolution kernel filter, usually with different step sizes and numbers of filters;

[0118] Max Pooling Layer: The pooling layer is used to reduce the amount of calculation and extract important features in the image by reducing the dimension. The most commonly used one is Max Pooling, which selects the maximum value in the pooling window.

[0119] Small basic blocks and activation functions (ReLu): Network modules composed of multiple convolutional layers to extract more complex features. Small basic blocks usually consist of convolutional layers, batch normalization layers, and activation functions (such as ReLU);

[0120] The output of the localization network is a location coordinate (Bounding Box), which is the location of the license plate in the image; (2) Recognition network

[0121] The recognition network is used to perform character recognition on the license plate area and map the located license plate area into a sequence of license plate number characters;

[0122] Input layer: The image of the license plate area (extracted by the localization network) is used as input. The image size is usually cropped and standardized and scaled to a fixed size, such as 32×100 pixels;

[0123] Convolutional layers and small basic blocks: Similar to the localization network, the recognition network also extracts image features through convolutional layers and small basic blocks. High-dimensional features are extracted through multiple convolutional layers;

[0124] LSTM layer (Long Short-Term Memory Network): LPRNet uses LSTM network to process the sequence relationship of characters. Since license plate characters are sequence data, LSTM layer can capture the dependency relationship between characters; Output layer: The output layer usually uses CTC (Connectionist Temporal Classification) loss function, and the output of this layer is the probability distribution of characters at each time step;

[0125] The specific structure of the LPRNet network is as follows:

[0126] Input layer: an image of size H×W×C (usually an image of the license plate area);

[0127] Convolutional layer and pooling layer: extract image features through multiple convolutional layers and pooling layers;

[0128] Basic Block: These modules consist of multiple convolutional layers and activation functions to further extract high-level features;

[0129] LSTM layer: After extracting the features, LSTM is used to process the character sequence information and capture the temporal relationship between characters;

[0130] CTC output layer: outputs the probability distribution of each character and is trained through CTC loss;

[0131] The network used in the vehicle recognition process is the LRPNet network released by Microsoft, which identifies the specific number of the license plate appearing in the video; LPRNet is a license plate recognition algorithm based on a deep neural network. It is an end-to-end training method that can recognize license plates directly from raw images. The LPRNet network structure consists of two parts: a positioning network and a recognition network. The positioning network is used to locate the license plate area, and the recognition network is used to recognize license plate characters. The advantage of LPRNet is that it does not require pre-segmentation of characters, so it can handle different types of license plates, including Chinese license plates; the core network of LPRNet is shown in Table 1 below:

[0132] Table 1 LPRNet core network structure

[0133] Layer Type Parameter Description Input 94×24 pixels RGB image Convolution #64 3×3stride 1 MaxPooling #64 3×3stride 1 Small basic block #128 3×3stride 1 MaxPooling #64 3×3stride 1 Small basic block #64 3×3stride(2,1) Small basic block #256 3×3stride 1 MaxPooling #256 3×3stride 1 Dropout #64 3×3stride 1 Convolution #64 3×3stride(2,1) Dropout 0.5ratio Convolution #class_number 1×13stride 1

[0134] S3 recognizes the license plate in the image: recognizes the license plate information of the vehicle in the image through the weight file of the trained LPRNet model, and verifies whether the license plate information is correct; stores the verified license plate information in the database;

[0135] In step S3, an image is obtained from the video stream collected from the area to be tested, and the image with vehicle data is input into the trained LPRNet model to identify the license plate content of the vehicle and obtain the license plate information; and the identified license plate information is modified into a string form for storage, and the time information of the vehicle appearing in the image is stored at the same time; Figure 4The figure shows the LPRNet network structure; Input represents the input layer, and its corresponding parameters represent that the length and width of the image are 94 pixels and 24 pixels respectively, and the image type is RGB image. Convolution represents that the number of output channels of the convolution layer is 64, and the corresponding convolution is 3*3, with a stride of 1. MaxPooling represents the maximum pooling layer, the number of output channels is 64, the corresponding convolution is 3*3, and the stride is 1; Small basicblock represents a small module, the number of output channels is 128, the corresponding convolution is 3*3, and the stride is 1; Dropout represents a random deactivation function;

[0136] S4 information interaction: interactively combining the geographical location of the image acquisition device with the license plate information and storing them in the database;

[0137] like Figure 6 As shown, the specific steps of step S4 are:

[0138] S41: The information of the image acquisition device defined by the user and the geographical location information of the image acquisition device are obtained; the image acquisition device is a camera; the geographical location information of the camera in step S41 includes the longitude, latitude and geographical location description of the camera; the database is stored in the form of a data table, and the data table includes the vehicle number, corresponding time, corresponding geographical location description, longitude and latitude and license plate number, and storage path; in the entire closed environment, for each camera, its geographical location is fixed. Therefore, the geographical location of the camera is written into the database as a field, and each piece of data in the database is bound to the geographical location of the generated camera. In this way, the recognized license plate can be combined with the geographical location of the camera, and then the geographical location can be integrated into the license plate information; such as Figure 5 As shown in the figure, the first field indicates the license plate number that appears at this moment, the second field is the corresponding time, the third field is the corresponding geographical location description, the fourth field indicates the dimension and the fifth field indicates the latitude, the sixth field indicates the license plate number, and the seventh field indicates the storage path of the detection result. Each time the license plate information is recognized, it is combined with the location information of the point to construct a Figure 5 The table shown;

[0139] S42: interacting the license plate information identified in step S3 with the geographical location information of the image acquisition device to generate an interactive data content of the license plate information and the geographical location information of the image acquisition device;

[0140] S43: storing the interactive data content in a database, that is, binding each piece of license plate information with the geographical location of the crossroads where the data of the license plate information is generated in the database; thus, the corresponding content is written into the database through this step;

[0141] In the database, since each piece of data is geographically bound to the intersection where the data is generated, it can be determined that the vehicle's location is certain at that moment. When the vehicle passes through the second intersection with a camera, a second set of vehicle-location related records is generated, and the record contains the geographic location coordinates. In the database, the coordinate information corresponding to the camera image of the vehicle is associated with the license plate number and stored, and the time factor is taken into account for the time association operation, so the data collection work is completed here;

[0142] S5 Vehicle track recovery: read the database, obtain the license plate information, and associate the point information corresponding to the vehicle, then restore the corresponding track of the vehicle through the multi-point path planning algorithm and display it on the map;

[0143] The specific steps of step S5 are:

[0144] S51: Read the data table in the database, make the database present on the web side, obtain the license plate information, and associate the point information corresponding to the vehicle; in order to present the point on the web side, in the step S51, the corresponding point is displayed through the open source webgis library of Openlayer;

[0145] S52: Restoring the trajectory corresponding to the vehicle through a multi-point path planning algorithm;

[0146] In order to restore the corresponding driving trajectory through the points, whether the road is passable, which lane, etc. are taken into account in the process of restoring the path; the specific steps of step S52 are:

[0147] S521: First, the path planning area is constructed using the road data. The entire road data covers all road network data in the area, and the driving direction and lane line are also distinguished for the corresponding road data;

[0148] S522: In the background, the data information corresponding to the same vehicle is queried through the backend, the point data where the vehicle appears are aggregated, and this series of data is mapped with the road data;

[0149] S523: Using Dijkstra algorithm to recover the trajectory of the points where the vehicle appears in the area;

[0150] The algorithm for path planning through fixed points is similar to the path planning algorithm for business travel problems, so a multi-point path planning algorithm is used to restore the corresponding trajectory of the vehicle; from the starting point to the end point, the way of thinking is to plan the path from the starting point to the end point through the path planning algorithm provided by navigation software such as Amap and Baidu Map. In terms of this feature, the selected solution is similar to the path planning algorithm in the map. First, the path planning is established based on the existing data. The existing data includes road data and monitoring point data in the new district. First, the area of ​​path planning is constructed using road data. The entire road data covers all road network data in the new district, and the driving direction and lane line are also distinguished for the corresponding road data. Such road data more accurately reflects the driving trajectory of the vehicle, and can better simulate the corresponding running trajectory of the vehicle in the later trajectory prediction. In the background, the data information corresponding to the same vehicle is queried through the backend, and the point data of the vehicle is aggregated together, and this series of data is mapped with the road data. At this point, the mapping of vehicle data and road network data (road data) is completed, and the next work is to restore the path according to the corresponding points. As far as the path recovery problem is concerned, in a relatively complex road network, the path planning algorithm of the commercial map is taken into account, and Dijkstra is used to restore the trajectory of the points where the vehicle appears. Since some road networks have certain errors, some paths can be preprocessed. The Dijkstra algorithm is an algorithm for finding the shortest path in a weighted graph and is one of the classic algorithms for solving the single-source shortest path problem. The goal of the Dijkstra algorithm is to find the shortest path from a vertex to all other vertices in the graph. The algorithm uses a greedy strategy to gradually determine the shortest path for each vertex until the shortest path to the target vertex is found;

[0151] The specific steps of step S523 are:

[0152] S5231 Initialization: Mark the starting vertex as having a distance of 0 and other vertices as infinity; create an empty priority queue (usually implemented using a minimum heap) to select the next vertex to explore;

[0153] S5232 Select the shortest path vertex: select the vertex with the shortest distance from the priority queue in step S531 as the current vertex;

[0154] S5233 Update the distance of adjacent vertices: For each adjacent vertex of the current vertex, calculate the distance from the starting vertex through the current vertex to the adjacent vertex; if the path through the current vertex is shorter than the known shortest path, update the distance of the adjacent vertex;

[0155] S5234 marks the current vertex: marks the current vertex as visited; repeats steps S532 and S533 until all vertices are traversed or the shortest path to the target vertex is found;

[0156] S5235 Final result: When the traversal is completed, the shortest path length of each vertex is calculated; by backtracking from the target vertex to the starting vertex, the specific path of the shortest path is obtained.

[0157] S53: Display the corresponding points through an open source solution so that the points are displayed on the map end;

[0158] In step S53, the corresponding points are displayed through the webgis library of Openlayer open source; in step S53, the corresponding arrow direction of the restored path is displayed on the map side; and related operations, etc. Since there are multiple possible results in planning, there are certain differences from the final path planning, so it is necessary to plan through the path of multiple points so that the final restored path can be similar to the real path.

[0159] The predicted results are as follows Figure 8 As shown in the figure above; assuming that the path in the figure above is generated by vehicle A, the colors of the different icons in the figure above represent vehicles at different times. The green mark indicates the point where the vehicle first appears, the red mark indicates the point where the vehicle last appears, the blue mark with the number 1 indicates the point where the vehicle appears for the second time, and the blue mark with the number 2 indicates the point where the vehicle appears for the third time. According to the latitude and longitude of the above positions, the corresponding vehicle trajectory information is drawn on the map.

[0160] Assumptions Fig. 9 The color of the different icons in the figure above represents the vehicles at different times. The green icon indicates the first time the camera captures the vehicle position, the red icon indicates the last time the camera captures the vehicle's track, the blue icon with the number 1 indicates the second time the camera captures the vehicle position, the blue icon with the number 2 indicates the third time the camera captures the vehicle position, and the blue icon with the number 3 indicates the fourth time the camera captures the vehicle position. According to the latitude and longitude of the above positions, the corresponding vehicle track information is drawn on the map.

[0161] For ordinary technicians in this field, the specific embodiments are only illustrative descriptions of the present invention. It is obvious that the specific implementation of the present invention is not limited to the above-mentioned methods. As long as various non-substantial improvements are made using the method concepts and technical solutions of the present invention, or the concepts and technical solutions of the present invention are directly applied to other occasions without improvement, they are all within the protection scope of the present invention.

Claims

1. A method for vehicle trajectory recovery based on computer vision, characterized in that: The specific steps include: S1 obtains the vehicle: collects the video stream and obtains the vehicle in the image; S2 builds the LPRNet network: builds the LPRNet network and uses the dataset to train the weight file of the LPRNet model; S3 recognizes the license plate in the image: recognizes the license plate information of the vehicle in the image through the weight file of the trained LPRNet model, and verifies whether the license plate information is correct; stores the verified license plate information in the database; S4 information interaction: interactively combining the geographical location of the image acquisition device with the license plate information and storing them in the database; S5 vehicle trajectory Recovery: Read the database, obtain the license plate information, and associate the point information corresponding to the vehicle. Then, restore the corresponding trajectory of the vehicle through the multi-point path planning algorithm and display it on the map.

2. The method for vehicle trajectory recovery based on computer vision according to claim 1, characterized in that: The specific steps of step S1 are: S11: collecting a corresponding intersection video stream by an image acquisition device arranged at the intersection; S12: Acquire an image through the collected video stream, and assign the geographical location information of the image acquisition device to the image; S13: Acquire vehicle data from the image to form a data set; and then divide the data set into a training set and a validation set.

3. The method for vehicle trajectory recovery based on computer vision according to claim 2, characterized in that: The LPRNet network constructed in step S2 includes a positioning network and a recognition network, the positioning network is used to locate the license plate area, and the recognition network is used to recognize the license plate characters; the structure of the LPRNet network specifically includes an input layer, a convolutional layer, a small basic block, a maximum pooling layer and an activation function; The specific steps of using the data set to train the LPRNet model in step S2 are: S21 data preprocessing: first perform image enhancement on the image, then use one-hot encoding or character index encoding; S22 defines the loss function: The LPRNet model uses the CTC loss function, the formula is: Where: Π is the set of all possible character sequences; P(π|x) is the probability of sequence π given input x; S23 Network training: When training the LPRNet model, the network's goal is to minimize the CTC loss function; the training steps are: S231 forward propagation: The input image is forward propagated through the network and the probability distribution of the license plate characters is output; S232 Loss calculation: Calculate the difference between the model output and the true label according to the CTC loss function; S233 Back propagation: Calculate the gradient through back propagation and update the network weights; S234 Optimization: Use an optimization algorithm to update model parameters to minimize the loss function; the formula is: Where: θ t is the current parameter; η is the learning rate; is the gradient of the loss function; S24 Validation and tuning: During the training process, the validation set is used for validation, the accuracy of the model is evaluated, and tuning is performed based on the validation results.

4. The method for vehicle trajectory recovery based on computer vision according to claim 2, characterized in that: In step S3, an image is obtained from the video stream collected from the area to be tested, and the image with vehicle data is input into the trained LPRNet model to identify the license plate content of the vehicle. After obtaining the license plate information, the license plate number rule is verified by the license plate verification regular expression. If correct, the recognized license plate information is modified into a string form for storage, and the time information of the vehicle appearing in the image is stored at the same time.

5. The method for vehicle trajectory recovery based on computer vision according to claim 4, characterized in that: The specific steps of step S4 are: S41: information of the image acquisition device defined by the user and obtaining geographical location information of the image acquisition device; S42: interacting the license plate information identified in step S3 with the geographical location information of the image acquisition device to generate an interactive data content of the license plate information and the geographical location information of the image acquisition device; S43: storing the interactive data content in a database, that is, binding each piece of license plate information with the geographical location of the intersection where the data of the license plate information is generated in the database.

6. The method for vehicle trajectory recovery based on computer vision according to claim 5, characterized in that: The image acquisition device is a camera; the geographic location information of the camera in step S41 includes the longitude, latitude and geographic location description of the camera; the database is stored in the form of a data table, and the data table includes the vehicle number, corresponding time, corresponding geographic location description, longitude and latitude and license plate number, and storage path.

7. The method for vehicle trajectory recovery based on computer vision according to claim 6, characterized in that: The specific steps of step S5 are: S51: Read the data table stored after the algorithm is executed in the database, present the corresponding points in the database on the web, obtain the license plate information, and associate the point information corresponding to the vehicle; S52: Restoring the trajectory corresponding to the vehicle through a multi-point path planning algorithm; S53: A web page is written through an open source development library to load corresponding points on the map for display on the map, so that the points are displayed on the map end.

8. The method for vehicle trajectory recovery based on computer vision according to claim 7, characterized in that: The specific steps of step S52 are: S521: First, the path planning area is constructed using the road data. The entire road data covers all road network data in the area, and the driving direction and lane line are also distinguished for the corresponding road data; S522: In the background, the data information corresponding to the same vehicle is queried through the backend, the point data where the vehicle appears are aggregated, and this series of data is mapped with the road data; S523: Using the Dijkstra algorithm to recover the trajectory of the points where the vehicle appears in the area.

9. The method for vehicle trajectory recovery based on computer vision according to claim 8, characterized in that: The specific steps of step S523 are: S5231 Initialization: Mark the starting vertex as having a distance of 0 and other vertices as infinity; Create an empty priority queue for selecting the next vertex to explore; S5232 Select the shortest path vertex: select the vertex with the shortest distance from the priority queue in step S531 as the current vertex; S5233 Update the distance between adjacent vertices: For each adjacent vertex of the current vertex, calculate the distance from the starting vertex through the current vertex to the adjacent vertex; If the path through the current vertex is shorter than the known shortest path, update the distance of the adjacent vertex; S5234 marks the current vertex: marks the current vertex as visited; repeats steps S532 and S533 until all vertices are traversed or the shortest path to the target vertex is found; S5235 Final result: When the traversal is completed, the shortest path length of each vertex is calculated; by backtracking from the target vertex to the starting vertex, the specific path of the shortest path is obtained.

10. The method for vehicle trajectory recovery based on computer vision according to claim 9, characterized in that: In both step S51 and step S53, the corresponding points are displayed through the open source webgis library of Openlayer; in step S53, the corresponding arrow direction of the restored path is displayed on the map side.

Images