A vehicle tracking method based on multiple cameras

Abstract

The present invention discloses a vehicle tracking method based on multiple cameras, comprising: obtaining road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and constructing a spatiotemporal topological relationship of a camera network; obtaining a target vehicle and tracking the target vehicle in a single camera; when the target vehicle disappears from the single camera, according to the relative motion type, vehicle driving direction and time information between the target vehicle and the single camera, searching for adjacent nodes of the single camera that meet constraint conditions as a candidate node set in the spatiotemporal topological relationship of the camera network; obtaining a license plate, a portrait and deep learning features of the target vehicle, and associating and identifying the license plate, the vehicle portrait and the deep learning features of the vehicle set obtained by the candidate node set with the target vehicle, and obtaining a target vehicle cross-camera tracking result. The method has a small amount of calculation and can obtain a reliable and robust target vehicle continuous tracking result.

Description

Technical Field

[0001] The invention relates to the technical field of video monitoring, and in particular to a vehicle tracking method based on multiple cameras. Background Art

[0002] With the widespread application of video surveillance technology, its mode has evolved from the isolated use of a single camera to a large-scale video surveillance network, achieving a wider monitoring range and more diverse observation angles, and being able to capture target information more comprehensively. At present, the application potential of video surveillance in public safety, intelligent transportation, etc. has attracted great attention from government departments and related industries. More and more high-definition cameras are installed at key roads and intersections to obtain and count various types of traffic data, which plays a great role in combating crime, capturing targets, and traffic management for public security and traffic management departments. Among them, the analysis of suspects' action routes and the monitoring of illegal vehicles' driving routes are one of the important application directions.

[0003] At present, although there have been several studies on multi-camera target tracking, there is no special mature method for the problem of vehicle motion tracking between multiple cameras. On the one hand, the camera topological relationship model only considers simple scenes with narrow roads and fewer target motion patterns, and cannot be applied to modeling in complex urban road structures and vehicle motion tracking. It also does not involve complex feature selection problems in vehicle tracking and cannot accurately and continuously track vehicles. On the other hand, when the existing technology tracks vehicles across cameras, it generally adopts a re-identification method, uses a deep learning model to extract features from query images and gallery sets, and then uses a certain metric method to calculate the feature distance between the query image and the image in the gallery set, and judges whether it is the same target by the size of the distance. This method does not use road data, the topological relationship between cameras, and vehicle motion information. Not only is the amount of calculation large, but it is also difficult to obtain reliable and robust tracking results when there are differences in resolution, lighting, and viewing angles of multiple cameras.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the invention

[0005] The technical problem to be solved by the present invention is that, in view of the above-mentioned defects of the prior art, a vehicle tracking method based on multiple cameras is provided, aiming to solve the problem that when multiple cameras are used for vehicle tracking in the prior art, road data, topological relationships between cameras and vehicle motion information are not utilized, which not only has a large amount of calculation, but also it is difficult to obtain reliable and robust tracking results when there are differences in resolution, lighting, viewing angle, etc. of multiple cameras.

[0006] The technical solution adopted by the present invention to solve the problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a vehicle tracking method based on multiple cameras, wherein the method comprises:

[0008] Obtain road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and build the spatiotemporal topological relationship of the camera network;

[0009] Acquire a target vehicle, and track the target vehicle within a single camera;

[0010] When the target vehicle disappears from the single camera, according to the relative motion type between the target vehicle and the single camera, the vehicle driving direction and time information, the adjacent nodes of the single camera that meet the constraint conditions are searched in the spatiotemporal topological relationship of the camera network as a candidate node set;

[0011] The target vehicle's license plate, target vehicle's portrait and deep learning features are obtained, and the license plate, vehicle portrait and deep learning features of the vehicle set obtained by the candidate node set are associated with the target vehicle for identification to obtain the target vehicle's cross-camera tracking result.

[0012] In one implementation, the acquiring of road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and constructing a spatiotemporal topological relationship of a camera network includes:

[0013] Obtain road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data;

[0014] Use the camera as a node;

[0015] For each of the nodes, the nearest nodes in each direction around it or the nodes with the shortest reachable paths in each travel direction are called adjacent nodes;

[0016] Taking the movement transfer pattern of the vehicle between each node and the adjacent nodes corresponding to the node as a directed edge;

[0017] Theoretical estimation and statistical analysis are performed based on camera installation parameters, geographic information data, historical traffic flow data, and driving record data of buses and taxis to obtain the time transfer relationship between the vehicle at each node and the adjacent nodes corresponding to the node;

[0018] According to the position change relationship of the vehicle relative to the node and the driving direction information of the vehicle before disappearing in the node, the connection relationship attribute information corresponding to each edge is obtained; wherein the connection relationship attribute information includes the relative motion type between the vehicle and the node and the driving direction information of the vehicle;

[0019] The nodes, the directed edges, the time transfer relations and the connection relation attributes are combined into a spatiotemporal topological relationship of a camera network.

[0020] In one implementation, for each of the nodes, the nodes closest to each other in each direction around the node or the nodes with the shortest reachable path in each travel direction are called adjacent nodes, including:

[0021] When the nodes are densely distributed, the nodes closest to the node in all travel directions in the monitoring area corresponding to the node are selected as adjacent nodes;

[0022] When the nodes are sparsely distributed, the nodes corresponding to the node and having the shortest paths thereto in all travel directions are selected as adjacent nodes, and only one adjacent node is selected for each road or intersection.

[0023] In one implementation, the step of using the movement transfer pattern of the vehicle between each node and the adjacent node corresponding to the node as a directed edge includes:

[0024] When the nodes are densely distributed, determining a movement transfer mode of a vehicle between the node and the adjacent nodes corresponding to the node according to the installation positions of the node and the adjacent nodes corresponding to the node, a monitoring area, known road structure information, and traffic rule constraints;

[0025] When the nodes are sparsely distributed, determining the movement transfer mode of the vehicle between the node and the adjacent nodes corresponding to the node according to the optimal path planning provided by the geographic information system;

[0026] The movement transfer pattern of the vehicle between each node and the adjacent nodes corresponding to the node is taken as a directed edge.

[0027] In one implementation, the step of acquiring road structure and traffic attributes, camera installation parameters, historical traffic data, and geographic information data, and constructing a spatiotemporal topological relationship of a camera network further includes:

[0028] A graph model is used to represent the spatiotemporal topological relationship of the camera network, and the expression is:

[0029] G=(V,E)

[0030] V={v i |1≤i≤N}

[0031] E={e(i,j)|1≤i,j≤N}

[0032] In the formula, V represents the node set, v iis the i-th node in the camera network topology graph, E represents the edge set between the node and its adjacent nodes, including a series of ordered node pairs, and e(i,j) is the edge from node v i To node v j The transition relationship is an edge. When there is a transition from the node v i to the node v j When the driving path is the same, e(i,j)=1, otherwise, e(i,j)=0, and because the edge has directionality, e(i,j) and e(j,i) are not necessarily equal; for each pair of transfer relations e(i,j), three attribute information are also included: the transfer time range, the flag bit F P The connection relationship between the vehicle and the node v is represented by i The relative motion type between the two and the flag F a The vehicle driving direction corresponding to the connection relationship is represented by .

[0033] In one implementation, the acquiring of the target vehicle and tracking of the target vehicle in a single camera includes:

[0034] Get the target vehicle;

[0035] According to the target vehicle, the YoloV4 target detection algorithm is used to detect the target vehicle in the video, and the detection time, vehicle type, number and image coordinates of the detection frame position are output;

[0036] According to the detection time, vehicle type, number, and image coordinates of the detection frame position, the Deep Sort target tracking algorithm is used to continuously track the movement of the target vehicle to obtain the target vehicle movement trajectory, wherein the target vehicle movement trajectory includes the target vehicle trajectory number, the detection time of each sampling point corresponding to the trajectory number, and the detection frame position coordinates;

[0037] Acquire the relative motion type between the target vehicle and the single camera; wherein the relative motion type includes the target vehicle approaching, moving away, or first approaching and then moving away from the single camera;

[0038] Extracting the target vehicle's driving direction indication information to obtain the target vehicle's driving direction; wherein the target vehicle's driving direction indication information includes lane markings, vehicle tracks, and vehicle indicator light flashing information;

[0039] Obtain a high-quality vehicle image of the target vehicle, and extract the license plate and vehicle portrait information of the target vehicle; wherein the vehicle portrait information includes vehicle model, vehicle brand and model, body color, objects or ornaments placed in the vehicle, whether there is a sunroof, whether there is a roof rack, and whether there is someone in the passenger seat.

[0040] In one implementation, when the target vehicle disappears from the single camera, searching for adjacent nodes of the single camera that meet the constraint conditions as a candidate node set in the camera network spatiotemporal topology relationship according to the relative motion type, vehicle driving direction and time information between the target vehicle and the single camera includes:

[0041] When the target vehicle disappears from the single camera, in the spatiotemporal topological relationship of the camera network, a set of downstream adjacent nodes within the transfer time range between the single camera and its downstream adjacent nodes, whose time difference between the target association calculation time and the vehicle disappearance time, is taken as a first candidate node set, and a connection relationship set corresponding to the first candidate node set is taken as a first candidate connection relationship set; wherein the first candidate connection relationship set includes directed edges connecting the single camera and nodes in the first candidate node set and attribute information of the directed edges;

[0042] According to the relative motion type between the target vehicle and the single camera, the first candidate node set and the first candidate connection relationship set are screened to obtain a second candidate node set and a second candidate connection relationship set;

[0043] According to the driving direction of the target vehicle before disappearing from the single camera, the second candidate node set and the second candidate connection relationship set are screened to obtain a candidate node set.

[0044] In one implementation, the step of acquiring the target vehicle license plate, the target vehicle portrait, and the deep learning features, associating and identifying the license plate, the vehicle portrait, and the deep learning features of the vehicle set acquired by the candidate node set with the target vehicle, and obtaining the target vehicle cross-camera tracking result includes:

[0045] Obtain the target vehicle license plate and target vehicle portrait;

[0046] Obtaining license plate and vehicle image information of the vehicle photographed by the candidate node set;

[0047] When both the target vehicle and the vehicle photographed by the candidate node set contain license plate information, the license plate of the vehicle photographed by the candidate node set is associated with the license plate of the target vehicle. When the license plate of the vehicle photographed by the candidate node set is the same as the license plate of the target vehicle, the association is judged to be successful, the corresponding node ID is recorded, and the transfer route of the target vehicle between the nodes is updated;

[0048] When the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, a first vehicle association score set of the target vehicle is obtained according to the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set;

[0049] A target vehicle cross-camera tracking result is obtained according to the first vehicle association score set.

[0050] In one implementation, when the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, obtaining the first vehicle association score set of the target vehicle according to the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set includes:

[0051] When the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, obtaining a set of valid vehicle portrait information numbers according to the vehicle portrait information of the target vehicle and the vehicle photographed by the candidate node set;

[0052] According to the consistency between the target vehicle image and the vehicle image information of the candidate node set, a set of the number of associated vehicle image information is obtained;

[0053] The first vehicle association score set of the target vehicle is obtained by dividing the associated vehicle portrait information number set by the valid vehicle portrait information number set.

[0054] In one implementation, obtaining a target vehicle cross-camera tracking result according to the first vehicle association score set includes:

[0055] Get the target vehicle image;

[0056] Obtain vehicle pictures taken by the candidate node set;

[0057] The BoT-BS algorithm is used to perform vehicle association recognition, and deep learning feature extraction and similarity calculation are performed on the target vehicle image and the vehicle images taken by the candidate node set to obtain a second vehicle association score set of the target vehicle;

[0058] A target vehicle cross-camera tracking result is obtained according to the first vehicle association score set and the second vehicle association score set.

[0059] In one implementation, obtaining a target vehicle cross-camera tracking result according to the first vehicle association score set and the second vehicle association score set includes:

[0060] Obtaining a vehicle target association total score set according to the first vehicle association score set and the second vehicle association score set;

[0061] When the total association score of the vehicle target association total score set is the largest and the number of consecutive associations to the vehicle with the same track number under the same adjacent node meets the preset value, it is determined that the candidate vehicle in the candidate node set is successfully associated with the target vehicle;

[0062] When the candidate vehicles in the candidate node set are successfully associated with the target vehicle, the corresponding node IDs are recorded, and the transfer routes of the target vehicle between the nodes are updated.

[0063] In one implementation, the step of acquiring the target vehicle license plate, the target vehicle portrait, and the deep learning features, associating and identifying the license plate, the vehicle portrait, and the deep learning features of the vehicle set acquired by the candidate node set with the target vehicle, and obtaining the target vehicle cross-camera tracking result further includes:

[0064] The current node where the target vehicle is located obtained by cross-camera tracking is used as the new single camera, and the single-camera tracking and cross-camera tracking of the target vehicle are continued, and the motion trajectory of the target vehicle obtained by the single-camera tracking is updated into the target vehicle driving trajectory record;

[0065] According to the transfer route of the target vehicle between the nodes, the driving track record of the target vehicle, and the installation location of the nodes, the driving time period and driving route of the target vehicle are obtained with the assistance of a geographic information system.

[0066] In a second aspect, an embodiment of the present invention further provides a vehicle tracking device based on multiple cameras, wherein the device comprises:

[0067] The camera network spatiotemporal topological relationship construction unit is used to obtain road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and construct the camera network spatiotemporal topological relationship;

[0068] A target vehicle motion trajectory acquisition unit, used to track the target vehicle within a single camera according to the target vehicle;

[0069] a candidate node set acquisition unit, configured to search, when the target vehicle disappears from the single camera, for adjacent nodes of the single camera that meet the constraint conditions in the camera network spatiotemporal topological relationship according to the relative motion type, vehicle driving direction and time information between the target vehicle and the single camera as a candidate node set;

[0070] The target vehicle cross-camera tracking processing unit is used to associate and identify the license plate, vehicle portrait and deep learning features of the vehicle set obtained by the candidate node set with the target vehicle based on the acquired target vehicle license plate, target vehicle portrait and deep learning features, so as to obtain the target vehicle cross-camera tracking result.

[0071] In a third aspect, an embodiment of the present invention further provides a server comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, wherein the one or more programs include a method for executing a multi-camera-based vehicle tracking method as described in any one of the above.

[0072] In a fourth aspect, an embodiment of the present invention further provides a non-temporary computer-readable storage medium, which, when instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to execute any of the multi-camera based vehicle tracking methods described above.

[0073] Beneficial effects of the present invention: The embodiment of the present invention first obtains road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and constructs a camera network spatiotemporal topological relationship; then obtains the target vehicle and tracks the target vehicle in a single camera; then when the target vehicle disappears from the single camera, according to the relative motion type, vehicle driving direction and time information between the target vehicle and the single camera, the adjacent nodes of the single camera that meet the constraint conditions are searched as a candidate node set in the camera network spatiotemporal topological relationship; finally, the target vehicle license plate, target vehicle portrait and deep learning features are obtained, and the license plate, vehicle portrait and deep learning features of the vehicle set obtained by the candidate node set are associated and identified with the target vehicle to obtain the target vehicle cross-camera tracking result; it can be seen that the vehicle tracking method based on multiple cameras in the embodiment of the present invention can realize continuous tracking of the target vehicle, and the method has a small amount of calculation and can obtain reliable and robust tracking results. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0075] Figure 1 A schematic flow chart of a vehicle tracking method based on multiple cameras provided in an embodiment of the present invention.

[0076] Figure 2 A functional block diagram of a vehicle tracking device based on multiple cameras provided in an embodiment of the present invention.

[0077] Figure 3 A block diagram of the internal structure principle of a server provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0078] The present invention discloses a vehicle tracking method based on multiple cameras. In order to make the purpose, technical solution and effect of the present invention clearer and more specific, the present invention is further described in detail with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0079] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein includes all or any unit and all combinations of one or more associated listed items.

[0080] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in general dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless specifically defined as herein.

[0081] At present, although there have been several studies on multi-camera target tracking, there is no special mature method for the problem of vehicle motion tracking between multiple cameras. On the one hand, the camera topological relationship model only considers simple scenes with narrow roads and fewer target motion patterns, and cannot be applied to modeling in complex urban road structures and vehicle motion tracking. It also does not involve complex feature selection problems in vehicle tracking and cannot accurately and continuously track vehicles. On the other hand, when the existing technology tracks vehicles across cameras, it generally adopts a re-identification method, uses a deep learning model to extract features from query images and gallery sets, and then uses a certain metric method to calculate the feature distance between the query image and the image in the gallery set, and judges whether it is the same target by the size of the distance. This method does not use road data, the topological relationship between cameras, and vehicle motion information. Not only is the amount of calculation large, but it is also difficult to obtain reliable and robust tracking results when there are differences in resolution, lighting, and viewing angles of multiple cameras.

[0082] In order to solve the problems of the prior art, this embodiment provides a vehicle tracking method based on multiple cameras, by which the target vehicle can be accurately tracked for a long time and a long distance, and the method has a small amount of calculation and can obtain reliable and robust tracking results. In the specific implementation, the road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data are first obtained, and the spatiotemporal topological relationship of the camera network is constructed to prepare for the subsequent cross-camera target tracking; then, the target vehicle is obtained and tracked in a single camera. In practice, it is necessary to obtain the continuously updated motion trajectory of the target vehicle tracked by all single cameras; when the target vehicle disappears from a single camera, according to the relative motion type, vehicle driving direction and time information between the target vehicle and the single camera, the adjacent nodes of the single camera that meet the constraints are searched in the spatiotemporal topological relationship of the camera network as a candidate node set, so as to narrow the candidate node search range as much as possible and improve the cross-camera tracking speed; finally, the license plate, image and deep learning features of the target vehicle are obtained, and the license plate, vehicle image and deep learning features of the vehicle set obtained by the candidate node set are associated and identified with the target vehicle to obtain the cross-camera tracking result of the target vehicle, which can realize continuous tracking of the target vehicle, and the method has a small amount of calculation and can obtain reliable and robust tracking results.

[0083] Example

[0084] With the widespread application of video surveillance technology, its mode has evolved from the isolated use of a single camera to a large-scale video surveillance network, achieving a wider monitoring range and more diverse observation angles, and being able to capture target information more comprehensively. At present, the application potential of video surveillance in public security, intelligent transportation, etc. has attracted great attention from government departments and related industries. Installers have installed more and more high-definition cameras at key roads and intersections to obtain and count various types of traffic data, which plays a great role in combating crime, capturing targets, and traffic management for public security and traffic management departments. Among them, the analysis of suspects' action routes and the monitoring of illegal vehicles' driving routes are one of the important application directions. The acquisition of vehicle driving routes requires the completion of vehicle target tracking within the field of view of a single camera and the association of vehicle targets across cameras. Target tracking within a single camera has achieved good research results, but there is no mature solution for cross-camera target association. Due to the differences in working parameters, imaging quality, installation angle, surrounding environment, etc. of different cameras, the appearance characteristics of vehicles captured by different cameras may be very different, resulting in poor cross-camera target association in the existing technology. Therefore, how to effectively utilize the spatiotemporal topological relationship between cameras, associate the same vehicle target data captured by different cameras, and achieve continuous tracking of the target vehicle is a key issue that needs to be solved in vehicle route analysis. In this embodiment, the road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data are first obtained, and the camera network spatiotemporal topological relationship is constructed to prepare for the subsequent cross-camera target vehicle association; then the target vehicle is obtained and the target vehicle is tracked in a single camera; then when the target vehicle disappears from the single camera, the adjacent nodes of the single camera that meet the constraint conditions are searched as candidate node sets in the camera network spatiotemporal topological relationship according to the relative motion type, vehicle driving direction and time information between the target vehicle and the single camera, so as to further narrow the target vehicle association range; finally, the target vehicle license plate, target vehicle portrait and deep learning features are obtained, and the license plate, vehicle portrait and deep learning features of the vehicle set obtained by the candidate node set are associated and identified with the target vehicle to obtain the target vehicle cross-camera tracking result, so that the vehicle tracking method based on multiple cameras of the present invention can realize continuous tracking of the target vehicle, and the method has a small amount of calculation and can obtain reliable and robust tracking results.

[0085] Exemplary Methods

[0086] This embodiment provides a vehicle tracking method based on multiple cameras, which can be applied to a server. Figure 1 As shown, the method includes:

[0087] Step S100: Obtain road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and construct a spatiotemporal topological relationship of the camera network;

[0088] Specifically, for urban road traffic scenes, cameras are usually installed on both sides of the road or at intersections to monitor the traffic status of roads and intersections. The spatiotemporal topological relationship of the camera network is closely related to the road topology and its properties. In order to realize vehicle tracking by multiple cameras, it is necessary to obtain road structure and traffic properties, camera installation parameters, historical traffic data and geographic information data, and construct the spatiotemporal topological relationship of the camera network.

[0089] In order to obtain the spatiotemporal topological relationship of the camera network, the acquisition of road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and the construction of the spatiotemporal topological relationship of the camera network include the following steps:

[0090] Step S101, obtaining road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data;

[0091] Step S102, taking the camera as a node;

[0092] Step S103: for each of the nodes, the nodes closest to it in each direction or the nodes with the shortest reachable paths in each travel direction are called adjacent nodes;

[0093] Step S104, taking the movement transfer pattern of the vehicle between each node and the adjacent node corresponding to the node as a directed edge;

[0094] Step S105, performing theoretical estimation and statistical analysis based on camera installation parameters, geographic information data, historical traffic flow data, and driving record data of buses and taxis to obtain the time transfer relationship between the vehicle at each node and the adjacent nodes corresponding to the node;

[0095] Step S106: obtaining connection relationship attribute information corresponding to each edge according to the position change relationship of the vehicle relative to the node and the driving direction information of the vehicle before disappearing from the node; wherein the connection relationship attribute information includes the relative motion type between the vehicle and the node and the driving direction information of the vehicle;

[0096] Step S107: The nodes, the directed edges, the time transfer relations and the connection relation attributes are combined into a spatiotemporal topological relationship of a camera network.

[0097] Specifically, in order to construct the spatiotemporal topological relationship of the camera network, this embodiment first needs to obtain the road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data; after obtaining the road structure and traffic attributes, this embodiment can obtain the type, quantity and traffic rules of urban roads, obtain the camera installation parameters, know the camera's position, posture and monitoring range, obtain historical traffic data to estimate the time of vehicle travel, and obtain geographic information data to assist in providing vehicle travel path planning information, which are all the basis for constructing the spatiotemporal topological relationship of the camera network. Next, take the camera as a node, and then determine the adjacent nodes of each node. For each of the nodes, the nearest node in each direction around it or the node with the shortest reachable path in each traffic direction is called an adjacent node.

[0098] In order to obtain adjacent nodes, for each node, the node closest to it in each direction or the node with the shortest reachable path in each travel direction is called an adjacent node, which includes the following steps: when the nodes are densely distributed, the node closest to it in all travel directions in the monitoring area corresponding to the node is selected as the adjacent node; when the nodes are sparsely distributed, the node with the shortest reachable path in all travel directions corresponding to the node is selected as the adjacent node, and only one adjacent node is selected for each road or intersection.

[0099] Specifically, considering that there are two situations of sparse and dense node distribution in reality, there are also two situations of obtaining adjacent nodes: when the nodes are densely distributed, the nodes closest to the nodes in all traffic directions in the monitoring area corresponding to the nodes are selected as adjacent nodes. At this time, several nodes are installed on each intersection and road, and the path of a vehicle disappearing from the field of view of one node and appearing in the field of view of another node is uniquely determined; when the nodes are sparsely distributed, the nodes with the shortest paths to the nodes in all traffic directions corresponding to the nodes are selected as adjacent nodes, and only one adjacent node is selected for each road or intersection. At this time, the nodes are separated by several intersections or long distances, the node monitoring areas do not overlap and are far apart, and the path of a vehicle disappearing from the field of view of one node and appearing in the field of view of another node is not unique.

[0100] After establishing the nodes and adjacent nodes, it is also necessary to establish the connection relationship between the nodes, that is, the directed edge, and use the movement transfer mode of the vehicle between each node and the adjacent node corresponding to the node as the directed edge. Accordingly, using the movement transfer mode of the vehicle between each node and the adjacent node corresponding to the node as the directed edge includes the following steps: when the nodes are densely distributed, the movement transfer mode of the vehicle between the node and the adjacent node corresponding to the node is determined according to the installation position, monitoring area, known road structure information and traffic rule constraints of the node and the adjacent node corresponding to the node; when the nodes are sparsely distributed, the movement transfer mode of the vehicle between the node and the adjacent node corresponding to the node is determined according to the optimal path planning provided by the geographic information system; the movement transfer mode of the vehicle between each node and the adjacent node corresponding to the node is used as a directed edge.

[0101] For example, when the nodes are densely distributed, the vehicle's motion transfer mode can be determined based on the node installation location, monitoring area, known road structure information and traffic rule constraints. For example, the vehicle's motion transfer mode can be determined based on whether the vehicle is traveling on a one-way or two-way road, whether the vehicle is located in a straight or turning lane, or a U-turn lane when traveling, in combination with the node installation location and monitoring area. In addition, for road areas where U-turns are allowed, the nodes also have a self-connection relationship, that is, after the vehicle disappears from the current node, it appears in the field of view of the current node in a very short time because of the U-turn. When the nodes are sparsely distributed, the bidirectional transfer relationship between the vehicle and its adjacent nodes can be determined based on the optimal path planning provided by the geographic information system (such as Baidu Map or Gaode Map), and the node and its adjacent nodes are connected according to the transfer direction. In this way, the connection relationship and directionality between each node and its adjacent nodes are determined, that is, the directed edges in the spatiotemporal topological relationship diagram of the camera network.

[0102] After obtaining the directed edges, theoretical estimation and statistical analysis are also required based on the camera installation parameters, geographic information data, historical traffic flow data, and driving record data of buses and taxis to obtain the time transfer relationship between the vehicle at each node and the adjacent nodes corresponding to the node. Specifically, statistical analysis and estimation can be performed using node installation parameters, traffic light control schemes, historical traffic flow data (vehicle speed, delay, etc.), and driving record data of service vehicles such as buses and taxis, and geographic information systems can also be used for assistance to determine the time range for vehicle transfers between each node and its adjacent nodes. For example, in the case of dense node distribution, since the distance between a node and its adjacent nodes is relatively close, its transfer time is usually very short and can be accurately estimated. At this time, the time range of the transfer between a node and its adjacent nodes can be obtained by combining the node location information and the signal light control scheme based on historical data statistics or theoretical estimation under certain vehicle driving state assumptions; in the case of sparse node distribution, since the distance between a node and its adjacent nodes is relatively far or there are several intersections between them, there are multiple recommended travel paths between the two. At this time, the travel time corresponding to the top three optimal path planning results output by the integrated geographic information system can be used to estimate the final time range of the transfer between the node and its adjacent nodes. In this way, the time transfer relationship between the vehicle and each of the nodes and the adjacent nodes corresponding to the node is obtained.

[0103] After obtaining the time transfer relationship between the vehicle at each node and the adjacent node corresponding to the node, it is also necessary to obtain the connection relationship attribute information corresponding to each edge according to the position change relationship of the vehicle relative to the node and the driving direction information of the vehicle before disappearing in the node; wherein the connection relationship attribute information includes the relative motion type between the vehicle and the node and the driving direction information of the vehicle. Specifically, each node may include several directed edges pointing to it and directed edges from it to other nodes, indicating the motion transfer mode between it and different adjacent nodes, respectively referred to as input directed edges and output directed edges. Each input directed edge has several corresponding output directed edges, which are related to the position change relationship of the vehicle relative to the node and the driving direction information of the vehicle before disappearing in the node. Therefore, attribute information should also be added to each directed edge (i.e., connection relationship) to assist in the subsequent accurate search for candidate node sets based on the driving information of the target vehicle. For example, for each node, the attribute auxiliary information of the connection relationship pair consisting of each node and its downstream adjacent node mainly includes two aspects: one is the correspondence between the connection relationship pair and the relative motion type between the vehicle and the node, and the other is the correspondence between the connection relationship pair and the driving direction of the vehicle before it disappears in the node. The correspondence between them can be determined according to the node installation position, installation direction, road type (one-way / two-way) and road structure. Among them, the relative motion type between the vehicle and the node can be divided into three situations: one is that the vehicle approaches the node from far to near, the second is that the vehicle moves away from the node from near to far, and the third is that the vehicle approaches the node first and then moves away from the node. The correspondence between the connection relationship pair and the relative motion type between the vehicle and the node is marked with the flag F. P To indicate that when a vehicle approaches the node, the flag bit F of the connection relationship between the node and its downstream adjacent node is P Set to 0; when the vehicle is far away from the node, the flag bit F of the connection relationship between the node and its downstream adjacent node is set P Set to 1; when the vehicle approaches and then moves away from the node, the flag bit F of the connection relationship between the node and its downstream adjacent node is set P Set to 2. It should be noted that the number of values ​​of this flag bit corresponding to each directed edge (i.e., the connection relationship pair) can be 1 or more. The driving direction of the vehicle before it disappears in the node includes straight, turning, U-turn and branch (fork), which can be obtained based on information such as traffic markings, driving lanes and vehicle trajectories. Among them, the branch corresponds to the situation where there are multiple forks in complex structure roads such as expressways, Y-shaped roads, non-cross intersections, etc. The corresponding relationship between the connection relationship pair composed of the node and its downstream adjacent node and the driving direction of the vehicle before it disappears in the node is represented by the flag bit F. a To indicate that the flag bit F aThe relationship between the value of and the vehicle's driving direction is as follows: When the vehicle is driving straight, the flag F a The value is 0; when the vehicle is turning right, the flag F a The value is 1; when the vehicle is turning left, the flag F a The value is 2; when the vehicle is turning, the flag F a The value is 3; when the vehicle is heading to a branch road, the F a The value of is specially marked. In practice, it can be flexibly designed and expanded based on the above ideas according to the characteristics of roads in different regions, and they will not be listed here one by one.

[0104] In this way, after obtaining the nodes, the directed edges, the time transfer relations and the connection relation attributes, the spatiotemporal topological relationship of the camera network is completely constructed by the above elements.

[0105] In order to quantitatively represent the spatiotemporal topological relationship of the camera network, an embodiment of the present invention constructs a graph model, wherein the graph model is constructed based on the temporal topological relationship and spatial topological relationship of the camera network, and includes a set of nodes, a set of edges between nodes, a time transfer relationship from one node to another, and attribute information of the edges; specifically, the graph model corresponding to the spatiotemporal topological relationship of the camera network can be expressed as:

[0106] G=(V,E)

[0107] V={v i |1≤i≤N}

[0108] E={e(i,j)|1≤i,j≤N}

[0109] In the formula, V represents the node set, v i is the i-th node in the camera network topology graph, E represents the edge set between the node and its adjacent nodes, including a series of ordered node pairs, and e(i,j) is the edge from node v i To node v j The transfer relationship is a directed edge. When there is a transition from the node v i to the node v j When the driving path is the same, e(i,j)=1, otherwise, e(i,j)=0, and because the edge has directionality, e(i,j) and e(j,i) are not necessarily equal; for each pair of transfer relations e(i,j), three attribute information are also included: the transfer time range, the flag bit F P The connection relationship between the vehicle and the node v is represented by i The relative motion type between the two and the flag F aThe vehicle driving direction corresponding to the connection relationship is represented by .

[0110] Due to the structural characteristics of urban roads, the graph model has strong sparsity. In order to improve the computational efficiency and storage resource utilization, the graph is represented in the form of an adjacency list.

[0111] After constructing the spatiotemporal topology of the camera network, you can execute the following Figure 1 The steps shown are: step S200, obtaining a target vehicle, and tracking the target vehicle in a single camera;

[0112] Specifically, if the target vehicle is to be tracked for a long time and a long distance, the motion trajectory of the target vehicle in each single camera must first be obtained, and then the motion trajectories of the target vehicle in all the single cameras obtained are connected in series according to the transfer sequence and transfer path between the nodes to form the overall driving trajectory of the target vehicle. Therefore, first, according to the obtained target vehicle information, the target vehicle is tracked in a camera containing the target vehicle to obtain the motion trajectory of the target vehicle in the single camera.

[0113] In order to obtain the motion trajectory of the target vehicle, the step of obtaining the target vehicle and tracking the target vehicle in a single camera includes the following steps:

[0114] Step S201, obtaining a target vehicle;

[0115] Step S202: According to the target vehicle, the target vehicle in the video is detected using the YoloV4 target detection algorithm, and the detection time, vehicle type, number and image coordinates of the detection frame position are output;

[0116] Step S203: According to the detection time, vehicle type, number, and image coordinates of the detection frame position, the DeepSort target tracking algorithm is used to continuously track the movement of the target vehicle to obtain a target vehicle movement trajectory, wherein the target vehicle movement trajectory includes the trajectory number of the target vehicle, the detection time of each sampling point corresponding to the trajectory number, and the detection frame position coordinates;

[0117] Step S204, obtaining the relative motion type between the target vehicle and the single camera; wherein the relative motion type includes the target vehicle approaching, moving away, or first approaching and then moving away from the single camera;

[0118] Step S205, extracting the target vehicle driving direction indication information to obtain the driving direction of the target vehicle; wherein the target vehicle driving direction indication information includes lane markings, vehicle tracks, and vehicle indicator light flashing information;

[0119] Step S206, obtaining a high-quality vehicle image of the target vehicle, and extracting the license plate and vehicle portrait information of the target vehicle; wherein the vehicle portrait information includes vehicle model, vehicle brand and model, body color, objects or ornaments placed in the vehicle, whether there is a sunroof, whether there is a roof rack, and whether there is someone in the passenger seat.

[0120] In this embodiment, the target vehicle is first acquired, and the YoloV4 target detection algorithm is used to detect the vehicle in the video according to the target vehicle, and the detection time, vehicle type, number and image coordinates of the detection frame position are output. Then, according to the detection time, vehicle type, number and image coordinates of the detection frame position, the Deep Sort target tracking algorithm is used to continuously track the movement of the target vehicle to obtain the target vehicle motion trajectory, wherein the target vehicle motion trajectory includes the trajectory number of the target vehicle, the detection time and the detection frame position coordinates of each sampling point corresponding to the trajectory number. Then, the relative motion type between the target vehicle and the single camera is obtained; wherein, the relative motion type includes the target vehicle approaching, moving away, or first approaching and then moving away from the single camera. For example, without adjusting the node lens parameters, the relative motion type between the target vehicle and the single camera can be determined according to the change of the size of the detection frame of the same target vehicle over time. When the target vehicle detection frame in the single camera gradually increases, it indicates that the target vehicle is moving towards the single camera, and when the target vehicle detection frame in the single camera gradually decreases, it indicates that the target vehicle is moving away from the single camera. After obtaining the relative motion type between the target vehicle and the single camera, it is also necessary to extract the target vehicle driving direction indication information to obtain the driving direction of the target vehicle; wherein, the target vehicle driving direction indication information includes lane markings, vehicle tracks, and vehicle indicator light flashing information; finally, it is also necessary to obtain a high-quality vehicle picture of the target vehicle, and extract the license plate and vehicle portrait information of the target vehicle; wherein, the vehicle portrait information includes vehicle model, vehicle brand and model, vehicle body color, objects or ornaments placed in the vehicle, whether there is a sunroof, whether there is a roof rack, and whether there is anyone in the co-pilot. After the target vehicle is tracked in a single camera, the identified target vehicle attribute information, the relative motion type between the target vehicle and the single camera, the target vehicle's driving direction and the vehicle image are reported to the server to prepare for subsequent cross-camera target vehicle tracking.

[0121] After the target vehicle is tracked in a single camera, the following steps are performed: Figure 1 Steps shown: Step S300, when the target vehicle disappears from the single camera, according to the relative motion type between the target vehicle and the single camera, the vehicle driving direction and time information, the adjacent nodes of the single camera that meet the constraint conditions are searched in the spatiotemporal topological relationship of the camera network as a candidate node set;

[0122] In practice, when the target vehicle disappears from the current single camera, it is necessary to quickly and timely find the next node where the target vehicle may appear. Therefore, based on the time information and the target vehicle motion information such as the relative motion type between the target vehicle and the current node and the vehicle driving direction information, the downstream adjacent nodes of the current node that meet certain constraints are found in the spatiotemporal topological relationship of the camera network as the candidate node set, so as to minimize the range of candidate nodes and improve the cross-camera tracking efficiency.

[0123] In order to obtain a candidate node set, when the target vehicle disappears from the single camera, according to the relative motion type between the target vehicle and the single camera, the vehicle driving direction and time information, searching for adjacent nodes of the single camera that meet the constraint conditions in the camera network spatiotemporal topological relationship as a candidate node set includes the following steps:

[0124] Step S301: when the target vehicle disappears from the single camera, in the spatiotemporal topological relationship of the camera network, a set of downstream adjacent nodes within a transfer time range between the single camera and its downstream adjacent nodes, whose time difference between the target association calculation time and the vehicle disappearance time, is taken as a first candidate node set, and a connection relationship set corresponding to the first candidate node set is taken as a first candidate connection relationship set; wherein the first candidate connection relationship set includes directed edges connecting the single camera and nodes in the first candidate node set and attribute information of the directed edges;

[0125] Step S302: Screening the first candidate node set and the first candidate connection relationship set according to the relative motion type between the target vehicle and the single camera to obtain a second candidate node set and a second candidate connection relationship set;

[0126] Step S303: According to the driving direction of the target vehicle before it disappears from the single camera, the second candidate node set and the second candidate connection relationship set are screened to obtain a candidate node set.

[0127] Specifically, when the target vehicle disappears from the single camera, in the spatiotemporal topological relationship of the camera network, the set of downstream adjacent nodes within the transfer time range between the single camera and its downstream adjacent nodes, which is the time difference between the target association calculation time and the vehicle disappearance time, is taken as the first candidate node set, and the connection relationship set corresponding to the first candidate node set is taken as the first candidate connection relationship set; wherein, the first candidate connection relationship set includes the directed edges connecting the single camera and the nodes in the first candidate node set and the attribute information of the directed edges. For example, when a vehicle disappears from a node, it will first appear in several nodes adjacent to it. Assuming that the target vehicle is to be continuously observed and tracked, the disappearance time of the target vehicle in the current node is t d , filter out the downstream nodes adjacent to the node from the camera network spatiotemporal topological relationship adjacency table to form a downstream adjacent node set, denoted as S v ={C i |i=1,2...,N}, the corresponding connection relationship set is denoted as S e , N is the number of nodes, the node and the downstream adjacent node set S v The transfer time range between the i-th nodes is [T i,min ,T i,max ]; if the target association calculation time is t c , t c is later than t d And earlier than the time when the target vehicle is found and tracked in the next node, the time difference between the target association calculation time and the vehicle disappearance time is calculated Δt=t c -t d , when Δt is in the downstream adjacent node set S v When the i-th node is within the transfer time range of the target vehicle, the node is determined to be the next node where the target vehicle may appear, thereby obtaining the first candidate node set S′ v ={C i |i=1,2...,N′,Δt∈[T i,min ,T i,max ]} and its corresponding first candidate connection relation set S′ e For example, assuming that the target vehicle disappears in the current node at 9:05:05, and the transfer time range between the current node and the nodes in the downstream adjacent node set includes 10 seconds to 20 seconds, 2 seconds to 5 seconds, etc., then when searching for the next node where the target vehicle may appear at 9:05:08, the time difference between the target association calculation time and the target vehicle disappearance time is 3 seconds. The downstream adjacent nodes with a transfer time range of 10 seconds to 20 seconds can be excluded, and the downstream adjacent nodes corresponding to the transfer time range with a time difference of 3 seconds (such as 2 seconds to 5 seconds) can be selected, thereby obtaining the first candidate node set S′ vand its corresponding first candidate connection relation set S′ e ;

[0128] In order to reduce the computational complexity, it is necessary to further narrow the range of the first candidate node set and the first candidate connection relationship set, and screen the first candidate node set and the first candidate connection relationship set according to the relative motion type between the target vehicle and the single camera to obtain the second candidate node set and the second candidate connection relationship set. For example, using the motion relationship of the target vehicle relative to the current node to S′ v and S′ e For further screening, when the target vehicle is gradually approaching the current node, e Filter flag F P The connection relations with a value of 0 constitute the second candidate connection relation set S″ e , the corresponding nodes constitute the second candidate node set S″ v ; When the target vehicle is gradually moving away from the current node, from S′ e Filter flag F P The connection relations with a value of 1 constitute the second candidate connection relation set S″ e , the corresponding nodes constitute the second candidate node set S″ v ; When the target vehicle approaches and then moves away from the current node, from S′ e Filter flag F P The connection relationship of 2 constitutes the second candidate connection relationship set S″ e , the corresponding nodes constitute the second candidate node set S″ v . In practice, compared to pedestrians, vehicles have a limited number of fixed driving patterns, so the range of the candidate node set can be further narrowed down. According to the driving direction of the target vehicle before it disappears in the single camera, the second candidate node set and the second candidate connection relationship set are screened to obtain a candidate node set. For example, assuming that the movement direction information of the target vehicle before it disappears in the current node is obtained using information such as traffic markings and vehicle trajectories, such as left turn, right turn, straight driving, U-turn, or driving towards a certain road branch, the second candidate connection relationship set S″ can be obtained based on the correspondence between the vehicle's driving direction and the connection relationship pair consisting of the node and its downstream adjacent nodes in the camera network spatiotemporal topological relationship graph model. e The flag bit F in a For the second candidate node set S″ v Further screening is performed to obtain the final set of candidate nodes, denoted as S″′ v For example, if the target vehicle is about to turn left at a traffic intersection, the second candidate node set S″ corresponding to the right turn, straight driving or U-turn can be obtained. v The nodes in are further filtered out.

[0129] After searching the adjacent nodes of the single camera that meet the constraint conditions in the spatiotemporal topology of the camera network as a candidate node set, the following can be performed: Figure 1 The steps shown are: Step S400, obtaining the target vehicle license plate, target vehicle portrait and deep learning features, associating and identifying the license plate, vehicle portrait and deep learning features of the vehicle set obtained by the candidate node set with the target vehicle, and obtaining the target vehicle cross-camera tracking result.

[0130] Considering that in practice, the field of view of adjacent nodes may not overlap or lack precise calibration information between the node and its adjacent nodes, it is difficult to reliably track the target vehicle by matching motion parameters through trajectory information and geometric transformation. Therefore, the embodiment of the present invention regards cross-camera tracking of the target vehicle as a target association recognition problem. After acquiring the target vehicle, the target vehicle license plate, target vehicle portrait and target vehicle picture can be obtained by tracking the target vehicle in a single camera; each node completes vehicle detection and tracking of its own captured video, obtains the position and size of the vehicle detection frame, completes license plate recognition and vehicle portrait information extraction according to the purpose and functional configuration of the node, and then transmits the extracted license plate, vehicle trajectory number, vehicle portrait, vehicle picture and node ID back to the server. Each node in the candidate node set can also directly upload its own captured video, picture or part of the processing results, and the remaining part of the above functions is completed by the server. After obtaining the candidate node set S″′ for the target vehicle transfer. v Later, the license plates, vehicle portraits and vehicle pictures of the vehicle set photographed by all nodes in the candidate node set can be obtained, and the above output result set of all candidate nodes is used as the candidate vehicle data set D for target association, that is, the vehicle set and its features obtained by the candidate node set mentioned above; the target vehicle information output by the single camera target vehicle tracking is associated and matched with the candidate vehicle data set D, and the deep learning features of the target vehicle and the candidate vehicle to be associated are extracted, and then the vehicle feature information is integrated, and step-by-step matching is performed according to a certain feature priority, and the final target vehicle association result is obtained by fusing multiple features such as semantic features and deep learning features, thereby obtaining the target vehicle cross-camera tracking result. The above target vehicle association processing process can be executed on the server.

[0131] In order to obtain the cross-camera vehicle tracking result, the target vehicle license plate, target vehicle portrait and deep learning features are obtained, and the license plate, vehicle portrait and deep learning features of the vehicle set obtained by the candidate node set are associated with the target vehicle for identification, and the cross-camera tracking result of the target vehicle is obtained, which includes the following steps:

[0132] Step S401, obtaining the target vehicle license plate and target vehicle portrait;

[0133] Step S402: obtaining the license plate and vehicle portrait information of the vehicle photographed by the candidate node set;

[0134] Step S403: When both the target vehicle and the vehicle photographed by the candidate node set contain license plate information, the license plate of the vehicle photographed by the candidate node set is associated with the license plate of the target vehicle. When the license plate of the vehicle photographed by the candidate node set is the same as the license plate of the target vehicle, the association is determined to be successful, the corresponding node ID is recorded, and the transfer route of the target vehicle between the nodes is updated;

[0135] Step S404: when the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, a first vehicle association score set of the target vehicle is obtained according to the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set;

[0136] Step S405: Obtain a target vehicle cross-camera tracking result according to the first vehicle association score set.

[0137] In this embodiment, in order to associate the target vehicle, the target vehicle license plate and the target vehicle portrait are first obtained; the license plate and vehicle portrait information of the vehicle photographed by the candidate node set are also obtained; when the target vehicle and the vehicle photographed by the candidate node set both contain license plate information, the license plate of the vehicle photographed by the candidate node set is associated with the target vehicle license plate, and when the license plate of the vehicle photographed by the candidate node set is the same as the license plate of the target vehicle, the association is judged to be successful; at this time, the node ID corresponding to the successfully associated vehicle is recorded, the transfer route of the target vehicle between the nodes is updated, and then the call execution of the cross-camera target vehicle tracking process is terminated. When the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, the first vehicle association score set of the target vehicle is obtained according to the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set.

[0138] In one implementation, when the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, obtaining the first vehicle association score set of the target vehicle according to the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set includes the following steps: when the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, obtaining a set of valid vehicle portrait information numbers according to the target vehicle and the vehicle portrait information of the vehicle photographed by the candidate node set; obtaining a set of associated vehicle portrait information numbers according to the degree of consistency between the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set; and obtaining the first vehicle association score set of the target vehicle by dividing the set of associated vehicle portrait information numbers by the set of valid vehicle portrait information numbers.

[0139] Specifically, when the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, the various features of the vehicle extracted can be used for subsequent association identification, such as vehicle portrait information, including vehicle type (such as sedan, SUV, medium / large truck, bus, etc.), vehicle brand, vehicle model, body color, objects / ornaments placed in the car, whether there is a sunroof, whether there is a roof rack, whether there is someone in the passenger seat, etc. For the target vehicle and each vehicle photographed by the candidate node set, the common and valid vehicle portrait category is called valid vehicle portrait information. By comparing and According to statistics, a set of valid vehicle portrait information numbers is obtained, and a sample in the vehicle set obtained by the candidate node set is called a candidate vehicle to be associated, and the corresponding number of valid vehicle portrait information is recorded as M (it is recommended that M ≥ 5); then, according to the consistency between the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set, a set of associated vehicle portrait information numbers is obtained, for example, the number of vehicle portrait information of the candidate vehicle to be associated and the target vehicle is recorded as M′; finally, the set of associated vehicle portrait information numbers is divided by the set of valid vehicle portrait information numbers to obtain a set of first vehicle association scores of the target vehicle, for example, the first vehicle association score of the candidate vehicle to be associated and the target vehicle is The first vehicle association score set of the target vehicle is denoted as S 1 .

[0140] After obtaining the first vehicle association score set, other vehicle information may be further used to calculate the target vehicle cross-camera tracking result through a machine learning algorithm according to the first vehicle association score set.

[0141] The step of obtaining the target vehicle cross-camera tracking result based on the first vehicle association score set includes the following steps: obtaining a target vehicle image; obtaining a vehicle image taken by the candidate node set; using the BoT-BS algorithm to perform vehicle association recognition, performing deep learning feature extraction and similarity calculation on the target vehicle image and the vehicle image taken by the candidate node set to obtain a second vehicle association score set for the target vehicle; and obtaining the target vehicle cross-camera tracking result based on the first vehicle association score set and the second vehicle association score set.

[0142] In this embodiment, after obtaining the first vehicle association score set, the target vehicle image and the vehicle image taken by the candidate node set are also obtained, and then the BoT-BS algorithm is used to perform association recognition of the target vehicle, and deep learning feature extraction and similarity calculation are performed on the target vehicle image and the vehicle image taken by the candidate node set, and the output similarity is used as the second vehicle association score set S of the target vehicle. 2 Finally, according to the first vehicle association score set S 1 and the second vehicle association score set S 2 , and obtain the target vehicle cross-camera tracking result.

[0143] Furthermore, in order to obtain the target vehicle cross-camera tracking result, the first vehicle association score set S 1 and the second vehicle association score set S 2 , obtaining the target vehicle cross-camera tracking result includes the following steps: obtaining a vehicle target association total score set according to the first vehicle association score set and the second vehicle association score set; when the total association score in the vehicle target association total score set is the largest and the number of consecutive associations to vehicles with the same trajectory number under the same adjacent node meets a preset value, it is determined that the candidate vehicles in the candidate node set are successfully associated with the target vehicle; when the candidate vehicles in the candidate node set are successfully associated with the target vehicle, the corresponding node IDs are recorded, and the transfer routes of the target vehicle between the nodes are updated.

[0144] Specifically, firstly according to the first vehicle association score set S 1 and the second vehicle association score set S 2 , the total vehicle target association score set is calculated as Then, the target vehicle is paired with each vehicle in the candidate vehicle dataset D (i.e., the vehicle set and its features obtained by the candidate node set) one by one. When the total association score of a candidate vehicle to be associated in the candidate vehicle dataset is the largest and has N consecutive F(For example, the value is 5) times when they are all vehicles with the same trajectory number in the same node, the candidate vehicle is considered to be the same vehicle as the target vehicle, and the candidate vehicle is determined to be successfully associated with the target vehicle, and its corresponding node ID is recorded, the transfer route of the target vehicle between the nodes is updated, and the call execution of the cross-camera target vehicle tracking process is terminated.

[0145] In addition, after the transfer route of the target vehicle between the nodes is obtained, the current node where the target vehicle is located obtained by cross-camera tracking is used as the new single camera, and the single-camera tracking and cross-camera tracking of the target vehicle are continued, and the motion trajectory of the target vehicle obtained by the single camera tracking is updated to the target vehicle driving trajectory record; according to the transfer route of the target vehicle between the nodes, the target vehicle driving trajectory record, and the installation location of the node, the driving period and driving route of the target vehicle are obtained with the assistance of a geographic information system.

[0146] Specifically, at each moment after the target vehicle disappears from a certain node, the target vehicle transfer candidate node selection method described in step S300 is used to screen out a set of candidate nodes to be associated, and the target vehicle cross-camera tracking method described in step S400 is used to perform target association identification until the next node where the target vehicle is located is confirmed; then, the current node where the target vehicle is located obtained by cross-camera tracking is used as the new single camera, and the single-camera tracking and subsequent cross-camera tracking of the target vehicle are continued, and the target trajectory tracked in the new single camera and the time information corresponding to the target vehicle at different locations are updated to the target vehicle driving record maintained by the server; according to the transfer route of the target vehicle between nodes and the node installation parameters (such as location), with the assistance of a geographic information system (such as Amap or Baidu Map), the driving period and driving route of the target vehicle are finally obtained.

[0147] Exemplary Devices

[0148] like Figure 2 As shown in , an embodiment of the present invention provides a vehicle tracking device based on multiple cameras, the device includes a camera network spatiotemporal topological relationship construction unit 501, a target vehicle motion trajectory acquisition unit 502, a candidate node set acquisition unit 503, and a target vehicle cross-camera tracking processing unit 504, wherein:

[0149] The camera network spatiotemporal topological relationship construction unit 501 is used to obtain road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and construct the camera network spatiotemporal topological relationship;

[0150] The target vehicle motion trajectory acquisition unit 502 is used to track the target vehicle in a single camera according to the target vehicle;

[0151] A candidate node set acquisition unit 503 is used to search for adjacent nodes of the single camera that meet the constraint conditions as a candidate node set in the camera network spatiotemporal topological relationship according to the relative motion type, vehicle driving direction and time information between the target vehicle and the single camera after the target vehicle disappears from the single camera;

[0152] The target vehicle cross-camera tracking processing unit 504 is used to associate and identify the license plate, vehicle portrait and deep learning features of the vehicle set obtained by the candidate node set with the target vehicle based on the obtained target vehicle license plate, target vehicle portrait and deep learning features, so as to obtain the target vehicle cross-camera tracking result.

[0153] Based on the above embodiment, the present invention further provides a server, whose principle block diagram is as follows: Figure 3 As shown. The server includes a processor, a memory, a network interface, a display screen, and a temperature sensor connected through a system bus. The processor of the server is used to provide computing and control capabilities. The memory of the server includes a non-volatile storage medium and an internal memory; the non-volatile storage medium stores an operating system and a computer program; the internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium; the computer program is executed by the processor to implement a vehicle tracking method based on multiple cameras. The network interface of the server is used to communicate with an external terminal through a network connection. The display screen of the server can be a liquid crystal display screen or an electronic ink display screen, and the temperature sensor of the server is pre-set inside the server to detect the operating temperature of the internal equipment.

[0154] Those skilled in the art will understand that Figure 3 The schematic diagram in the figure is only a block diagram of a partial structure related to the solution of the present invention, and does not constitute a limitation on the server to which the solution of the present invention is applied. The specific server may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0155] In one embodiment, a server is provided, comprising a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include instructions for performing the following operations:

[0156] Obtain road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and build the spatiotemporal topological relationship of the camera network;

[0157] Acquire a target vehicle, and track the target vehicle within a single camera;

[0158] When the target vehicle disappears from the single camera, according to the relative motion type between the target vehicle and the single camera, the vehicle driving direction and time information, the adjacent nodes of the single camera that meet the constraint conditions are searched in the spatiotemporal topological relationship of the camera network as a candidate node set;

[0159] The target vehicle's license plate, target vehicle's portrait and deep learning features are obtained, and the license plate, vehicle portrait and deep learning features of the vehicle set obtained by the candidate node set are associated with the target vehicle for identification to obtain the target vehicle's cross-camera tracking result.

[0160] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by computer program instructions and related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided by the present invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0161] The highlights of the present invention are:

[0162] 1. By comprehensively utilizing the camera topology and vehicle features, a solution for long-term and long-distance continuous tracking of vehicles is proposed, which can obtain a complete driving route of the target vehicle;

[0163] 2. For traffic application scenarios, we make full use of multi-source data such as road structure information, historical traffic flow data, and camera installation parameters. We not only clarify adjacent nodes and their connection relationships, but also add attribute information to the connection relationship based on the relative position change between the vehicle and the current node and the vehicle's driving direction, thereby constructing a more sophisticated spatiotemporal topological relationship of the camera network, which can effectively narrow the search range of vehicle transfer nodes and improve the efficiency of target association;

[0164] 3. When performing cross-camera target association, the target vehicle is associated with the license plate first according to the richness of the vehicle features. When the license plate cannot be obtained, the associated vehicle is determined by fusing the vehicle portrait information and the deep learning feature association score, which improves the association accuracy and can be applied to most practical occasions.

[0165] 4. All nodes and servers have synchronized time information. Each node reports real-time time information, detection results, tracking results and vehicle features synchronously for cross-camera target vehicle tracking and target vehicle global driving record generation, so that the final target vehicle driving route contains both time and location information, thereby conveniently determining the location of the target vehicle at each moment.

[0166] In summary, the present invention discloses a vehicle tracking method based on multiple cameras, a server, and a storage medium. The method includes: obtaining road structure and traffic attributes, camera installation parameters, historical traffic data, and geographic information data, and constructing a camera network spatiotemporal topological relationship; obtaining a target vehicle and tracking the target vehicle in a single camera; when the target vehicle disappears from the single camera, according to the relative motion type, vehicle driving direction, and time information between the target vehicle and the single camera, searching for adjacent nodes of the single camera that meet the constraint conditions in the camera network spatiotemporal topological relationship as a candidate node set; obtaining the target vehicle license plate, target vehicle portrait, and deep learning features, and associating and identifying the license plate, vehicle portrait, and deep learning features of the vehicle set obtained by the candidate node set with the target vehicle to obtain a target vehicle cross-camera tracking result. The method has a small amount of calculation and can obtain reliable and robust target vehicle continuous tracking results.

[0167] Based on the above embodiments, the present invention discloses a vehicle tracking method based on multiple cameras. It should be understood that the application of the present invention is not limited to the above examples. For ordinary technicians in this field, it can be improved or transformed according to the above description. All these improvements and transformations should fall within the scope of protection of the claims attached to the present invention.

Claims

1. A vehicle tracking method based on multiple cameras, characterized in that: The method comprises: Obtain road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and build the spatiotemporal topological relationship of the camera network; Acquire a target vehicle, and track the target vehicle within a single camera; When the target vehicle disappears from the single camera, according to the relative motion type between the target vehicle and the single camera, the vehicle driving direction and time information, the adjacent nodes of the single camera that meet the constraint conditions are searched in the spatiotemporal topological relationship of the camera network as a candidate node set; Obtain the target vehicle's license plate, target vehicle portrait, and deep learning features, associate and identify the target vehicle with the license plate, vehicle portrait, and deep learning features of the vehicle set obtained by the candidate node set, and obtain the target vehicle's cross-camera tracking result; The method of obtaining road structure and traffic attributes, camera installation parameters, historical traffic flow data and geographic information data, and constructing the spatiotemporal topological relationship of the camera network includes: obtaining road structure and traffic attributes, camera installation parameters, historical traffic flow data and geographic information data; taking the camera as a node; for each node, the nearest node in each direction around it or the node with the shortest reachable path in each traffic direction is called an adjacent node; the movement transfer mode of the vehicle between each node and the adjacent node corresponding to the node is taken as a directed edge; theoretical estimation and statistical analysis are performed based on the camera installation parameters, geographic information data, historical traffic flow data, and driving record data of buses and taxis to obtain the time transfer relationship between the vehicle and each node and the adjacent node corresponding to the node; according to the position change relationship of the vehicle relative to the node and the driving direction information of the vehicle before disappearing in the node, the connection relationship attribute information corresponding to each edge is obtained; wherein the connection relationship attribute information includes the relative movement type between the vehicle and the node and the driving direction information of the vehicle; the node, the directed edge, the time transfer relationship and the connection relationship attribute constitute the spatiotemporal topological relationship of the camera network; For each node, the node closest to it in each direction or the node with the shortest reachable path in each travel direction is called an adjacent node, including: when the nodes are densely distributed, the node closest to it in all travel directions in the monitoring area corresponding to the node is selected as the adjacent node; when the nodes are sparsely distributed, the node with the shortest reachable path in all travel directions corresponding to the node is selected as the adjacent node, and only one adjacent node is selected for each road or intersection.

2. The vehicle tracking method based on multiple cameras according to claim 1, characterized in that: The method of using the movement transfer mode of the vehicle between each node and the adjacent nodes corresponding to the node as a directed edge includes: When the nodes are densely distributed, determining a movement transfer mode of a vehicle between the node and the adjacent nodes corresponding to the node according to the installation positions of the node and the adjacent nodes corresponding to the node, a monitoring area, known road structure information, and traffic rule constraints; When the nodes are sparsely distributed, determining the movement transfer mode of the vehicle between the node and the adjacent nodes corresponding to the node according to the optimal path planning provided by the geographic information system; The movement transfer pattern of the vehicle between each node and the adjacent nodes corresponding to the node is taken as a directed edge.

3. The vehicle tracking method based on multiple cameras according to claim 1, characterized in that: The acquisition of road structure and traffic attributes, camera installation parameters, historical traffic data and geographic information data, and the construction of the spatiotemporal topological relationship of the camera network also include: A graph model is used to represent the spatiotemporal topological relationship of the camera network, and the expression is: In the formula, represents a node set, The camera network topology diagram is nodes, Represents the edge set between a node and its adjacent nodes, including a series of ordered node pairs. For slave nodes To Node The transition relationship is an edge. When there is a transition from the node To the node When the driving path ,otherwise, , and because the edge has directionality, and Not necessarily equal; for each pair of transfer relations , and also includes three attribute information: transfer time range, flag bit The vehicle and the node corresponding to the connection relationship are represented by The relative motion type between The vehicle driving direction corresponding to the connection relationship is represented by .

4. The vehicle tracking method based on multiple cameras according to claim 1, characterized in that: The step of acquiring a target vehicle and tracking the target vehicle in a single camera includes: Get the target vehicle; According to the target vehicle, the YoloV4 target detection algorithm is used to detect the target vehicle in the video, and the detection time, vehicle type, number and image coordinates of the detection frame position are output; According to the detection time, vehicle type, number, and image coordinates of the detection frame position, the Deep Sort target tracking algorithm is used to continuously track the movement of the target vehicle to obtain the target vehicle movement trajectory, wherein the target vehicle movement trajectory includes the target vehicle trajectory number, the detection time of each sampling point corresponding to the trajectory number, and the detection frame position coordinates; Acquire the relative motion type between the target vehicle and the single camera; wherein the relative motion type includes the target vehicle approaching, moving away, or first approaching and then moving away from the single camera; Extracting the target vehicle's driving direction indication information to obtain the target vehicle's driving direction; wherein the target vehicle's driving direction indication information includes lane markings, vehicle tracks, and vehicle indicator light flashing information; Obtain a high-quality vehicle image of the target vehicle, and extract the license plate and vehicle portrait information of the target vehicle; wherein the vehicle portrait information includes vehicle model, vehicle brand and model, body color, objects or ornaments placed in the vehicle, whether there is a sunroof, whether there is a roof rack, and whether there is someone in the passenger seat.

5. The vehicle tracking method based on multiple cameras according to claim 1, characterized in that: When the target vehicle disappears from the single camera, searching for adjacent nodes of the single camera that meet the constraint conditions as a candidate node set in the camera network spatiotemporal topology according to the relative motion type, vehicle driving direction and time information between the target vehicle and the single camera includes: When the target vehicle disappears from the single camera, in the spatiotemporal topological relationship of the camera network, a set of downstream adjacent nodes within the transfer time range between the single camera and its downstream adjacent nodes, whose time difference between the target association calculation time and the vehicle disappearance time, is taken as a first candidate node set, and a connection relationship set corresponding to the first candidate node set is taken as a first candidate connection relationship set; wherein the first candidate connection relationship set includes directed edges connecting the single camera and nodes in the first candidate node set and attribute information of the directed edges; According to the relative motion type between the target vehicle and the single camera, the first candidate node set and the first candidate connection relationship set are screened to obtain a second candidate node set and a second candidate connection relationship set; According to the driving direction of the target vehicle before disappearing from the single camera, the second candidate node set and the second candidate connection relationship set are screened to obtain a candidate node set.

6. The vehicle tracking method based on multiple cameras according to claim 1, characterized in that: The acquiring of the target vehicle license plate, the target vehicle portrait and the deep learning features, associating and identifying the license plate, the vehicle portrait and the deep learning features of the vehicle set acquired by the candidate node set with the target vehicle, and obtaining the target vehicle cross-camera tracking result includes: Obtain the target vehicle license plate and target vehicle portrait; Obtaining license plate and vehicle image information of the vehicle photographed by the candidate node set; When both the target vehicle and the vehicle photographed by the candidate node set contain license plate information, the license plate of the vehicle photographed by the candidate node set is associated with the license plate of the target vehicle. When the license plate of the vehicle photographed by the candidate node set is the same as the license plate of the target vehicle, the association is judged to be successful, the corresponding node ID is recorded, and the transfer route of the target vehicle between the nodes is updated; When the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, a first vehicle association score set of the target vehicle is obtained according to the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set; A target vehicle cross-camera tracking result is obtained according to the first vehicle association score set.

7. The vehicle tracking method based on multiple cameras according to claim 6, characterized in that: When the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, obtaining the first vehicle association score set of the target vehicle according to the target vehicle portrait and the vehicle portrait information of the vehicle photographed by the candidate node set includes: When the license plate of the target vehicle or the vehicle photographed by the candidate node set is not obtained, obtaining a set of valid vehicle portrait information numbers according to the vehicle portrait information of the target vehicle and the vehicle photographed by the candidate node set; According to the consistency between the target vehicle image and the vehicle image information of the candidate node set, a set of the number of associated vehicle image information is obtained; The first vehicle association score set of the target vehicle is obtained by dividing the associated vehicle portrait information number set by the valid vehicle portrait information number set.

8. The vehicle tracking method based on multiple cameras according to claim 6, characterized in that: The obtaining the target vehicle cross-camera tracking result according to the first vehicle association score set includes: Get the target vehicle image; Obtain vehicle pictures taken by the candidate node set; The BoT-BS algorithm is used to perform vehicle association recognition, and deep learning feature extraction and similarity calculation are performed on the target vehicle image and the vehicle images taken by the candidate node set to obtain a second vehicle association score set of the target vehicle; A target vehicle cross-camera tracking result is obtained according to the first vehicle association score set and the second vehicle association score set.

9. The vehicle tracking method based on multiple cameras according to claim 8, characterized in that: The obtaining the target vehicle cross-camera tracking result according to the first vehicle association score set and the second vehicle association score set includes: Obtaining a vehicle target association total score set according to the first vehicle association score set and the second vehicle association score set; When the total association score of the vehicle target association total score set is the largest and the number of consecutive associations to the vehicle with the same track number under the same adjacent node meets the preset value, it is determined that the candidate vehicle in the candidate node set is successfully associated with the target vehicle; When the candidate vehicles in the candidate node set are successfully associated with the target vehicle, the corresponding node IDs are recorded, and the transfer routes of the target vehicle between the nodes are updated.

10. The vehicle tracking method based on multiple cameras according to claim 6, characterized in that: The obtaining of the target vehicle license plate, the target vehicle portrait and the deep learning features, associating and identifying the license plate, the vehicle portrait and the deep learning features of the vehicle set obtained by the candidate node set with the target vehicle, and obtaining the target vehicle cross-camera tracking result also includes: The current node where the target vehicle is located obtained by cross-camera tracking is used as the new single camera, and the single-camera tracking and cross-camera tracking of the target vehicle are continued, and the motion trajectory of the target vehicle obtained by the single-camera tracking is updated into the target vehicle driving trajectory record; According to the transfer route of the target vehicle between the nodes, the driving track record of the target vehicle, and the installation location of the nodes, the driving time period and driving route of the target vehicle are obtained with the assistance of a geographic information system.

11. A server, characterized in that: The device comprises a memory and one or more programs, wherein the one or more programs are stored in the memory and are configured to be executed by one or more processors, and the one or more programs include being used to execute the method according to any one of claims 1 to 10.

12. A non-transitory computer-readable storage medium, characterized in that: When the instructions in the storage medium are executed by a processor of an electronic device, the electronic device is enabled to execute the method as claimed in any one of claims 1 to 10.

Images