Vehicle matching method and device, and electronic device

Abstract

The application discloses a vehicle matching method and device and electronic equipment, wherein the method comprises the following steps: determining a first similarity matrix between vehicles based on a first video image and a second video image, wherein the first video image is obtained by a first camera, and the second video image is obtained by a second camera; determining a direction weight matrix based on a positional relationship between the first camera and the second camera and a vehicle trajectory in the first video image; determining a second similarity matrix based on the direction weight matrix and the first similarity matrix; and matching vehicles in the first video image and the second video image based on the second similarity matrix.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, and in particular to a vehicle matching method and device and an electronic device. BACKGROUND

[0002] Vehicle monitoring technology is an important technology in the related technology of smart city, and accurate tracking of vehicles is the cornerstone of subsequent technologies such as vehicle anomaly detection. The key technology involved in vehicle monitoring technology is multi-target tracking technology. The main task of multi-target tracking is to locate multiple target vehicles in a given video image and record their trajectories.

[0003] Multi-target tracking using multiple cameras can improve tracking efficiency. However, a major problem in multi-target tracking in a multi-camera scenario is how to match the same vehicle in multiple different cameras. SUMMARY

[0004] To solve the above technical problems, the embodiments of the present application provide a vehicle matching method and device, an electronic device, a chip and a computer readable storage medium.

[0005] The vehicle matching method provided by the embodiments of the present application comprises:

[0006] determining a first similarity matrix between vehicles based on a first video image and a second video image, the first video image being obtained by a first camera, and the second video image being obtained by a second camera;

[0007] determining a direction weight matrix based on a positional relationship between the first camera and the second camera and vehicle trajectories in the first video image;

[0008] determining a second similarity matrix based on the direction weight matrix and the first similarity matrix;

[0009] matching vehicles in the first video image and the second video image based on the second similarity matrix.

[0010] In some optional embodiments, the determining of the direction weight matrix based on the positional relationship between the first camera and the second camera and the vehicle trajectories in the first video image comprises:

[0011] determining a weight of each vehicle in the first video image based on the positional relationship between the first camera and the second camera and a motion trajectory of each vehicle in the first video image;

[0012] determining the direction weight matrix based on the weight of each vehicle in the first video image.

[0013] In some optional embodiments, the determining the weight of each vehicle in the first video image based on the positional relationship between the first camera and the second camera and the motion trajectory of each vehicle in the first video image comprises:

[0014] For each vehicle in the first video image, determining N prediction scores of the vehicle based on the motion trajectory of the vehicle, N being a positive integer;

[0015] normalizing the N prediction scores to obtain N probability values, each of the N probability values representing a probability of the vehicle in a driving direction;

[0016] selecting one probability value from the N probability values based on the positional relationship between the first camera and the second camera, and determining the weight of the vehicle based on the selected probability value.

[0017] In some optional embodiments, the determining the N prediction scores of the vehicle based on the motion trajectory of the vehicle comprises:

[0018] dividing a road surface in the first video image into M1 entry regions and M2 exit regions, wherein each of the M1 entry regions comprises N entry sub-regions, M1 and M2 being positive integers;

[0019] determining a target entry region corresponding to the vehicle from the M1 entry regions;

[0020] determining a bounding box of the vehicle based on the motion trajectory of the vehicle, and determining an overlap rate of the bounding box of the vehicle with N entry sub-regions included in the target entry region, and an overlap rate of the bounding box of the vehicle with the M2 exit regions;

[0021] determining the N prediction scores of the vehicle based on the determined overlap rates, each of the N prediction scores corresponding to a prediction score of the vehicle in a driving direction.

[0022] In some optional embodiments, the dividing the road surface in the first video image into M1 entry regions and M2 exit regions comprises:

[0023] recognizing road surface markers in the first video image, the road surface markers comprising lane markers and / or lane lines;

[0024] dividing the road surface in the first video image into M1 entry regions and M2 exit regions based on the road surface markers in the first video image.

[0025] In some optional embodiments, the determining the first similarity matrix between the vehicles based on the first video image and the second video image comprises:

[0026] identifying features of each vehicle in the first video image, and identifying features of each vehicle in the second video image;

[0027] calculating the first similarity matrix based on the features of each vehicle in the first video image and the features of each vehicle in the second video image, wherein each element in the first similarity matrix represents a feature distance between one vehicle in the first image and one vehicle in the second image.

[0028] In some optional embodiments, the matching the vehicles in the first video image and the second video image based on the second similarity matrix comprises:

[0029] calculating a sum of feature distances of multiple pairs of vehicles under a vehicle matching relationship based on the second similarity matrix;

[0030] adjusting the vehicle matching relationship with a minimum sum of feature distances as an optimization target;

[0031] taking the vehicle matching relationship corresponding to the minimum sum of feature distances as a vehicle matching result.

[0032] The vehicle matching apparatus provided by the embodiments of the present application comprises:

[0033] a first determining unit configured to determine a first similarity matrix between vehicles based on a first video image and a second video image, the first video image being captured by a first camera, and the second video image being captured by a second camera;

[0034] a second determining unit configured to determine a direction weight matrix based on a positional relationship between the first camera and the second camera and vehicle trajectories in the first video image;

[0035] a third determining unit configured to determine a second similarity matrix based on the direction weight matrix and the first similarity matrix;

[0036] a matching unit configured to match the vehicles in the first video image and the second video image based on the second similarity matrix.

[0037] In some optional embodiments, the second determining unit is configured to determine a weight of each vehicle in the first video image based on the positional relationship between the first camera and the second camera and a motion trajectory of each vehicle in the first video image, and determine the direction weight matrix based on the weight of each vehicle in the first video image.

[0038] In some optional embodiments, the second determining unit is configured to determine, for each vehicle in the first video image, N prediction scores of the vehicle based on a motion trajectory of the vehicle, N being a positive integer; normalize the N prediction scores to obtain N probability values, each of the N probability values representing a probability of the vehicle in one driving direction; select one probability value from the N probability values based on a positional relationship between the first camera and the second camera, and determine a weight of the vehicle based on the selected probability value.

[0039] In some optional embodiments, the apparatus further comprises:

[0040] a road surface dividing unit configured to divide a road surface in the first video image into M1 entry regions and M2 exit regions, wherein each of the M1 entry regions comprises N entry sub-regions, M1 and M2 being positive integers;

[0041] the second determining unit is configured to determine a target entry region corresponding to the vehicle from the M1 entry regions, determine a detection box of the vehicle based on a motion trajectory of the vehicle, determine an overlap rate of the detection box of the vehicle and N entry sub-regions included in the target entry region, and determine overlap rates of the detection box of the vehicle and the M2 exit regions, and determine N prediction scores of the vehicle based on the determined overlap rates, each of the N prediction scores corresponding to a prediction score of the vehicle in one driving direction.

[0042] In some optional embodiments, the apparatus further comprises:

[0043] a road surface mark identifying unit configured to identify road surface marks in the first video image, the road surface marks comprising lane marks and / or lane lines;

[0044] the road surface dividing unit is configured to divide the road surface in the first video image into M1 entry regions and M2 exit regions based on the road surface marks in the first video image.

[0045] In some optional embodiments, the first determining unit is configured to identify features of each vehicle in the first video image and identify features of each vehicle in the second video image, and calculate a first similarity matrix based on the features of each vehicle in the first video image and the features of each vehicle in the second video image, wherein each element in the first similarity matrix represents a feature distance between one vehicle in the first image and one vehicle in the second image.

[0046] In some optional implementations, the matching unit is configured to calculate a sum of feature distances of multiple pairs of vehicles in a vehicle matching relationship based on the second similarity matrix; adjust the vehicle matching relationship with a minimum sum of feature distances as an optimization target; and determine a vehicle matching relationship corresponding to the minimum sum of feature distances as a vehicle matching result.

[0047] The electronic device provided by the embodiments of the present application includes a processor and a memory. The memory is configured to store a computer program. The processor is configured to invoke and run the computer program stored in the memory to execute any of the vehicle matching methods described above.

[0048] The chip provided by the embodiments of the present application includes a processor. The processor is configured to invoke and run a computer program from a memory, so that a device installed with the chip executes any of the vehicle matching methods described above.

[0049] The computer readable storage medium provided by the embodiments of the present application is configured to store a computer program. The computer program causes a computer to execute any of the vehicle matching methods described above.

[0050] In the technical solution of the embodiments of the present application, a direction weight matrix is determined based on the positional relationship between cameras and the vehicle trajectory. The direction weight matrix can reflect the driving direction of the vehicle. The original similarity matrix (i.e., the first similarity matrix) is weighted by using the direction weight matrix, and a new similarity matrix (i.e., the second similarity matrix) is obtained. The new similarity matrix can better reflect the similarity between vehicles. By matching vehicles by using the new similarity matrix, the matching accuracy of the vehicles can be improved. In addition, since the new similarity matrix introduces the positional relationship between cameras as a factor, the new similarity matrix better adapts to actual application scenarios, and the matching accuracy of the vehicles is improved. BRIEF DESCRIPTION OF DRAWINGS

[0051] Figure 1 FIG. 1 is an architecture diagram of a vehicle tracking system provided by the embodiments of the present application;

[0052] Figure 2 FIG. 2 is a flowchart of a vehicle matching method provided by the embodiments of the present application; Figure One

[0053] Figure 3 FIG. 3 is a schematic diagram of a road surface division region provided by the embodiments of the present application; Figure One

[0054] Figure 4 FIG. 4 is a schematic diagram of a road surface division region provided by the embodiments of the present application; Figure Two

[0055] Figure 5-1 FIG. 5 is a flowchart of a vehicle matching method provided by the embodiments of the present application;​​​Figure Two ;

[0056] Figure 5-2 is a schematic diagram of a vehicle in a video image provided by an embodiment of the present application;

[0057] Figure 6 is a schematic diagram of the structural composition of a vehicle matching device provided by an embodiment of the present application;

[0058] Figure 7 is a schematic structural diagram of an electronic device provided by an embodiment of the present application;

[0059] Figure 8 is a schematic structural diagram of a chip according to an embodiment of the present application. DETAILED DESCRIPTION

[0060] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all of the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0061] It should be noted that in the embodiments of the present application, the term "and / or" is only used to describe the association relationship of the associated objects, and can represent the existence of three relationships, for example, A and / or B can represent the existence of A alone, A and B together, and B alone. In addition, in the embodiments of the present application, the character " / " generally represents an "or" relationship between the front and rear associated objects.

[0062] It should be understood that "indication" mentioned in the embodiments of the present application can be direct indication, or indirect indication, or can represent an associated relationship. For example, A indicates B, which can mean that B can be obtained through A; or A indirectly indicates B, for example, A indicates C, and B can be obtained through C; or A and B have an associated relationship.

[0063] In the description of the embodiments of the present application, the term "corresponding" can represent a direct or indirect corresponding relationship between the two, or can represent an associated relationship between the two, or can represent an indication and being indicated, configuration and being configured, etc.

[0064] In order to facilitate the understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any way, and all of them belong to the protection scope of the embodiments of the present application.

[0065] Vehicle monitoring technology is an important technology in the related technologies of smart city, and accurate tracking of vehicles is the cornerstone of related subsequent technologies such as vehicle anomaly detection. The key technology involved in vehicle monitoring technology is multi-target tracking technology. The main task of multi-target tracking is to locate multiple target vehicles of interest in a given video image at the same time, and maintain their IDs and record their trajectories. Multi-target tracking includes the following two aspects: multi-target tracking inside a single camera and multi-target tracking of multiple cameras. Whether it is multi-target tracking inside a single camera or multi-target tracking of multiple cameras, the key technology involves associating (i.e., matching) targets in different video images.

[0066] For multi-target tracking inside a single camera, the association between targets is completed based on the detection data of the current frame and the historical detection data of the previous frame. Specifically, the spatial distance and semantic feature distance between targets are calculated, and the Kalman or Hungarian matching algorithm is used to complete the association between targets.

[0067] For multi-target tracking using multiple cameras, it is mainly achieved by a camera calibration-based method and a feature matching-based method. For the camera calibration-based method, the projection matrix of the camera is used to project the video images of different cameras onto the same screen. For the feature matching-based method, efficient apparent features and spatio-temporal information are used to improve the matching result.

[0068] Considering the actual application situation, multiple cameras are often needed to be erected on the road, and multi-target tracking using multiple cameras is a very important research topic in the field of monitoring video, and has a very large research space.

[0069] On the one hand, a big problem of multi-target tracking in the multi-camera scene is how to match the same vehicle in multiple different cameras. In a complex background environment, such as occlusion, light change, target posture change, etc., vehicle license plate recognition becomes difficult, and also faces the problems of dynamic change of target number and mutual occlusion between multiple targets. The existence of these difficulties leads to low matching accuracy. In view of this situation, the technical scheme of the embodiment of the present application enhances the effective information and improves the matching accuracy by predicting the driving direction of the vehicle and then predicting the probability of the vehicle appearing in other cameras according to the driving direction.

[0070] On the other hand, most matching models are trained on public data sets and then migrated to the scene for adaptation. However, in actual applications, the matching model often cannot well adapt to the real scene. In order to make the most of the resource information in the actual scene, the technical scheme of the embodiment of the present application introduces the positional relationship between cameras into the matching model to improve the adaptation degree of the matching model to the actual scene.

[0071] The following will first describe a vehicle tracking system according to an embodiment of the present application. It should be noted that the following technical solution of the embodiment of the present application can be implemented in any combination with the content in the vehicle tracking system, and all of them belong to the protection scope of the embodiment of the present application.

[0072] The embodiment of the present application proposes a vehicle tracking system, as shown in the figure, comprising an image acquisition module 11, a target detection module 12, a single camera tracking module 13, and a cross-camera tracking module 14. Among them, Figure 1

[0073] The image acquisition module 11 is configured to acquire a video image of a vehicle.

[0074] The target detection module 12 is configured to identify a vehicle in the video image.

[0075] In some optional embodiments, the vehicle in the video image can be identified by using at least one of the following algorithms: Faster-RCNN algorithm, SSD algorithm, YOLO algorithm.

[0076] The single camera tracking module 13 is configured to determine a driving track (referred to as vehicle track) of the vehicle based on the vehicle identification result of the multiple frames of video images.

[0077] In some optional embodiments, the same vehicle in the multiple frames of video images can be matched by using at least one of the following algorithms: DeepSort algorithm, Hungarian algorithm, Kalman algorithm. The driving track of the vehicle is determined according to the position of the same vehicle in the multiple frames of video images.

[0078] The cross-camera tracking module 14 is configured to determine a direction weight matrix based on the relative position relationship between the cameras and the vehicle track; perform weighted processing on the original first similarity matrix based on the direction weight matrix to obtain a new second similarity matrix; and match the same vehicle in the video images of the two cameras based on the second similarity matrix. Further, the driving track of the vehicle is determined according to the position of the same vehicle in the video images of the two cameras.

[0079] The technical solution of the embodiment of the present application, for a multi-camera scene, uses a feature matching algorithm to determine whether the vehicles in the video images of the two cameras are the same vehicle. The feature similarity matrix (i.e. the first feature similarity matrix) calculated based on the vehicle features is easily affected by related interference such as light and weather. In order to optimize the first feature similarity matrix, the technical solution of the embodiment of the present application introduces the position relationship between the cameras and the vehicle track as two factors to obtain a more accurate second feature similarity matrix. Specifically, the direction weight matrix is determined based on the relative position relationship between the cameras and the vehicle track, and the second similarity matrix is obtained by performing weighted processing on the original first similarity matrix based on the direction weight matrix. ​

[0080] The introduction of the direction weight matrix in the technical solution of the embodiment of the application is based on the following consideration: the driving track of a vehicle in the video image of the first camera and the positional relationship between the first camera and the second camera are related to the probability of the vehicle appearing in the video image of the second camera. As an example, the first camera and the second camera are located on the same street in front and back, that is, the angle of the first camera and the second camera is 0 degrees, in other words, the video image of the first camera and the video image of the second camera are parallel, then the vehicle that turns left or right in the video image of the first camera should not appear in the video image of the second camera. Therefore, a direction weight matrix is quantified according to the positional relationship between the vehicle track and the camera, and a more accurate feature similarity matrix can be obtained by weighting the existing feature similarity matrix through the direction weight matrix, thereby improving the cross-camera vehicle matching accuracy.

[0081] Figure 2 is a flowchart of a vehicle matching method provided by the embodiment of the application Figure One As shown in Figure 2 the vehicle matching method comprises the following steps:

[0082] Step 201: determining a first similarity matrix between vehicles based on a first video image and a second video image, the first video image being obtained by a first camera, and the second video image being obtained by a second camera.

[0083] In the embodiment of the application, the first camera has an image acquisition module, and the first camera acquires the first video image in real time by using the image acquisition module. The second camera also has an image acquisition module, and the second camera acquires the second video image in real time by using the image acquisition module.

[0084] In the embodiment of the application, the determination of the first similarity matrix between vehicles based on the first video image and the second video image can be realized by the following way:

[0085] 1) identifying the features of each vehicle in the first video image, and identifying the features of each vehicle in the second video image;

[0086] 2) calculating the first similarity matrix based on the features of each vehicle in the first video image and the features of each vehicle in the second video image, wherein each element in the first similarity matrix represents the feature distance between one vehicle in the first image and one vehicle in the second image.

[0087] Herein, features of the plurality of vehicles in the first video image are learned according to the first video image of the first camera; features of the plurality of vehicles in the second video image are learned according to the second video image of the first camera. Then, a first similarity matrix is calculated based on the features of each vehicle in the first video image and the features of each vehicle in the second video image.

[0088] In some optional embodiments, the features of the vehicles can be learned through the appearance of the vehicles, license plates, and relationship information such as time, space, and speed.

[0089] For example, the first video image includes 3 vehicles, which are denoted as vehicle 1, vehicle 2, and vehicle 3 for ease of description; the second video image includes 3 vehicles, which are denoted as vehicle 4, vehicle 5, and vehicle 6 for ease of description. The features of the 3 vehicles in the first video image are denoted as f1, f2, and f3 respectively, and the features of the 3 vehicles in the second video image are denoted as f4, f5, and f6 respectively. The first similarity matrix is:

[0090]

[0091] wherein fij represents the feature distance between vehicle i in the first video image and vehicle j in the second video image, fij = fi-fj, i takes values of 1, 2, and 3, and j takes values of 4, 5, and 6.

[0092] Step 202: determining a direction weight matrix based on the positional relationship between the first camera and the second camera and the vehicle trajectories in the first video image.

[0093] Specifically, the weight of each vehicle in the first video image is determined based on the positional relationship between the first camera and the second camera and the motion trajectory of each vehicle in the first video image; and the direction weight matrix is determined based on the weight of each vehicle in the first video image.

[0094] In the embodiments of this application, the weight of each vehicle in the first video image can be determined in the following ways:

[0095] 1) For each vehicle in the first video image, N prediction scores of the vehicle are determined based on the motion trajectory of the vehicle, where N is a positive integer.

[0096] Specifically, the road surface in the first video image is divided into M1 entering regions and M2 exiting regions, wherein each of the M1 entering regions comprises N entering sub-regions, and M1 and M2 are positive integers; a target entering region corresponding to the vehicle is determined from the M1 entering regions; a detection box of the vehicle is determined based on a motion trajectory of the vehicle, and an overlap rate of the detection box of the vehicle and the N entering sub-regions included in the target entering region and an overlap rate of the detection box of the vehicle and the M2 exiting regions are determined; and N prediction scores of the vehicle are determined based on the determined overlap rates, each of the N prediction scores corresponding to a prediction score of the vehicle in one driving direction.

[0097] In some optional embodiments, a road surface identifier in the first video image is identified, and the road surface identifier comprises a lane identifier and / or a lane line; and the road surface in the first video image is divided into M1 entering regions and M2 exiting regions based on the road surface identifier in the first video image.

[0098] 2) The N prediction scores are normalized to obtain N probability values, each of the N probability values being used to represent a probability of the vehicle in one driving direction.

[0099] 3) One probability value is selected from the N probability values based on a positional relationship between the first camera and the second camera, and a weight of the vehicle is determined based on the selected probability value.

[0100] The following describes how to determine a direction weight matrix in combination with a specific application example.

[0101] I) A road surface identifier in the first video image is identified.

[0102] A) The original first video image is preprocessed.

[0103] In some optional embodiments, the original first video image can be converted into a gray video image, thereby providing a premise for subsequent improvement of identification efficiency of the road surface identifier.

[0104] B) A lane identifier on the road surface is identified.

[0105] Specifically, for each frame of video image, the lane identifier on the road surface is identified, and the lane identifier is classified. Here, the categories of the lane identifier include but are not limited to the following: a left-turn identifier, a right-turn identifier, a straight-going identifier, and a U-turn identifier.

[0106] In some optional embodiments, the Fast-RCNN algorithm can be used to identify the lane identifier on the road surface.

[0107] C) identify lane lines on the road surface.

[0108] Specifically, for each frame of video image, lane lines on the road surface are identified, and lane regions are divided according to the lane lines.

[0109] In some optional embodiments, gradient edge detection method and color threshold detection method can be used to identify lane lines on the road surface.

[0110] II) divide the road surface into multiple regions based on the road surface identification.

[0111] A) divide the road surface into M1 entry regions and M2 exit regions according to the lane identification and / or lane lines.

[0112] Here, the exit region refers to the region where the starting position of the vehicle is located, and the entry region refers to the region where the reaching position of the vehicle is located.

[0113] As an example, refer to Figure 3 For example, at a crossroads, it can be divided into east (E) west (W) south (S) north (N) four directions, and left (L) right (R) two directions, according to the vehicle driving specification, R is the entry region, L is the exit region, it can be seen that there are 4 entry regions and 4 exit regions, among which, the 4 entry regions are respectively: SR region, ER region, NR region, WR region; 4 exit regions are respectively: SL region, EL region, NL region, WL region.

[0114] Further, for each entry region, the lane identification and / or lane lines can divide the entry region into multiple entry sub-regions. As an example, refer to Figure 3 For the SR region, the lane identification and / or lane lines can divide the entry region into 3 entry sub-regions, which are respectively: left turn region, straight region, right turn region. Each entry sub-region corresponds to a driving direction of the vehicle.

[0115] As an example, refer to Figure 4 For example, at a T-shaped intersection, it can be divided into east (E) west (W) south (S) three directions, and left (L) right (R) two directions, according to the vehicle driving specification, R is the entry region, L is the exit region, it can be seen that there are 3 entry regions and 3 exit regions, among which, the 3 entry regions are respectively: SR region, ER region, WR region; 3 exit regions are respectively: SL region, EL region, WL region.

[0116] Further, for each entry region, the lane identification and / or lane lines can divide the entry region into multiple entry sub-regions. As an example, refer to Figure 4For the SR region, the lane mark and / or the lane line can divide the entering region into 2 entering sub-regions, which are respectively a left-turn region and a right-turn region. Each entering sub-region corresponds to a driving direction of the vehicle.

[0117] III) identifying a vehicle trajectory in the first video image and determining a weight of the vehicle.

[0118] A) determining a probability of the vehicle in a plurality of driving directions.

[0119] Here, the vehicle trajectory in the first video image can be obtained by using the single camera tracking module, and the bounding box of the vehicle can be determined according to the vehicle trajectory, where the bounding box of the vehicle is the smallest bounding box covering the vehicle trajectory.

[0120] For a frame of video image, the overlap rate calculated based on the frame of video image is:

[0121] p i =(the overlapping area of the bounding box and the region i) / the area of the bounding box.

[0122] wherein p i represents the overlap rate of the bounding box of the vehicle in the frame of video image and the region i.

[0123] For a plurality of frames of video image within a time length t, the average overlap rate is obtained by adding the overlap rate calculated for each frame of video image and dividing by the time length:

[0124] P i =∑p i / t.

[0125] wherein P i represents the average overlap rate of the bounding box of the same vehicle in the plurality of frames of video image and the region i.

[0126] Referring to Figure 3 , taking the vehicle entering from the SR region as an example, the SR region is the target entering region, according to the above formula, the overlap rates of the bounding box of the vehicle and the 3 entering sub-regions (i.e. the left-turn region, the straight-going region, and the right-turn region) included in the SR region are respectively P1, P2, and P3, in addition, according to the above formula, the overlap rates of the bounding box of the vehicle and the 4 exiting regions (i.e. the WL region, the NL region, and the EL region) are respectively P W , P N , and P E . Based on the determined overlap rates, the 3 prediction scores of the vehicle are respectively:

[0127] S1=(a*P1+P2+P3)*P W ;

[0128] S2 = (P1 + a*P2 + P3)*P N ;

[0129] S3 = (P1 + P2 + a*P3)*P E ;

[0130] wherein S1 represents a left-turn prediction score, S2 represents a straight prediction score, and S3 represents a right-turn prediction score. a is a weight factor, which is used to weight the direction of the next possible driving of the vehicle according to the direction of the lane where the vehicle is currently located.

[0131] The three prediction scores are normalized using the following softmax function:

[0132] (D1, D2, D3) = softmax(S1, S2, S3);

[0133] wherein D i represents the probability of the vehicle driving in the direction corresponding to the region i, i is 1, 2, 3. Specifically, D1 is used to represent the probability of the vehicle driving in the left-turn direction corresponding to the left-turn region, D2 is used to represent the probability of the vehicle driving in the straight direction corresponding to the straight region, and D3 is used to represent the probability of the vehicle driving in the right-turn direction corresponding to the right-turn region.

[0134] B) determining the weight of the vehicle according to the positional relationship between the cameras.

[0135] The weight of the vehicle is determined by the following formula:

[0136] λ = -log(D i );

[0137] wherein i is 1, 2, 3, and the value of i depends on the positional relationship between the cameras. As an example, if the included angle between the cameras is 90 degrees, the value of i is 1; if the included angle between the cameras is 0 degrees, the value of i is 2; and if the included angle between the cameras is -90 degrees, the value of i is 3.

[0138] Taking the included angle between the first camera and the second camera as 0 degrees as an example, according to the first video image of the first camera, it is determined that the A vehicle drives from the left-turn region to the left-turn direction or from the right-turn region to the right-turn direction. Since the included angle between the first camera and the second camera is 0 degrees, the A vehicle will not appear in the second video image of the B camera. The feature distance of this A vehicle relative to each vehicle in the second video image is processed by weighting with the weight λ, λ = -log(D2).

[0139] C) determining the direction weight matrix according to the weight of each vehicle.

[0140] The first video image of the first camera includes a plurality of vehicles, and the weight of each vehicle is determined by the above scheme. A direction weight matrix is formed based on the weight of each vehicle, and the direction weight matrix is as follows:

[0141] W(λ1, λ2, … λ L );

[0142] Wherein, the first video image includes L vehicles, λ j is the weight corresponding to the vehicle i, and i ranges from 1 to L.

[0143] Step 203: determining a second similarity matrix based on the direction weight matrix and the first similarity matrix.

[0144] In the embodiments of the present application, the original similarity matrix (i.e. the first similarity matrix) is weighted based on the direction weight matrix to obtain a new similarity matrix (i.e. the second similarity matrix).

[0145] Specifically, the first similarity matrix is weighted to obtain the second similarity matrix by the following formula:

[0146] F’ = W*F;

[0147] Wherein, F’ represents the second similarity matrix, W represents the direction weight matrix, and F represents the first similarity matrix.

[0148] Step 204: matching the vehicles in the first video image and the second video image based on the second similarity matrix.

[0149] Specifically, the sum of the feature distances of a plurality of pairs of vehicles under a vehicle matching relationship is calculated based on the second similarity matrix; the vehicle matching relationship is adjusted with the minimum sum of the feature distances as the optimization target; and the vehicle matching relationship corresponding to the minimum sum of the feature distances is taken as the vehicle matching result.

[0150] In some optional embodiments, a greedy algorithm or a dynamic programming algorithm can be used to match the vehicles in the two video images with the minimum sum of the feature distances between multiple targets as the optimization target.

[0151] The technical scheme of the embodiment of the application determines a direction weight matrix based on the positional relationship between cameras and vehicle trajectories, the direction weight matrix can reflect the driving direction of a vehicle, and the original similarity matrix (i.e., a first similarity matrix) is weighted using the direction weight matrix to obtain a new similarity matrix (i.e., a second similarity matrix), which can better reflect the similarity between vehicles, and the matching accuracy of vehicles can be improved by matching the vehicles using the new similarity matrix. In addition, the new similarity matrix introduces the positional relationship between cameras as a factor, so that the new similarity matrix better adapts to actual application scenarios and improves the matching accuracy of vehicles.

[0152] Figure 5-1 is a flowchart of a vehicle matching method provided by the embodiment of the application Figure Two As shown in Figure 5-1 , the vehicle matching method comprises the following steps:

[0153] Step 501: capturing a first video image by a first camera and capturing a second video image by a second camera.

[0154] As an example, referring to Figure 5-2 , the first video image includes three vehicles, which are recorded as vehicle 1, vehicle 2 and vehicle 3 for ease of description; and the second video image includes three vehicles, which are recorded as vehicle 4, vehicle 5 and vehicle 6 for ease of description.

[0155] Step 502: calculating a first similarity matrix of vehicles according to the first video image and the second video image.

[0156] As an example, the first video image includes three vehicles, which are recorded as vehicle 1, vehicle 2 and vehicle 3 for ease of description; and the second video image includes three vehicles, which are recorded as vehicle 4, vehicle 5 and vehicle 6 for ease of description. The features of the three vehicles in the first video image are f1, f2 and f3 respectively, and the features of the three vehicles in the second video image are f4, f5 and f6 respectively. The first similarity matrix is:

[0157]

[0158] wherein fij represents the feature distance between vehicle i in the first video image and vehicle j in the second video image, fij = fi-fj, the value of i is 1, 2, 3, and the value of j is 4, 5, 6.

[0159] Step 503: determining a direction weight matrix based on the positional relationship between the first camera and the second camera and the vehicle trajectories of the vehicles in the first video image.

[0160] Here, the manner of determining the direction weight matrix can refer to the description of the foregoing related solutions.

[0161] Step 504: performing weighted processing on the first similarity matrix by using the direction weight matrix, to obtain a second similarity matrix.

[0162] Specifically, the first similarity matrix is processed by using the following formula to obtain the second similarity matrix:

[0163] F' = W * F;

[0164] Wherein, F' represents the second similarity matrix, W represents the direction weight matrix, and F represents the first similarity matrix.

[0165] Step 505: matching the vehicles in the first video image and the second video image according to the second similarity matrix.

[0166] In some optional embodiments, a greedy algorithm or a dynamic programming algorithm can be used to match the vehicles in the two video images, with the sum of feature distances between multiple targets being minimized as an optimization target.

[0167] Figure 6 is a structural composition schematic diagram of a vehicle matching device provided by an embodiment of the present application, as shown in Figure 6 The vehicle matching device comprises:

[0168] A first determination unit 601 is configured to determine a first similarity matrix between vehicles based on a first video image and a second video image, the first video image being obtained by a first camera, and the second video image being obtained by a second camera.

[0169] A second determination unit 602 is configured to determine a direction weight matrix based on a positional relationship between the first camera and the second camera and a vehicle trajectory in the first video image.

[0170] A third determination unit 603 is configured to determine a second similarity matrix based on the direction weight matrix and the first similarity matrix.

[0171] A matching unit 604 is configured to match the vehicles in the first video image and the second video image based on the second similarity matrix.

[0172] In some optional embodiments, the second determination unit 602 is configured to determine a weight of each vehicle in the first video image based on the positional relationship between the first camera and the second camera and a motion trajectory of each vehicle in the first video image, and determine the direction weight matrix based on the weight of each vehicle in the first video image.

[0173] In some optional embodiments, the second determining unit 602 is configured to determine, for each vehicle in the first video image, N prediction scores of the vehicle based on a motion trajectory of the vehicle, N being a positive integer; normalize the N prediction scores to obtain N probability values, each of the N probability values representing a probability of the vehicle in a driving direction; select one probability value from the N probability values based on a positional relationship between the first camera and the second camera, and determine a weight of the vehicle based on the selected probability value.

[0174] In some optional embodiments, the apparatus further includes:

[0175] a road surface dividing unit 605 configured to divide a road surface in the first video image into M1 driving-in areas and M2 driving-out areas, each of the M1 driving-in areas including N driving-in sub-areas, M1 and M2 being positive integers.

[0176] The second determining unit 602 is configured to determine a target driving-in area corresponding to the vehicle from the M1 driving-in areas, determine a detection box of the vehicle based on the motion trajectory of the vehicle, determine an overlap rate of the detection box of the vehicle and the N driving-in sub-areas included in the target driving-in area, and determine overlap rates of the detection box of the vehicle and the M2 driving-out areas, and determine N prediction scores of the vehicle based on the determined overlap rates, each of the N prediction scores corresponding to a prediction score of the vehicle in a driving direction.

[0177] In some optional embodiments, the apparatus further includes:

[0178] a road surface mark identifying unit 606 configured to identify road surface marks in the first video image, the road surface marks including lane marks and / or lane lines.

[0179] The road surface dividing unit 605 is configured to divide the road surface in the first video image into the M1 driving-in areas and the M2 driving-out areas based on the road surface marks in the first video image.

[0180] In some optional embodiments, the first determining unit 601 is configured to identify features of each vehicle in the first video image, and identify features of each vehicle in the second video image; and calculate a first similarity matrix based on the features of each vehicle in the first video image and the features of each vehicle in the second video image, wherein each element in the first similarity matrix represents a feature distance between one vehicle in the first image and one vehicle in the second image.

[0181] In some optional implementations, the matching unit 604 is configured to calculate a sum of feature distances of pairs of vehicles in a vehicle matching relationship based on the second similarity matrix; adjust the vehicle matching relationship with a minimum sum of feature distances as an optimization target; and determine a vehicle matching result corresponding to the minimum sum of feature distances.

[0182] Those skilled in the art should understand that the above description of the vehicle matching apparatus of the embodiments of the present application can be understood with reference to the description of the vehicle matching method of the embodiments of the present application.

[0183] Figure 7 is a schematic structural diagram of an electronic device 700 provided by an embodiment of the present application. The electronic device can include the vehicle matching apparatus described in the above solutions. Figure 7 The electronic device 700 shown includes a processor 710, which can call and run a computer program from a memory to implement the method in the embodiments of the present application.

[0184] Optionally, as shown in Figure 7 The electronic device 700 can also include a memory 720. The processor 710 can call and run a computer program from the memory 720 to implement the method in the embodiments of the present application.

[0185] Optionally, as shown in Figure 7 The electronic device 700 can also include a transceiver 730, and the processor 710 can control the transceiver 730 to communicate with other devices, specifically, to send information or data to other devices or receive information or data sent by other devices.

[0186] Figure 8 is a schematic structural diagram of a chip according to an embodiment of the present application. Figure 8 The chip 800 shown includes a processor 810, which can call and run a computer program from a memory to implement the method in the embodiments of the present application.

[0187] Optionally, as shown in Figure 8 The chip 800 can also include a memory 820. The processor 810 can call and run a computer program from the memory 820 to implement the method in the embodiments of the present application.

[0188] The memory 820 can be a separate device independent of the processor 810, or can be integrated in the processor 810.

[0189] Optionally, the chip 800 can further include an input interface 830. The processor 810 can control the input interface 830 to communicate with other devices or chips, and specifically, can acquire information or data sent by other devices or chips.

[0190] Optionally, the chip 800 can further include an output interface 840. The processor 810 can control the output interface 840 to communicate with other devices or chips, and specifically, can output information or data to other devices or chips.

[0191] It should be understood that the chip mentioned in the embodiments of the present application can also be referred to as a system chip, a system chip, a chip system or a system on chip, etc.

[0192] It should be understood that the processor of the embodiments of the present application can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method embodiments can be completed by integrated logic circuits or instructions in the form of software in the processor. The processor mentioned above can be a general processor, a digital signal processor (Digital Signal Processor, DSP), an application specific integrated circuit (Application Specific Integrated Circuit, ASIC), a ready programmable gate array (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as hardware code processor execution, or executed by hardware and software modules in the code processor. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is mature in the art. The storage medium is located in the memory, and the processor reads the information in the memory, and combines the hardware to complete the steps of the above method.

[0193] It is to be understood that the memory in the embodiments of the present application can be a volatile memory or a nonvolatile memory, or can include both volatile and nonvolatile memory. Among them, the nonvolatile memory can be a read-only memory (Read-Only Memory, ROM), a programmable read-only memory (Programmable ROM, PROM), an erasable programmable read-only memory (Erasable PROM, EPROM), an electrically erasable programmable read-only memory (Electrically EPROM, EEPROM) or a flash memory. The volatile memory can be a random access memory (Random Access Memory, RAM) used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (Static RAM, SRAM), dynamic random access memory (Dynamic RAM, DRAM), synchronous dynamic random access memory (Synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (Double Data Rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (Enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (Synchlink DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM). It should be noted that the memory of the system and method described herein is intended to include, but not limited to, these and any other suitable types of memory.

[0194] It should be understood that the above-mentioned memory is exemplary but not limiting, for example, the memory in the embodiments of the present application can also be static random access memory (static RAM, SRAM), dynamic random access memory (dynamic RAM, DRAM), synchronous dynamic random access memory (synchronous DRAM, SDRAM), double data rate synchronous dynamic random access memory (double data rate SDRAM, DDR SDRAM), enhanced synchronous dynamic random access memory (enhanced SDRAM, ESDRAM), synchronous link dynamic random access memory (synch link DRAM, SLDRAM) and direct memory bus random access memory (Direct Rambus RAM, DR RAM) and the like. That is, the memory in the embodiments of the present application is intended to include, but not limited to, these and any other suitable types of memory.

[0195] The embodiment of the present application further provides a computer readable storage medium for storing the computer program.

[0196] Optionally, the computer readable storage medium can be applied to the network device in the embodiment of the present application, and the computer program makes the computer execute the corresponding process realized by the network device in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0197] Optionally, the computer readable storage medium can be applied to the mobile terminal / terminal device in the embodiment of the present application, and the computer program makes the computer execute the corresponding process realized by the mobile terminal / terminal device in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0198] The embodiment of the present application further provides a computer program product comprising computer program instructions.

[0199] Optionally, the computer program product can be applied to the network device in the embodiment of the present application, and the computer program instructions make the computer execute the corresponding process realized by the network device in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0200] Optionally, the computer program product can be applied to the mobile terminal / terminal device in the embodiment of the present application, and the computer program instructions make the computer execute the corresponding process realized by the mobile terminal / terminal device in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0201] The embodiment of the present application further provides a computer program.

[0202] Optionally, the computer program can be applied to the network device in the embodiment of the present application, and when the computer program runs on the computer, makes the computer execute the corresponding process realized by the network device in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0203] Optionally, the computer program can be applied to the mobile terminal / terminal device in the embodiment of the present application, and when the computer program runs on the computer, makes the computer execute the corresponding process realized by the mobile terminal / terminal device in the various methods of the embodiment of the present application, which will not be repeated here for the sake of brevity.

[0204] Those skilled in the art can clearly understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized by electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0205] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working processes of the above-described system, device and unit can refer to the corresponding processes in the foregoing method embodiments, which will not be repeated here.

[0206] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the above-described device embodiments are merely schematic, for example, the division of the units is only a logical function division, and actual implementation can have another division manner, for example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0207] The units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0208] In addition, each functional unit in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically independently, or two or more units can be integrated into one unit.

[0209] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application essentially or the parts that contribute to the prior art or parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0210] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A vehicle matching method characterized by, The method comprises: determining a first similarity matrix between vehicles based on a first video image and a second video image, the first video image being captured by a first camera, and the second video image being captured by a second camera; determining a direction weight matrix based on a positional relationship between the first camera and the second camera and a vehicle trajectory in the first video image, the positional relationship being an included angle of the first camera and the second camera, and the vehicle trajectory being used to determine a probability of a vehicle in each driving direction; determining a second similarity matrix based on the direction weight matrix and the first similarity matrix; matching vehicles in the first video image and the second video image based on the second similarity matrix.

2. The method of claim 1, wherein, The determining of the direction weight matrix based on the positional relationship between the first camera and the second camera and the vehicle trajectory in the first video image comprises: determining a weight of each vehicle in the first video image based on the positional relationship between the first camera and the second camera and a motion trajectory of each vehicle in the first video image; determining the direction weight matrix based on the weight of each vehicle in the first video image.

3. The method of claim 2, wherein, The determining of the weight of each vehicle in the first video image based on the positional relationship between the first camera and the second camera and the motion trajectory of each vehicle in the first video image comprises: for each vehicle in the first video image, determining N prediction scores of the vehicle based on a motion trajectory of the vehicle, N being a positive integer; normalizing the N prediction scores to obtain N probability values, each of the N probability values being used to represent a probability of the vehicle in one driving direction; selecting one probability value from the N probability values based on the positional relationship between the first camera and the second camera, and determining the weight of the vehicle based on the selected probability value.

4. The method of claim 3, wherein, The determining of the N prediction scores of the vehicle based on the motion trajectory of the vehicle comprises: dividing a road surface in the first video image into M1 entry regions and M2 exit regions, wherein each of the M1 entry regions comprises N entry sub-regions, M1 and M2 being positive integers; determining a target entry region corresponding to the vehicle from the M1 entry regions; determining a bounding box of the vehicle based on the motion trajectory of the vehicle, and determining an overlap rate of the bounding box of the vehicle and N entry sub-regions included in the target entry region, and determining overlap rates of the bounding box of the vehicle and the M2 exit regions; determining the N prediction scores of the vehicle based on the determined overlap rates, each of the N prediction scores corresponding to a prediction score of the vehicle in one driving direction.

5. The method of claim 4, wherein, The dividing of the road surface in the first video image into the M1 entry regions and the M2 exit regions comprises: recognizing road surface identifiers in the first video image, the road surface identifiers comprising lane identifiers and / or lane lines. divide a road surface in the first video image into M1 entry regions and M2 exit regions based on road surface marks in the first video image.

6. The method according to any one of claims 1 to 5, characterized in that, The determining a first similarity matrix between vehicles based on the first video image and the second video image comprises: identifying features of each vehicle in the first video image, and identifying features of each vehicle in the second video image; calculating the first similarity matrix based on the features of each vehicle in the first video image and the features of each vehicle in the second video image, wherein each element in the first similarity matrix represents a feature distance between one vehicle in the first video image and one vehicle in the second video image.

7. The method according to any one of claims 1 to 5, characterized in that, The matching vehicles in the first video image and the second video image based on the second similarity matrix comprises: calculating a sum of feature distances of multiple pairs of vehicles under a vehicle matching relationship based on the second similarity matrix; adjusting the vehicle matching relationship with a minimum sum of feature distances as an optimization target; taking the vehicle matching relationship corresponding to the minimum sum of feature distances as a vehicle matching result.

8. A vehicle matching device characterized by comprising: The apparatus comprises: a first determining unit configured to determine a first similarity matrix between vehicles based on a first video image and a second video image, the first video image being captured by a first camera, and the second video image being captured by a second camera; a second determining unit configured to determine a direction weight matrix based on a positional relationship between the first camera and the second camera and vehicle trajectories in the first video image, the positional relationship being an included angle of the first camera and the second camera, and the vehicle trajectories being used to determine probabilities of vehicles in each driving direction; a third determining unit configured to determine a second similarity matrix based on the direction weight matrix and the first similarity matrix; a matching unit configured to match vehicles in the first video image and the second video image based on the second similarity matrix.

9. An electronic device, comprising: comprise: a processor and a memory for storing a computer program, the processor being configured to invoke and run the computer program stored in the memory to perform the method according to any one of claims 1 to 7.

10. A chip, characterized by comprise: a processor configured to invoke and run a computer program from a memory, so that a device installed with the chip performs the method according to any one of claims 1 to 7.

11. A computer readable storage medium, characterized in that, a computer program for causing a computer to perform the method according to any one of claims 1 to 7.

Images