Vehicle tracking method, computer equipment and storage device

Abstract

The present application discloses a vehicle tracking method, computer equipment and storage device. The method comprises: detecting a first vehicle part on a current video frame of a preset video to obtain a part detection frame to be processed; determining each candidate part detection frame corresponding to the part detection frame to be processed, the candidate part detection frame comprising a detection frame obtained by detecting a second vehicle part on a historical video frame; the first vehicle part and the second vehicle part are of different types, and the historical video frame comprises a video frame before the current video frame in the preset video; determining a target part detection frame from each candidate part detection frame, and associating the target part detection frame with the part detection frame to be processed; the second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle; tracking the target vehicle based on the associated target part detection frame and the part detection frame to be processed, which can improve the tracking efficiency of the target vehicle.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to a vehicle tracking method, computer equipment and storage device. Background Art

[0002] With the development of modern traffic roads and the popularization of intelligent traffic management systems, traffic safety has become increasingly important. Auxiliary methods such as residents' lives, electronic police, intelligent high-definition checkpoints, illegal parking evidence collection, and radar speeding capture play a key role in the field of intelligent transportation.

[0003] Intelligent transportation systems can track and capture vehicles at entrances and exits, parking lots, roads and other traffic scenes in various places. At present, most monitoring systems used for vehicle capture have recognition functions. They can capture vehicles during the tracking process and recognize the vehicle's license plate information after capturing the vehicle. At present, due to the instability of vehicle tracking, when capturing vehicles, vehicles may be missed or captured multiple times, and the vehicle's license plate information may not be accurately recognized, resulting in low vehicle tracking efficiency. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a vehicle tracking method, a computer device and a storage device, which can improve the tracking efficiency of the target vehicle.

[0005] In order to solve the above problems, the first aspect of the present application provides a vehicle tracking method, which includes: detecting a first vehicle part on a current video frame of a preset video to obtain a detection frame of a part to be processed; determining each candidate part detection frame corresponding to the detection frame of the part to be processed, the candidate part detection frame includes a detection frame obtained by detecting a second vehicle part on a historical video frame; the first vehicle part and the second vehicle part are of different types, and the historical video frame includes a video frame before the current video frame in the preset video; determining a target part detection frame from each candidate part detection frame, and associating the target part detection frame with the part detection frame to be processed; the second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle; tracking the target vehicle based on the associated target part detection frame and the part detection frame to be processed.

[0006] In order to solve the above problem, the second aspect of the present application provides a computer device, which includes a memory and a processor coupled to each other, the memory stores program data, and the processor is used to execute the program data to implement any step in the above vehicle tracking method.

[0007] In order to solve the above problem, the third aspect of the present application provides a storage device, which stores program data that can be executed by a processor, and the program data is used to implement any step in the above vehicle tracking method.

[0008] The above scheme detects the first vehicle part on the current video frame of the preset video to obtain the part detection frame to be processed; determines each candidate part detection frame corresponding to the part detection frame to be processed, and the candidate part detection frame includes the detection frame obtained by detecting the second vehicle part on the historical video frame; the first vehicle part and the second vehicle part are of different types, and the historical video frame includes the video frame before the current video frame in the preset video; by detecting the local key areas of the different parts of the vehicle, the detection frames corresponding to the first vehicle part and the second vehicle part are obtained, which can narrow the effective feature focus range of the target vehicle; determines the target part detection frame from each candidate part detection frame, and associates the target part detection frame with the part detection frame to be processed; wherein, the second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle; based on the associated target part detection frame and the part detection frame to be processed, the target vehicle is tracked, which can reduce the overlapping area of ​​the target vehicle when it intersects densely or at close range, and improve the stability of tracking the target vehicle, and the tracking efficiency of the target vehicle can be improved by tracking the target part detection frame and the part detection frame to be processed in a linked manner and capturing the complementary method. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in this application, the following briefly introduces the drawings required in the description of the embodiments. Obviously, the drawings described below are only some embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0010] Figure 1 It is a flow chart of the first embodiment of the vehicle tracking method of the present application;

[0011] Figure 2 is a schematic diagram of a first embodiment of a first vehicle part and a second vehicle part of a vehicle of the present application;

[0012] Figure 3 is a schematic diagram of a second embodiment of a first vehicle part and a second vehicle part of the vehicle of the present application;

[0013] Figure 4 is a schematic diagram of a third embodiment of a first vehicle part and a second vehicle part of the vehicle of the present application;

[0014] Figure 5 This application Figure 1The schematic diagram of the process of the first embodiment of step S13;

[0015] Figure 6 This application Figure 5 An example schematic diagram of an embodiment of step S1313;

[0016] Figure 7 This application Figure 1 The flowchart of the second embodiment of step S13 is shown in FIG.

[0017] Figure 8 This application Figure 1 The flowchart of step S13 in the third embodiment;

[0018] Fig. 9 This application Figure 1 A schematic diagram of a flow chart of an embodiment of step S14;

[0019] Fig.10 It is a flow chart of the second embodiment of the vehicle tracking method of the present application;

[0020] Fig.11 It is a flow chart of the third embodiment of the vehicle tracking method of the present application;

[0021] Fig.12 It is a structural schematic diagram of the first embodiment of the vehicle capture device of the present application;

[0022] Fig.13 It is a structural schematic diagram of the second embodiment of the vehicle capture device of the present application;

[0023] Fig.14 It is a structural schematic diagram of an embodiment of a computer device of the present application;

[0024] Fig.15 It is a structural diagram of an embodiment of a storage device of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0026] The terms "first" and "second" in this application are only used for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of indicated technical features. Thus, the features defined as "first" and "second" can explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "multiple" is at least two, such as two, three, etc., unless otherwise clearly and specifically defined. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device that includes a series of steps or units is not limited to the listed steps or units, but optionally also includes steps or units that are not listed, or optionally also includes other steps or units inherent to these processes, methods, products or devices.

[0027] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described herein may be combined with other embodiments.

[0028] The present application provides the following embodiments, and each embodiment is described in detail below.

[0029] See also Figure 1 , Figure 1 1 is a flow chart of the first embodiment of the vehicle tracking method of the present application. The method may include the following steps:

[0030] S11: Detecting a first vehicle part on a current video frame of a preset video to obtain a detection frame of the part to be processed.

[0031] The preset video is a video obtained by shooting a vehicle, and the preset video may be a surveillance video. For example, in a smart transportation scenario, a surveillance video may be obtained by shooting a vehicle on a road, parking lot, or the like using a surveillance system, wherein the surveillance video shot by the surveillance system in real time may be obtained, or the surveillance video may be shot by the surveillance system in a preset time period, and this application does not impose any restrictions on this.

[0032] In some embodiments, the first vehicle part may include at least one of a vehicle part, a vehicle face part, a license plate part, a vehicle light part, a vehicle window part, etc. The vehicle part includes the entire vehicle, and the vehicle face part includes the front vehicle face part and / or the rear vehicle face part.

[0033] In some embodiments, a detection model can be used to detect a first vehicle part on a video frame of a preset video to obtain a detection result of a detection frame of the part to be processed; wherein the detection result includes coordinate frame information, confidence, and classification result; the classification result can represent the category of the first vehicle part, for example, it can represent the category: vehicle part, vehicle face part, license plate part, headlight part, or window part.

[0034] In some implementations, the detection model is used to detect the first vehicle part of the current video frame of the preset video to obtain a detection frame of the part to be processed. If the current video frame includes multiple vehicles, the detection frames of the parts to be processed of the multiple vehicles can be obtained.

[0035] For example, if the first vehicle part includes the vehicle face part or the license plate part, the detection model can be used to locate and detect the vehicle face part and the license plate part of the vehicle in the current video frame of the preset video to obtain the detection result of the first vehicle part in the current video frame, that is, to obtain the detection frame and detection result corresponding to the vehicle face part or the license plate part. The detection result may include the detection result of the vehicle face detection frame or the license plate detection frame.

[0036] In some embodiments, the detection model can be an algorithm model based on deep learning. For example, the YOLO algorithm (You Only Look Once) can be used to detect the vehicle face parts and license plate parts of all vehicles in the video frames of the video to obtain the detection results of the first vehicle parts of all vehicles, that is, to output the vehicle face detection frames and license plate detection frames detected for all vehicles.

[0037] The detection result may include the position information (or coordinate information) of the detection frame, the classification result (the category to which the detection frame belongs), and the confidence corresponding to the detection frame. The category to which the detection frame belongs may include the category of the vehicle face part or the license plate part. The classification result of the detection frame may be represented by characters, for example, the first character "0" may be used to represent the category of the vehicle face part, and the second character "1" may be used to represent the category of the license plate part. Thus, the YOLO algorithm may be used to detect the detection result of each video frame in the preset video.

[0038] In some embodiments, after obtaining the detection result of the detection frame of the to-be-processed part of the vehicle in the current video frame of the preset video, it can be determined whether the confidence of the detection result is greater than a preset confidence threshold. If the confidence is greater than the preset confidence threshold, the video frame can be used as the starting video frame of the target vehicle for the current video frame, that is, the target vehicle can be tracked from the video frame. Among them, the preset confidence threshold can be a parameter preset by the user. That is, the vehicle of the current video frame can be used as the target vehicle, and the current video frame can be used as the starting video frame of the target vehicle, so that the target vehicle can be tracked in the video frame after the starting video frame. By using the current video frame with a confidence greater than the preset confidence threshold as the starting video frame of the target vehicle in the preset video, the effectiveness of the determined target vehicle can be improved, and the stability of the starting state of the effective target vehicle can be ensured.

[0039] S12: Determine each candidate part detection frame corresponding to the part detection frame to be processed, the candidate part detection frame includes a detection frame obtained by detecting the second vehicle part in a historical video frame; the first vehicle part and the second vehicle part are of different types, and the historical video frame includes a video frame before the current video frame in a preset video.

[0040] After obtaining the detection frame of the part to be processed, the candidate part detection frames corresponding to the detection frame of the part to be processed can be determined, and the candidate part detection frame includes the detection frame obtained by detecting the second vehicle part on the historical video frame. Prior to this, the second vehicle part can be detected on the historical video frame to obtain the candidate part detection frame, wherein the historical video frame includes the video frame before the current video frame in the preset video, and the second vehicle part can also be a vehicle part, a vehicle face part, a license plate part, a headlight part or a window part, and the second vehicle part is of a different type from the first vehicle part. In the present application, the process of obtaining the candidate part detection frame by detecting the second vehicle part on the historical video frame can refer to the above step S11, in which the first vehicle part is detected on the current video frame of the preset video to obtain the process of the detection frame of the part to be processed, and the present application will not repeat it here.

[0041] In some implementations, in step S11, during the process of detecting the first vehicle part on the current video frame of the preset video and obtaining the detection frame of the part to be processed, the second vehicle part may be detected on the current video frame of the preset video, and the type of the second vehicle part is different from that of the first vehicle part. This process may refer to the specific implementation process of the above step S11, and this application will not repeat it here.

[0042] In some embodiments, in step S12, each candidate part detection frame corresponding to the part detection frame to be processed can be determined, and the candidate part detection frame includes a detection frame obtained by detecting the second vehicle part of the current video frame; the types of the first vehicle part and the second vehicle part are different.

[0043] In some embodiments, in step S12, each candidate part detection frame corresponding to the part detection frame to be processed can be determined, and the candidate part detection frame includes a detection frame obtained by detecting the second vehicle part in the current video frame and the historical video frame; wherein the first vehicle part and the second vehicle part are of different types, and the historical video frame includes a video frame before the current video frame in a preset video.

[0044] In some embodiments, there is an overlapping area between the first vehicle part and the second vehicle part of the same target vehicle. The existence of an overlapping area may indicate that the first vehicle part and the second vehicle part are contained in the same target vehicle, for example, the first vehicle part contains the second vehicle part, or the second vehicle part contains the first vehicle part. The existence of an overlapping area may also indicate that there is a partially overlapping area between the positions of the first vehicle part and the second vehicle part in the same target vehicle. The first vehicle part and the second vehicle part are of different types. The first vehicle part and the second vehicle part may include at least one of a vehicle part, a vehicle face part, a license plate part, a vehicle light part, a vehicle window part, etc. For example, if the first vehicle part is a vehicle part, then the second part is at least one of a vehicle face part, a license plate part, a vehicle light part, a vehicle window part, etc. This application does not impose any restrictions on this.

[0045] In some embodiments, the first vehicle part includes at least one of a license plate part and a headlight part, and the second vehicle part includes a vehicle face part, wherein the vehicle face part includes a front vehicle face part and / or a rear vehicle face part. For example, the first vehicle part includes a license plate part, and the second vehicle part includes a vehicle face part. For example, the first vehicle part includes a headlight part, and the second vehicle part includes a vehicle face part.

[0046] In some embodiments, the first vehicle part includes a vehicle face part, and the second vehicle part includes at least one of a license plate part and a headlight part. For example, the first vehicle part includes a vehicle face part, and the second vehicle part includes a license plate part. For example, the first vehicle part includes a vehicle face part, and the second vehicle part includes a headlight part.

[0047] In some embodiments, the first vehicle part includes a vehicle part, and the second vehicle part includes at least one of a vehicle face part, a license plate part, a vehicle light part, and a vehicle window part. For example, if the first vehicle part includes a vehicle part, the second vehicle part may include a vehicle face part, a license plate part, a vehicle light part, or a vehicle window part.

[0048] In some embodiments, the first vehicle part includes at least one of a vehicle face part, a license plate part, a vehicle light part, and a vehicle window part, and the second vehicle part includes the vehicle part. For example, if the first vehicle part includes a vehicle face part, a license plate part, a vehicle light part, or a vehicle window part, then the second vehicle part may include a vehicle part.

[0049] It can be understood that the combination of the first vehicle part and the second vehicle part in the present application may also be other parts of the target vehicle, and the present application is not limited thereto.

[0050] Please refer to 2 to Figure 4 , the vehicle 100 may include multiple parts, for example, at least one of a vehicle part 101, a vehicle face part 102, a license plate part 103, a vehicle light part 104, and a vehicle window part 105, wherein the vehicle part 101, the vehicle face part 102, the license plate part 103, the vehicle light part 104, or the vehicle window part 105 may be used as a first vehicle part or a second vehicle part. The first vehicle part and the second vehicle part belonging to the same target vehicle are of different types, and there is an overlapped area between the first vehicle part and the second vehicle part. The sizes of the detection frames corresponding to the first vehicle part and the second vehicle part may be different.

[0051] Among them, there is an overlapping area between the vehicle part 101 and the vehicle face part 102 , the license plate part 103 , the headlight part 104 or the vehicle window part 105 , and there is an overlapping area between the vehicle face part 102 and the license plate part 103 or the headlight part 104 .

[0052] Taking the first vehicle part as the license plate part 103 and the second vehicle part as the vehicle face part 102 as an example, the license plate part 103 is detected on the current video frame, and a detection frame of the part to be processed corresponding to the license plate part 103 can be obtained. The candidate part detection frames corresponding to the detection frame of the part to be processed are determined, and the candidate part detection frames include the detection frames obtained by detecting the vehicle face part 102 on the historical video frame. The size of the detection frame of the part to be processed corresponding to the license plate part 103 can be smaller than the candidate part detection frame corresponding to the vehicle face part 102. Of course, in some application scenarios, they can also be the same size, and this application does not limit this.

[0053] S13: Determine a target part detection frame from each candidate part detection frame, and associate the target part detection frame with the part detection frame to be processed; wherein the second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle.

[0054] In order to associate the detection frames of different types of parts belonging to the same target vehicle, the target part detection frame can be determined from each candidate part detection frame, and the target part detection frame and the part detection frame to be processed can be associated. The second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle, that is, the part detection frame to be processed corresponding to the first vehicle part and the target part detection frame corresponding to the second vehicle part belong to the same target vehicle.

[0055] In some embodiments, the target part detection frame can be determined based on the similarity between each candidate part detection frame and the part detection frame to be processed. For example, the candidate part detection frame with the highest similarity can be used as the target part detection frame. The similarity between the candidate part detection frame and the part detection frame to be processed can be expressed in various ways, such as the overlap between the candidate part detection frame and the part detection frame to be processed, the similarity of feature information between the candidate part detection frame and the part to be processed, etc., and this application does not impose any restrictions on this.

[0056] In some embodiments, the target part detection frame can be determined based on the coordinate frame information of each candidate part detection frame and the part detection frame to be processed. The positional relationship between the candidate part detection frame and the part detection frame to be processed can be determined through the coordinate frame information of the candidate part detection frame and the part detection frame to be processed, so as to determine the target part detection frame through the positional relationship.

[0057] S14: Tracking the target vehicle based on the associated target part detection frame and the to-be-processed part detection frame.

[0058] Since the detection frames of the first vehicle part and the second vehicle part of different types belonging to the same target are associated, the target vehicle can be tracked based on the associated target part detection frame and the to-be-processed part detection frame. Since the target part detection frame and the to-be-processed part detection frame are associated, the target vehicle can be tracked in a video frame after the current video frame by the target part detection frame and the to-be-processed part detection frame of the target vehicle in a linked and complementary manner.

[0059] In this embodiment, by detecting local key areas of unused parts of the vehicle, detection frames corresponding to the first vehicle part and the second vehicle part are obtained, so that the effective feature focus range of the target vehicle can be narrowed; the target part detection frame is determined from each candidate part detection frame, and the target part detection frame and the to-be-processed part detection frame are associated; wherein, the second vehicle part identified by the target part detection frame and the first vehicle part identified by the to-be-processed part detection frame belong to the same target vehicle; based on the associated target part detection frame and to-be-processed part detection frame, the target vehicle is tracked, which can reduce the overlapping area of ​​the target vehicle when it intersects densely or at close range, and improve the stability of tracking the target vehicle, and by tracking and capturing the target part detection frame and the to-be-processed part detection frame in a linked manner and complementing each other, the tracking efficiency of the target vehicle can be improved.

[0060] In some embodiments, see Figure 5 In the above step S13, the target part detection frame can be determined based on the similarity between each candidate part detection frame and the part detection frame to be processed. In step S13, determining the target part detection frame from each candidate part detection frame can include the following steps:

[0061] S1311: Obtain the degree of overlap between each candidate part detection frame and the part detection frame to be processed.

[0062] The similarity may include a degree of overlap, wherein the degree of overlap may be represented by an area ratio of an overlapping region between a candidate part detection frame and a to-be-processed part detection frame.

[0063] In some implementations, the degree of overlap between each candidate part detection frame and the part detection frame to be processed, that is, the area ratio of the overlapping region, can be obtained based on the coordinate frame information of each candidate part detection frame and the part detection frame to be processed.

[0064] As an example, taking the example that the first vehicle part corresponding to the to-be-processed part detection frame is the license plate part, and the second vehicle part corresponding to the candidate part detection frame is the vehicle face part, the to-be-processed part detection frame can be represented as the license plate detection frame, and the candidate part detection frame can be represented as the vehicle face detection frame. The following describes the degree of overlap between each candidate part detection frame and the to-be-processed part detection frame.

[0065] Based on the coordinate frame information of the vehicle face detection frame and the license plate detection frame, the area ratio of the overlapping area of ​​the vehicle face detection frame and the license plate detection frame is obtained.

[0066] The coordinate frame information of the vehicle face detection frame can indicate the position information of the vehicle face detection frame, and the coordinate frame information of the license plate detection frame can indicate the position information of the license plate detection frame. The coordinate frame information includes at least one of the coordinate information of the preset corners of the detection frame, the center coordinates, the width information, and the height information. For example, in the coordinate frame information of the license plate detection frame, the coordinate information of the preset corners of the detection frame may include the upper left corner, the upper right corner, the lower right corner, the lower left corner, etc. The preset corner can be set according to the shape of the detection frame, and this application does not limit this.

[0067] Based on the information of the coordinate frames of each vehicle face detection frame and the license plate detection frame, the area ratio of the overlapping area of ​​each vehicle face detection frame and the license plate detection frame is obtained through IOS (Intersection Over Self) operation, so that the key pair relationship between the license plate detection frame and the vehicle face detection frame can be established based on the area ratio of the overlapping area.

[0068] Specifically, the coordinate frame information of each vehicle face detection frame and the license plate detection frame can be used to obtain the area value of the overlapped area of ​​each vehicle face detection frame and the license plate detection frame. The coordinate frame information includes at least one of the coordinate information of the preset corners of the detection frame, the center coordinates, the width information, and the height information.

[0069] Assume that the coordinate information of the car face detection frame is F_bbox(x f ,y f , w f ,h f ), where (x f ,y f ) represents the center coordinate of the vehicle face detection frame, w f Indicates the width information of the car face detection frame, h f Indicates the height information of the vehicle face detection frame. The license plate frame coordinate frame is P_bbox(x p ,y p , w p ,h p ), where (x p ,y p ) represents the center coordinate of the vehicle face detection frame, w p Indicates the width information of the car face detection frame, h p Indicates the height information of the vehicle face detection frame. The following judgment formula can be used to determine whether the vehicle face detection frame and the license plate detection frame intersect, as follows:

[0070]

[0071] If the above formula (1) holds true, the vehicle face detection frame and the license plate detection frame intersect, which means that there is an intersection area between the vehicle face detection frame and the license plate detection frame, that is, there is an overlap area. Otherwise, the above formula (1) does not hold true, the vehicle face detection frame and the license plate detection frame do not intersect, that is, there is no overlap area.

[0072] In some of the above methods, if the preset corner of the vehicle face detection frame and the license plate detection frame is the upper left corner, (x1, y1) is set as the coordinate information of the upper left corner of the license plate detection frame P_bbox, and (x2, y2) is set as the coordinate information of the upper left corner of the vehicle face detection frame F_bbox, then the coordinate information of the upper left corner of the vehicle face detection frame and the license plate detection frame is calculated, and the calculation formula is as follows:

[0073]

[0074] From the above, we can get the coordinate information bbox[x left ,y top , x right ,y bottom ], where the coordinate information of the overlapping area can be obtained by the following formula, as follows:

[0075]

[0076] The area value Area_inter of the overlapped area between the license plate detection frame and the vehicle face detection frame can be obtained by the following calculation formula, where the calculation formula is:

[0077]

[0078] In the above formula (4), area_width represents the width information of the overlapping area, area_height represents the height information of the overlapping area, and Area_inter represents the area value of the overlapping area.

[0079] In some implementations, the ratio of the area value of the overlapped area between the license plate detection frame and the vehicle face detection frame to the area value of the license plate detection frame may be used as the area ratio of the overlapped area between each vehicle face detection frame and the license plate detection frame. The area ratio of the overlapped area may be expressed by the following formula:

[0080]

[0081] In the above formula (5), IOS_plate represents the area ratio of the overlapping area between the vehicle face detection frame and the license plate detection frame, Area_inter represents the area value of the overlapping area, and w p *h pThe area value of the license plate detection frame, where the area value is also the width information w of the license plate detection frame p With height information h p The product of .

[0082] In some of the above methods, the area ratio IOS_plate of the overlapping area of ​​each vehicle face detection frame and the license plate detection frame can be used as the overlap degree of each candidate part detection frame and the part detection frame to be processed.

[0083] S1312: Determine the candidate part detection frame corresponding to the overlap degree greater than the preset overlap threshold as the target part candidate detection frame.

[0084] After obtaining the degree of overlap between each candidate part detection frame and the part detection frame to be processed, each degree of overlap can be compared with a preset overlap threshold to determine the candidate part detection frame corresponding to the degree of overlap greater than the preset overlap threshold, so that the candidate part detection frame can be used as the candidate detection frame of the target part.

[0085] As an example, the first vehicle part corresponding to the part detection frame to be processed is the license plate part, and the second vehicle part corresponding to the candidate part detection frame is the vehicle face part. The part detection frame to be processed can be represented as the license plate detection frame, and the candidate part detection frame can be represented as the vehicle face detection frame. It can be determined whether the area ratio of the overlapping area between each face detection frame and the license plate detection frame is greater than the preset overlap threshold. If it is determined that the area ratio of the overlapping area is greater than the preset overlap threshold, the vehicle face detection frame corresponding to the area greater than the preset overlap threshold can be used as the target part candidate detection frame.

[0086] S1313: Determine a target part detection frame based on the target part candidate detection frame.

[0087] In some implementations, if the number of target part candidate detection frames is one, the target part candidate detection frame may be determined as the target part detection frame.

[0088] In some embodiments, if there are multiple candidate target part detection frames, the maximum value of the overlap degree greater than a preset overlap threshold can be determined, and the target part candidate detection frame corresponding to the maximum value is determined as the target part detection frame.

[0089] In some embodiments, if there are multiple candidate target part detection frames, based on the Euclidean distance values ​​between the candidate target part detection frames and the part detection frames to be processed, the target part candidate detection frame corresponding to the minimum Euclidean distance value is determined as the target part detection frame.

[0090] As an example, taking the example that the first vehicle part corresponding to the part detection frame to be processed is the license plate part, and the second vehicle part corresponding to the candidate part detection frame is the vehicle face part, the part detection frame to be processed can be represented as the license plate detection frame, and the candidate part detection frame can be represented as the vehicle face detection frame.

[0091] If the area ratio of the overlapping area between the vehicle face detection frame and multiple license plate detection frames is greater than the preset overlap threshold, the Euclidean distance values ​​between the vehicle face detection frame and the coordinate frame information of each license plate detection frame greater than the preset overlap threshold are obtained respectively. For example, when the area ratio of the overlapping area between a vehicle face detection frame F_bbox and multiple license plate detection frames P_bbox is greater than the preset overlap threshold, the Euclidean distance values ​​between the vehicle face detection frame F_bbox and the multiple license plate detection frames P_bbox are obtained respectively.

[0092] In some implementations, the Euclidean distance formula can be used to obtain the center coordinate information (x f ,y f ) and the center coordinates (x p ,y p ) as the Euclidean distance between the vehicle face detection frame and the license plate detection frame. The Euclidean distance formula can be as follows:

[0093]

[0094] In the above formula (6), Eu represents the Euclidean distance between the vehicle face detection frame and the license plate detection frame, (x p ,y p ) represents the center coordinate of the license plate detection frame, (x f ,y f ) represents the center coordinate information of the vehicle face detection frame.

[0095] The vehicle face detection frame corresponding to the smallest Euclidean distance value is determined as the target part detection frame to associate the vehicle face detection frame with the license plate detection frame.

[0096] See also Figure 6, multiple vehicle face detection frames can be used as vehicle face queues, and multiple license plate detection frames can be used as license plate queues. A key pair relationship can be established between the license plate detection frames in the license plate queue and the multiple vehicle face detection frames in the vehicle face queue, and the area ratio values ​​of the overlapping areas between the license plate detection frames in the license plate queue and the multiple vehicle face detection frames in the vehicle face queue are obtained in sequence, for example, the area ratio values ​​of the overlapping areas between the license plate detection frame 1 and the vehicle face detection frame 1, the vehicle face detection frame 2, the vehicle face detection frame 3, the vehicle face detection frame 4, the vehicle face detection frame 5 and the vehicle face detection frame 6 are obtained in sequence. If the area ratio value of the overlapping area between the license plate detection frame 1 and the vehicle face detection frame 1, the vehicle face detection frame 2, the vehicle face detection frame 3, the vehicle face detection frame 4 and the vehicle face detection frame 5 is greater than the preset overlap threshold, then the Euclidean distance values ​​between the license plate detection frame 1 and the vehicle face detection frame 1, the vehicle face detection frame 2, the vehicle face detection frame 3, the vehicle face detection frame 4 and the vehicle face detection frame 5 are obtained respectively. If the Euclidean distance between the vehicle license plate detection frame 1 and the vehicle face detection frame 1 is the smallest, the vehicle face detection frame 1 and the vehicle license plate detection frame 1 can be target associated.

[0097] In this embodiment, by obtaining the degree of overlap between each candidate part detection frame and the part detection frame to be processed, and determining the target part detection frame by combining the degree of overlap and the Euclidean distance value of the center points of the two, the target part detection frame can be effectively selected, thereby improving the reliability of the association between the part to be processed and the target part detection frame.

[0098] In some embodiments, see Figure 7 In the above step S13, the target part detection frame can be determined based on the coordinate frame information of each candidate part detection frame and the part detection frame to be processed. The step S13 determines the target part detection frame from each candidate part detection frame, which may include the following steps:

[0099] S1321: Obtain the positional relationship between the coordinate frame information of each candidate part detection frame and the to-be-processed part detection frame.

[0100] The coordinate frame information may include at least one of the coordinate information of the preset corners of the detection frame, the center coordinates, the width information, and the height information. For example, in the coordinate frame information of the license plate detection frame, the coordinate information of the preset corners of the detection frame may include the upper left corner, the upper right corner, the lower right corner, the lower left corner, etc. The preset corners may be set according to the shape of the detection frame, and this application does not limit this.

[0101] In some embodiments, the coordinate frame information includes the coordinates of the center point of the detection frame. The coordinate frame information of the candidate part detection frame and the part detection frame to be processed can be used to obtain the positional relationship between the coordinate frame information of the candidate part detection frame and the part detection frame to be processed. For example, the coordinate frame information can be used to determine whether there is an inclusion relationship between each candidate part detection frame and the part detection frame to be processed.

[0102] S1322: Determine the candidate part detection frame corresponding to the coordinate frame information having a positional inclusion relationship as the target part candidate detection frame.

[0103] Since there is an overlapping area between the candidate part detection frame and the part detection frame to be processed belonging to the same target vehicle, the overlapping area can represent an inclusion relationship or partial overlap. The candidate part detection frame with an inclusion relationship can be determined as the candidate part detection frame of the target part through the positional relationship between the candidate part detection frame and the part detection frame to be processed.

[0104] In some embodiments, the inclusion relationship includes: the coordinates of the center point of the candidate part detection frame are located in the to-be-processed part detection frame, and / or the coordinates of the center point of the to-be-processed part detection frame are located in the candidate part detection frame.

[0105] In some embodiments, if the coordinates of the center point of the candidate part detection frame are located in the part detection frame to be processed, it means that there is an overlapping area between the candidate part detection frame and the part detection frame to be processed, and it can be determined that there is an inclusion relationship between their positional relationship.

[0106] In some embodiments, if the coordinates of the center point of the to-be-processed part detection frame are located in the candidate part detection frame, it means that there is an overlapping area between the candidate part detection frame and the to-be-processed part detection frame, and it can be determined that there is an inclusion relationship between their positional relationship.

[0107] In some embodiments, if the coordinates of the center point of the candidate part detection frame are located in the part detection frame to be processed, and the coordinates of the center point of the part detection frame to be processed are located in the candidate part detection frame, it means that there is an overlapping area between the candidate part detection frame and the part detection frame to be processed, or, it means that the candidate part detection frame completely overlaps with the part detection frame to be processed, or, the part detection frame to be processed completely overlaps with the candidate part detection frame, and it can be determined that there is an inclusion relationship between their positional relationship.

[0108] S1323: Determine a target part detection frame based on the target part candidate detection frame.

[0109] In some implementations, if the number of target part candidate detection frames is one, the target part candidate detection frame may be determined as the target part detection frame.

[0110] In some embodiments, if there are multiple candidate target part detection frames, based on the Euclidean distance values ​​between the candidate target part detection frames and the part detection frames to be processed, the target part candidate detection frame corresponding to the minimum Euclidean distance value is determined as the target part detection frame.

[0111] In this embodiment, the target part detection frame can be determined by determining whether there is an inclusion relationship between each candidate part detection frame and the part detection frame to be processed. The target part detection frame can be effectively selected, thereby improving the reliability of the association between the part to be processed and the target part detection frame.

[0112] In some embodiments, see Figure 8 In the above step S13, associating the target part detection frame with the to-be-processed part detection frame may include the following steps:

[0113] S1331: Use the first part identifier or the second part identifier as the target vehicle identifier of the target vehicle.

[0114] The first vehicle part corresponds to the first part identifier, and the second vehicle part corresponds to the second part identifier. The to-be-processed part detection frame corresponds to the first part identifier, and the target part detection frame corresponds to the second part identifier.

[0115] Because the target part detection frame and the part detection frame to be processed belong to the same target vehicle, when associating the target part detection frame and the part detection frame to be processed, the identifiers corresponding to the target part detection frame and the part detection frame to be processed (target vehicle identifier) ​​can be unified, and the first part identifier or the second part identifier can be used as the target vehicle identifier of the target vehicle.

[0116] For example, the first vehicle part is the license plate part, and the second vehicle part is the vehicle face part. The first vehicle part corresponding to the to-be-processed part detection frame is the vehicle face part, and the second vehicle part corresponding to the target part detection frame is the vehicle face part. The to-be-processed part detection frame can be represented as the license plate detection frame, and the target part detection frame can be represented as the vehicle face detection frame. Since the vehicle face detection frame is provided with a corresponding vehicle face identifier (second part identifier), and the license plate detection frame is provided with a corresponding license plate identifier (first part identifier), after the vehicle face detection frame and the license plate detection frame are key-associated by the target, the vehicle face detection frame and the license plate detection frame can belong to the local feature area of ​​the same target vehicle. In order to make the associated vehicle face detection frame and license plate detection frame share the same identifier, the vehicle face detection frame and the license plate detection frame can share the target identifier (target vehicle identifier).

[0117] The identifier can be represented by an ID (Identity document, ID card identification number), for example, the vehicle face identifier and the license plate identifier can be represented by ID respectively, for example, the license plate identifier is 1, 2, 3, etc. In addition, the vehicle face detection frame and the license plate detection frame can also be provided with a target vehicle identifier, and the target vehicle identifier can be used to identify the vehicle face detection frame and the license plate detection frame of a certain target, and the target vehicle identifier can also be represented by ID. It can be understood that the identifier of the present application can also be represented by other methods, and the present application does not limit this. Among them, the target identifier is the vehicle face identifier of the vehicle face detection frame or the license plate identifier of the license plate detection frame. In the selection of the identifier of the vehicle face detection frame or the license plate detection frame, the one with a larger or longer number of frames in the cumulative valid state in the video can be used as the target identifier, and the identifier with a smaller value of the vehicle face identifier and the license plate identifier can also be used as the target identifier. Of course, the present application can also use other methods to determine the target identifier shared by the associated vehicle face detection frame and the license plate detection frame, and the present application does not limit this.

[0118] S1332: Setting a target vehicle identifier for the target part detection frame and the part detection frame to be processed, wherein the target vehicle identifier is used to identify that the second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle.

[0119] After unifying the target vehicle identification of the target part detection frame and the part detection frame to be processed belonging to the same target vehicle, the unified target vehicle identification is set to the second part identification corresponding to the target part detection frame and the first part identification corresponding to the part detection frame to be processed, to indicate that the target part detection frame and the part detection frame to be processed belong to the same target vehicle.

[0120] Since the successfully associated target part detection frame and the to-be-processed part detection frame share the same target vehicle identification, the capture rate of target vehicles in harsh environments can be effectively improved by tracking them in conjunction and capturing them in a complementary manner.

[0121] In some embodiments, see Fig. 9 In the above step S14, tracking the target vehicle based on the associated target part detection frame and the to-be-processed part detection frame may include the following steps:

[0122] S141: Tracking the target vehicle in subsequent video frames based on the target detection frame and the to-be-processed part detection frame after video association.

[0123] The associated target detection frame and the part detection frame to be processed share the same target vehicle identification and belong to the same target vehicle. When tracking the target vehicle in the subsequent video frames of the preset video, the target detection frame or the part detection frame to be processed in the subsequent video frames can be tracked separately. If any of the associated target detection frames and the part detection frame to be processed exists in the video frame, the target vehicle is successfully tracked, and it can be determined that the valid state of the target vehicle identification of the target vehicle continues to be valid until the target is captured.

[0124] S142: Setting the effective state of the target vehicle identification of the target detection frame and the part detection frame to be processed, where the effective state includes effective and invalid.

[0125] In the process of tracking the target vehicle in the subsequent video frames, the valid state of the target vehicle identification corresponding to the target vehicle can also be set, wherein the valid state includes valid and invalid.

[0126] In some embodiments, when setting the effective state of the target vehicle identification of the target detection frame and the part detection frame to be processed, each subsequent video frame of the preset video can be used as a video frame to be processed, and the following processing is performed on each video frame to be processed:

[0127] The first vehicle part and the second vehicle part are detected on the to-be-processed video frame of the preset video to obtain a to-be-matched part detection frame; wherein the to-be-matched part detection frame includes a first vehicle part detection frame corresponding to the first vehicle part and / or a second vehicle part detection frame corresponding to the second vehicle part. This process can refer to the implementation process of step S11 in the above embodiment, and this application will not repeat it here.

[0128] In some embodiments, if the part detection frame to be processed corresponding to the first vehicle part matches the first vehicle part detection frame, the effective state of the target vehicle is set to effective; and / or, if the target detection frame corresponding to the second vehicle part matches the second vehicle part detection frame, the effective state of the target vehicle identification is set to effective. That is, if the part detection frame to be processed matches one of the part detection frames to be matched corresponding to the video frame to be processed, it can be determined that the effective state of the target vehicle identification corresponding to the part detection frame to be processed is effective. For example, the part detection frame to be processed is a license plate detection frame or a vehicle face detection frame, and the part detection frame to be matched is a license plate detection frame to be matched or a vehicle face detection frame to be matched. In the vehicle face detection frame to be matched or the license plate detection frame to be matched detected with the processed video frame, if the license plate detection frame matches the license plate detection frame to be matched, or the vehicle face detection frame matches the vehicle face detection frame to be matched, then the effective state of the target vehicle identification corresponding to the target vehicle of the vehicle face detection frame and the license plate detection frame can be set to effective.

[0129] In some embodiments, if the to-be-processed part detection frame corresponding to the first vehicle part does not match the first vehicle part detection frame, and the target detection frame corresponding to the second vehicle part does not match the second vehicle part detection frame, the validity status of the target vehicle identification is set to invalid, and the target vehicle identification is deleted so that the target vehicle is no longer tracked.

[0130] In some embodiments, if the valid state of the target vehicle identification of the target vehicle of the to-be-processed video frames of consecutive K frames is invalid, that is, the to-be-processed detection frame of the target vehicle of the to-be-processed video frames of consecutive K frames does not match the to-be-matched part detection frame, then the valid state of the target vehicle identification is set to invalid, and the target vehicle identification is deleted to no longer track the target vehicle. Wherein, K is an integer greater than or equal to 1.

[0131] In some embodiments, if the moving displacement of the target vehicle in N consecutive frames of the to-be-processed video frames is less than the preset moving displacement, it can be indicated that the target vehicle is a stationary vehicle, or it can be defined as an invalid target, and the target can be not captured. Thus, the valid state of the target vehicle identification can be set to invalid, and the target vehicle identification can be deleted. Wherein, N is an integer greater than or equal to 1. The preset moving displacement can be expressed by the following formula:

[0132] Δs=h_valid*shift_ratio (7)

[0133] In the above formula (7), Δs represents the preset shift displacement, shift_ratio represents the effective shift coefficient for detecting the target, and h_valid represents the longitudinal height of the preset capture area of ​​the target.

[0134] See also Fig.10 , Fig.10 1 is a flow chart of the second embodiment of the vehicle tracking method of the present application. The method may include the following steps:

[0135] S21: Detecting a first vehicle part on a current video frame of a preset video to obtain a detection frame of the part to be processed.

[0136] The specific implementation process of this step can refer to the implementation process of step S11 in the above embodiment, which will not be repeated here.

[0137] S22: Determine whether the to-be-processed part detection frame matches the candidate part detection frame corresponding to the first vehicle part.

[0138] Among them, the candidate part detection frame corresponding to the first vehicle part includes the detection frame obtained by detecting the first vehicle part in the historical video frame.

[0139] The historical video frame is a video frame before the current video frame. The process of detecting the first vehicle part of the historical video frame to obtain the detection frame can refer to the specific implementation process of step S11 in the above embodiment, which will not be repeated here.

[0140] The valid state of the first part identification corresponding to the candidate part detection frame corresponding to the first vehicle part is valid. It can be determined whether the part detection frame to be processed matches the candidate part detection frame corresponding to the first vehicle part. For example, it can be determined whether the positions of the part detection frame to be processed and the candidate part detection frame are consistent. If they are consistent, it can be considered that the part detection frame to be processed matches the candidate part detection frame corresponding to the first vehicle part, otherwise, they do not match.

[0141] In step S22, if the to-be-processed part detection frame matches the candidate part detection frame corresponding to the first vehicle part, step S23 is executed.

[0142] If the to-be-processed part detection frame does not match the candidate part detection frame corresponding to the first vehicle part, step S24 is executed.

[0143] S23: setting the first part identifier corresponding to the part to be processed as the first part identifier corresponding to the candidate part detection frame corresponding to the first vehicle part.

[0144] If the part detection frame to be processed matches, it means that the part detection frame to be processed obtained by performing the first vehicle part detection on the current video frame and the candidate part detection frame obtained by performing the first vehicle part detection on the historical video frame belong to the same target vehicle, that is, the part detection frame to be processed and the candidate part detection frame corresponding to the first vehicle part belong to the same target vehicle, because the first vehicle part corresponds to the first part identifier and the second vehicle part corresponds to the second part identifier. The part detection frame to be processed and the candidate part detection frame can be set to the same first part identifier.

[0145] As an example, taking the example that the first vehicle part corresponding to the part detection frame to be processed is the license plate part, and the second vehicle part corresponding to the candidate part detection frame is the vehicle face part, the part detection frame to be processed can be represented as the license plate detection frame, and the candidate part detection frame can be represented as the vehicle face detection frame.

[0146] In some embodiments, a first flag bit may be set for the part detection frame to be processed, and a second flag bit may be set for the vehicle face detection frame and the license plate detection frame. If the part detection frame to be processed is matched successfully, the state of the first flag bit corresponding to the part detection frame to be processed is adjusted to "1", indicating that the detection frame result is matched successfully, and the state of the second flag bit corresponding to the vehicle face detection frame or the license plate detection frame that successfully matches the part detection frame to be processed is set to "1", which is used to indicate that the matched vehicle face detection frame or the license plate detection frame is matched, so that the identifier is tracked based on the state of the second flag bit of the vehicle face detection frame or the license plate detection frame. Similarly, if the part detection frame to be processed is not matched successfully, the state of the first flag bit corresponding to the part detection frame to be processed is kept set to "0", indicating that the part detection frame to be processed is not matched successfully. If the state of the second flag bit corresponding to the vehicle face detection frame or the license plate detection frame in the current video frame is "0", it means that there is no part detection frame to be processed in the current video frame that matches the vehicle face detection frame or the license plate detection frame, and active tracking is required.

[0147] S24: Allocate a new first part identifier for the first part identifier corresponding to the part detection frame to be processed.

[0148] If the part detection frame to be processed does not match, it means that the part detection frame to be processed and the candidate part detection frame corresponding to the first vehicle part do not belong to the same target vehicle, and need to be assigned a different first part identifier. Then, a new first part identifier can be assigned to the part detection frame to make the first vehicle part identified by the part detection frame to be processed belong to another target vehicle.

[0149] S25: Extract the geometric moment feature information of the detection frame of the part to be processed, and use the coordinate frame information of the candidate part detection frame corresponding to the first vehicle part of the historical video frame to obtain the main direction of movement of the target vehicle, and use the preset video frame rate and the movement rate of the target vehicle to obtain the position information of the target vehicle in the current video frame.

[0150] The detection frame to be processed belongs to different target vehicles, and the detection frame to be processed needs to be actively tracked.

[0151] Extract the geometric moment feature information of the detection frame of the part to be processed, wherein the Hu moment feature descriptor can be used to perform feature compression on the detection frame of the part to be processed to extract a 1*7 Hu moment feature vector (that is, geometric feature information). The Hu moment feature vector of an image is an image feature that is invariant to translation, rotation, and scale.

[0152] Specifically, for example, for a two-dimensional discrete image function f(x, y) of size M×N, the two-dimensional discrete image function f(x, y) can represent the data information of the current video frame of the video, and the current video frame of the video is discretized using the (p+q)-order standard moment calculation formula, where the (p+q)-order standard moment calculation formula can be expressed as:

[0153]

[0154] In the above formula (8), N and M represent the columns and rows of the video frame respectively; pq The physical meaning of each moment can be expressed as follows: For example, the 0th moment (m 00 ): quality of the target area; first-order moment (m 01 ,m 10 ): The center of mass of the target area; 2nd moment (m 02 ,m 11 ,m 20 ): rotation radius of the target area; third-order moment (m 03 ,m 12 ,m 21 ,m 30 ): The orientation and slope of the target area, reflecting the distortion of the target. The target area can be the area where the detection box result is located.

[0155] If we choose to construct the central moment with the centroid of the target area as the center, the moment calculation is always the point in the target area relative to the centroid of the target area, and has nothing to do with the position of the target area, and has translation invariance. The following (p+q)-order central moment calculation formula can be used to construct the central moment:

[0156]

[0157] In the above formula (9), p = 0, 1, 2, ...; q = 0, 1, 2 ..., the centroid coordinates of the target area can be obtained by the 0th-order origin moment and the 1st-order origin moment, where Represents the center of mass of the image. The coordinates of the center of mass can be expressed as:

[0158] In order to offset the impact of scale change on the central moment, the above formulas (8) and (9) are used to normalize the central distances of each order to obtain the normalized central moment:

[0159]

[0160] From the above formula, we can see that the 0th-order moment represents the mass (area) of the target area. If the scale of the target area changes (reduced by 2 times), its 0th-order central moment will obviously decrease accordingly, making the moment scale-invariant.

[0161] Using the normalized 2nd and 3rd order moments and central moment η pq , we can get the following 7 invariant moment groups, namely Hu moment vectors [M1, M2, …, M7], which remain unchanged when the image is translated, rotated, and scaled. They are as follows:

[0162]

[0163] The above Hu moment feature vector can be used as geometric moment feature information of the detection frame of the part to be processed.

[0164] The coordinate frame information of the candidate part detection frame corresponding to the first vehicle part of the historical video frame is used to obtain the main direction of the target vehicle's movement, and the position information of the target vehicle in the current video frame is obtained by using the preset video frame rate and the movement speed of the target vehicle.

[0165] Specifically, for example, if the state value of the second flag bit of the identifier of the valid license plate detection frame or vehicle face detection frame in the current video frame is not 1, for example, 0, active tracking is required. The coordinate information of the candidate part detection frame in the historical video frame of the preset video can be fitted by linear regression. For example, the coordinate information can be the center coordinates of the candidate part detection frame. The displacement vector of the center coordinates of the candidate part detection frame in the historical video frame can be fitted by linear regression to obtain the main direction of the movement of the target vehicle, that is, to simulate the moving direction of the target vehicle. Among them, the objective function equation of the displacement linear regression of the target vehicle is:

[0166]

[0167] In the above formula (12), minf represents the minimum value of the objective function of displacement linear regression, x, y are the coordinates of the center point of the detection box result in the target displacement vector, and variables k and b are constants.

[0168] In some implementations, the optimal solution of the objective function f of the above formula (12) is obtained, that is, the solution of the variables k and b is obtained so that the value of the objective function f is minimized.

[0169] In the process of finding the solution of variables k and b, it is necessary to calculate the first-order partial derivatives on both sides of the formula and set the partial derivative function equal to 0, and we can get:

[0170]

[0171] After simplifying and splitting the above formula (13), we can get the following form:

[0172]

[0173] Let the above formula Then the variables k and b are calculated by formula (14) as follows:

[0174]

[0175] In some embodiments, the obtained variable parameters k and b are brought into the above parameter equation (objective function minf), and combined with the frame rate of the preset video and the movement rate of the target, the movement rate can be the displacement speed of the target vehicle, wherein the frame rate is known, and the time interval between two video frames is 1 / f seconds as the frame rate, and the displacement can be calculated in combination with the displacement speed, so that the position information of the target vehicle in the current video frame can be obtained by using the frame rate of the preset video and the movement rate of the target vehicle, and the iterative starting position of the target vehicle corresponding to the candidate part detection frame of the historical video frame for active tracking in the current video frame is estimated. In this way, the time consumption of active tracking can be reduced and the accuracy of the positioning result can be guaranteed.

[0176] S26: Perform feature matching using the geometric moment feature information and the position information to obtain a detection frame corresponding to the first vehicle part in the current video frame.

[0177] The geometric moment feature information of the detection frame of the part to be processed is used to perform feature matching with the position information of the target vehicle in the current video frame to determine the detection frame corresponding to the target vehicle in the current video frame corresponding to the candidate part detection frame, that is, the detection frame of the part to be processed that matches the first vehicle part of the target vehicle. The detection frame of the part to be processed with the highest overlap of feature matching can be selected as the detection frame corresponding to the first vehicle part detection of the target vehicle in the current video frame. Active tracking in this way can effectively reduce the complexity of the target iterative search and positioning time.

[0178] In some implementations, the present embodiment may also implement the above steps for the second vehicle part. Specifically, the second vehicle part is detected on the current video frame of the preset video to obtain a detection frame of the part to be processed. It is determined whether the detection frame of the part to be processed matches the candidate part detection frame corresponding to the second vehicle part, wherein the candidate part detection frame corresponding to the second vehicle part includes the detection frame obtained by detecting the second vehicle part on the historical video frame. If the detection frame of the part to be processed matches, the second part identifier corresponding to the part to be processed is set to the second part identifier corresponding to the candidate part detection frame corresponding to the second vehicle part. If the detection frame of the part to be processed does not match, a new second part identifier is assigned to the second part identifier corresponding to the part to be processed. The geometric moment feature information of the detection frame of the part to be processed is extracted, and the coordinate frame information of the candidate part detection frame corresponding to the second vehicle part of the historical video frame is used to obtain the main direction of the movement of the target vehicle, and the position information of the target vehicle in the current video frame is obtained by using the frame rate of the preset video and the movement rate of the target vehicle; feature matching is performed using the geometric moment feature information and the position information to obtain the detection frame corresponding to the second vehicle part in the current video frame.

[0179] In this embodiment, after step S23 or step S26, the above-mentioned step S12 to step S14 may be performed, which will not be described in detail here.

[0180] See also Fig.11 , Fig.11 1 is a flow chart of the third embodiment of the vehicle tracking method of the present application. The method may include the following steps:

[0181] S31: During the process of tracking the target vehicle, if it is detected that the target vehicle performs a preset action in the preset capture area, the target vehicle is captured.

[0182] Among them, the preset capture area is an effective capture area of ​​a convex polygon. The effective capture area of ​​a convex polygon can be set according to the specific capture section and can adapt to a variety of scenarios. This application does not impose any restrictions on this.

[0183] The preset action includes the target vehicle's movement displacement reaching the preset capture displacement. In the process of tracking the target vehicle in the video frame, if the target's movement displacement in the preset capture area is detected in the video frame to be greater than or equal to the preset movement displacement, the capture requirement for the target vehicle is met, and the target vehicle can be captured. In the process of capturing the target vehicle, the valid state of the target vehicle identifier of the target vehicle is valid.

[0184] In this embodiment, the preset capture area is an effective capture area of ​​a convex polygon. The traditional capture line is replaced by the effective capture area of ​​the convex polygon. Combined with whether the target has a preset capture displacement in the capture area, it is used as a judgment condition for capturing the target vehicle. A flexible capture operation is adopted for the target, using "surface" instead of "line", which can effectively filter out random invalid targets and improve the capture rate of the target vehicle.

[0185] In some implementations, a target vehicle that meets the capture requirements may be captured multiple times to obtain multiple capture results of the target vehicle.

[0186] In some implementations, multiple snapshots of a target vehicle that meets the snapshot requirements may be taken at multiple locations and at different times to obtain multiple snapshot results of the target vehicle.

[0187] In some implementations, the above-mentioned steps S11 to S14 may be included before step S31, which will not be described in detail in this application.

[0188] S32: Perform license plate recognition on the captured result of the target vehicle to obtain license plate recognition information of the target vehicle.

[0189] The license plate recognition technology can be used to perform license plate recognition on the captured results of the target vehicle to obtain the license plate recognition information of the target vehicle. Among them, the license plate recognition technology refers to the technology of automatically reading the license plate number, license plate type, license plate color and other information after the vehicle image or video sequence taken by the camera is processed by machine vision, image processing and pattern recognition algorithms. The license plate recognition process also includes algorithms such as license plate positioning, character segmentation, and character recognition. The license plate recognition technology has been widely used in various occasions of intelligent transportation systems. The license plate recognition process can refer to the license plate recognition technology of various occasions of the current intelligent transportation system, and this application will not repeat it.

[0190] In some embodiments, license plate recognition can be performed on multiple snapshot results of the target vehicle to obtain multiple license plate recognition information of the target vehicle. The multiple snapshot results can be snapshot results of the target vehicle from the same place, or can be snapshot results of the target vehicle from different places at different times, and this application does not limit this.

[0191] Since the capture results are obtained for invalid target vehicles that have been filtered out, the capture rate and image quality of the target vehicle are improved. The effect of license plate recognition depends on the image quality of the capture results. By detecting that the target vehicle's movement displacement in the preset capture area reaches the preset capture displacement, the capture result of the target vehicle is used for license plate recognition, which can improve the license plate recognition quality and save computing resources.

[0192] S33: storing the license plate recognition information of the target vehicle into the license plate recognition queue of the target vehicle.

[0193] The license plate recognition information of the same target vehicle may be stored in the license plate recognition queue of the target vehicle, so that the license plate recognition information belonging to the same target vehicle is stored in the same license plate recognition queue.

[0194] In some embodiments, multiple license plate recognition information of the same target vehicle can be stored in a license plate recognition queue of a preset length. The preset length can be set according to scenarios such as the recognition time and number of license plates, and this application does not impose any restrictions on this.

[0195] S34: Select the best license plate recognition information from the license plate recognition queue as the license plate recognition result of the target vehicle.

[0196] The best license plate recognition information can be selected from multiple license plate recognition information in the license plate recognition queue belonging to the same target vehicle, wherein the best license plate recognition information can be selected from the license plate recognition queue by voting as the license plate recognition result of the target vehicle.

[0197] The voting preference may be to select the license plate recognition information that appears the most times in the license plate recognition queue. For example, in the license plate recognition queue of the target vehicle, the license plate recognition information "A123456" appears the most times, and the license plate recognition information "A123456" in the license plate recognition queue may be used as the license plate recognition result of the target vehicle. Of course, the present application may also use other selection methods to select the best license plate recognition information, and the present application does not limit this.

[0198] S35: Reporting the capture result of the target vehicle and the license plate recognition result of the target vehicle.

[0199] After selecting the best license plate recognition information, the best license plate recognition information can be reported as the license plate recognition of the target vehicle. The report can be reported to a preset system, such as a parking management, traffic induction, traffic law enforcement, highway inspection, vehicle dispatch, vehicle detection, etc. This application does not impose any restrictions on this.

[0200] In some embodiments, the capture result of the target vehicle and the license plate recognition result of the target vehicle can also be reported, so that the correctness of the license plate recognition result can be verified again by manual or other verification methods. Among them, the license plate recognition result can be verified by the capture result to ensure the correctness of the license plate recognition result of the target vehicle.

[0201] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0202] For the above embodiment, the present application also provides a vehicle tracking device. Fig.12 , Fig.12 1 is a schematic diagram of the structure of the first embodiment of the license plate capture device of the present application. The license plate capture device 40 comprises: a detection module 41, a determination module 42, an association module 43 and a tracking module 44, wherein the detection module 41, the determination module 42, the association module 43 and the tracking module 44 are connected.

[0203] The detection module 41 is used to detect the first vehicle part on the current video frame of the preset video to obtain a detection frame of the part to be processed.

[0204] The determination module 42 is used to determine each candidate part detection frame corresponding to the part detection frame to be processed, and the candidate part detection frame includes the detection frame obtained by detecting the second vehicle part in the historical video frame; wherein, the types of the first vehicle part and the second vehicle part are different, and the historical video frame includes the video frame before the current video frame in the preset video.

[0205] The association module 43 is used to determine the target part detection frame from each candidate part detection frame, and associate the target part detection frame with the part detection frame to be processed; wherein the second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle.

[0206] The tracking module 44 is used to track the target vehicle based on the associated target part detection frame and the to-be-processed part detection frame.

[0207] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0208] In some embodiments, see Fig.13 , Fig.13 The license plate capturing device 40 may further include a matching module 45 and a capturing module 46, wherein the detection module 41, the determination module 42, the association module 43, the tracking module 44, the matching module 45 and the capturing module 46 are connected.

[0209] The matching module 45 is used to determine whether the part detection frame to be processed matches the candidate part detection frame corresponding to the first vehicle part, wherein the candidate part detection frame corresponding to the first vehicle part includes the detection frame obtained by detecting the first vehicle part in the historical video frame.

[0210] If the detection frame of the part to be processed matches, the first part identifier corresponding to the part to be processed is set to the first part identifier corresponding to the candidate part detection frame corresponding to the first vehicle part.

[0211] If the detection frame of the part to be processed does not match, the geometric moment feature information of the detection frame of the part to be processed is extracted, and the coordinate frame information of the candidate part detection frame corresponding to the first vehicle part of the historical video frame is used to obtain the main direction of movement of the target vehicle, and the frame rate of the preset video and the movement rate of the target vehicle are used to obtain the position information of the target vehicle in the current video frame; the geometric moment feature information and the position information are used for feature matching to obtain the detection frame corresponding to the detection of the first vehicle part in the current video frame.

[0212] If the detection frame of the part to be processed does not match, a new first part identifier is allocated to the first part identifier corresponding to the detection frame of the part to be processed.

[0213] The capture module 46 is used to capture the target vehicle if it is detected that the target vehicle performs a preset action in a preset capture area during the process of tracking the target vehicle.

[0214] The preset capture area includes a valid capture area of ​​a convex polygon; and the preset action includes the moving displacement of the target vehicle reaching the preset capture displacement.

[0215] In some implementations, the capture module 46 can also be used to perform license plate recognition on the capture result of the target vehicle to obtain the license plate recognition information of the target vehicle; store the license plate recognition information of the target vehicle in the license plate recognition queue of the target vehicle; select the best license plate recognition information from the license plate recognition queue as the license plate recognition result of the target vehicle; and report the capture result of the target vehicle and the license plate recognition result of the target vehicle. Of course, this implementation can also be executed by other modules with the implementation function of this embodiment, and the present application is not limited thereto.

[0216] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0217] For the above embodiment, the present application provides a computer device, see Fig.14 , Fig.14 1 is a schematic diagram of the structure of an embodiment of a computer device of the present application. The computer device 50 includes a memory 51 and a processor 52, wherein the memory 51 and the processor 52 are coupled to each other, the memory 51 stores program data, and the processor 52 is used to execute the program data to implement the steps in any embodiment of the above-mentioned vehicle tracking method.

[0218] In this embodiment, the processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having signal processing capabilities. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. A general-purpose processor may be a microprocessor or the processor 52 may also be any conventional processor, etc.

[0219] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0220] The method of the above embodiment can be implemented in the form of a computer program, so the present application proposes a storage device, see Fig.15 , Fig.15 The storage device 60 stores program data 61 that can be run by a processor, and the program data 61 can be executed by the processor to implement the steps of any embodiment of the above vehicle tracking method.

[0221] The specific implementation of this embodiment can refer to the implementation process of the above embodiment, which will not be repeated here.

[0222] The storage device 60 of this embodiment can be a medium that can store program data, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, or it can also be a server that stores the program data. The server can send the stored program data to other devices for execution, or it can also execute the stored program data itself.

[0223] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0224] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0225] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0226] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a storage device, which is a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) or a processor to perform all or part of the steps of each implementation method of the present application.

[0227] Obviously, those skilled in the art should understand that the above modules or steps of the present application can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. In this way, the present application is not limited to any specific combination of hardware and software.

[0228] The above descriptions are merely embodiments of the present application and are not intended to limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A vehicle tracking method, characterized in that: The method comprises: Performing detection of a first vehicle part on a current video frame of a preset video to obtain a detection frame of a part to be processed; Determine each candidate part detection frame corresponding to the part detection frame to be processed, the candidate part detection frame includes a detection frame obtained by detecting a second vehicle part in a historical video frame; the first vehicle part and the second vehicle part are of different types, and the historical video frame includes a video frame before the current video frame in the preset video; Determine a target part detection frame from the candidate part detection frames, and associate the target part detection frame with the part detection frame to be processed; wherein the second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle; Tracking the target vehicle based on the associated target part detection frame and the to-be-processed part detection frame; Wherein, after detecting the first vehicle part on the current video frame of the preset video to obtain a detection frame of the part to be processed, the method further includes: Determine whether the to-be-processed part detection frame matches a candidate part detection frame corresponding to the first vehicle part, wherein the candidate part detection frame corresponding to the first vehicle part includes a detection frame obtained by detecting the first vehicle part on the historical video frame; If the detection frame of the part to be processed does not match, the geometric moment feature information of the detection frame of the part to be processed is extracted, and the coordinate frame information of the candidate part detection frame corresponding to the first vehicle part of the historical video frame is used to obtain the main direction of movement of the target vehicle, and the frame rate of the preset video and the movement rate of the target vehicle are used to obtain the position information of the target vehicle in the current video frame; the geometric moment feature information and the position information are used to perform feature matching to obtain the detection frame corresponding to the first vehicle part in the current video frame.

2. The method according to claim 1, characterized in that The step of determining a target part detection frame from each of the candidate part detection frames comprises: Determine the target part detection frame based on the similarity between each candidate part detection frame and the part detection frame to be processed; or, The target part detection frame is determined based on the coordinate frame information of each candidate part detection frame and the to-be-processed part detection frame.

3. The method according to claim 2, characterized in that The similarity includes the degree of overlap, and the target part detection frame is determined based on the similarity between each candidate part detection frame and the part detection frame to be processed, including: Obtaining the degree of overlap between each candidate part detection frame and the to-be-processed part detection frame; Determine the candidate part detection frame corresponding to the overlap degree greater than a preset overlap threshold as the target part candidate detection frame; Based on the target part candidate detection frame, the target part detection frame is determined.

4. The method according to claim 3, characterized in that The determining the target part detection frame based on the target part candidate detection frame includes: If the number of the target part candidate detection frame is one, determining the target part candidate detection frame as the target part detection frame; If there are multiple candidate detection frames for the target part, based on the Euclidean distance values ​​between the candidate detection frames for the target part and the detection frames for the part to be processed, the candidate detection frame for the target part corresponding to the minimum value of the Euclidean distance value is determined as the target part detection frame; or, the maximum value of the degree of overlap that is greater than the preset overlap threshold is determined, and the candidate detection frame for the target part corresponding to the maximum value is determined as the target part detection frame.

5. The method according to claim 2, characterized in that: The step of determining the target part detection frame based on the coordinate frame information of each candidate part detection frame and the part detection frame to be processed includes: Obtaining the positional relationship between the coordinate frame information of each candidate part detection frame and the to-be-processed part detection frame; Determine the candidate part detection frame corresponding to the coordinate frame information having a containment relationship in the positional relationship as the target part candidate detection frame; wherein the coordinate frame information includes the center point coordinates of the detection frame, and the containment relationship includes: the center point coordinates of the candidate part detection frame are located in the part detection frame to be processed, and / or the center point coordinates of the part detection frame to be processed are located in the candidate part detection frame; Based on the target part candidate detection frame, the target part detection frame is determined.

6. The method according to claim 3, characterized in that There is an overlapping area between the first vehicle part and the second vehicle part of the same target vehicle.

7. The method according to claim 6, characterized in that The first vehicle part includes at least one of a license plate part and a headlight part, and the second vehicle part includes a vehicle face part, wherein the vehicle face part includes a front vehicle face part and / or a rear vehicle face part; or, The first vehicle part includes the vehicle face part, and the second vehicle part includes at least one of the license plate part and the headlight part; or The first vehicle part includes a vehicle part, and the second vehicle part includes at least one of the vehicle face part, the license plate part, the headlight part, and the window part; or, The first vehicle part includes at least one of the vehicle face part, the license plate part, the headlight part, and the window part, and the second vehicle part includes the vehicle part.

8. The method according to claim 1, characterized in that The first vehicle part corresponds to a first part identifier, and the second vehicle part corresponds to a second part identifier; The associating the target part detection frame with the to-be-processed part detection frame comprises: Using the first part identification or the second part identification as the target vehicle identification of the target vehicle; The target vehicle identifier is set for the target part detection frame and the part detection frame to be processed; wherein the target vehicle identifier is used to identify that the second vehicle part identified by the target part detection frame and the first vehicle part identified by the part detection frame to be processed belong to the same target vehicle.

9. The method according to claim 1, characterized in that: The step of tracking the target vehicle based on the associated target part detection frame and the to-be-processed part detection frame comprises: Based on the associated target part detection frame and the to-be-processed part detection frame, tracking the target vehicle in subsequent video frames of the preset video; The valid state of the target vehicle identification of the target part detection frame and the part to be processed detection frame is set, and the valid state includes valid and invalid.

10. The method according to claim 9, characterized in that The setting of the valid state of the target vehicle identification of the target part detection frame and the to-be-processed part detection frame includes: Each subsequent video frame of the preset video is used as a video frame to be processed, and the following processing is performed on each of the video frames to be processed: Detecting the first vehicle part and the second vehicle part on the to-be-processed video frame of the preset video to obtain a to-be-matched part detection frame; wherein the to-be-matched part detection frame includes a first vehicle part detection frame corresponding to the first vehicle part and / or a second vehicle part detection frame corresponding to the second vehicle part; If the to-be-processed part detection frame corresponding to the first vehicle part matches the first vehicle part detection frame, the validity state of the target vehicle is set to the valid state; and / or if the target part detection frame corresponding to the second vehicle part matches the second vehicle part detection frame, the validity state of the target vehicle identification is set to the valid state; If the to-be-processed part detection frame corresponding to the first vehicle part does not match the first vehicle part detection frame, and the target part detection frame corresponding to the second vehicle part does not match the second vehicle part detection frame, then the validity state of the target vehicle identification is set to invalid, and the target vehicle identification is deleted; or If the moving displacement of the target vehicle in N consecutive frames of the video frames to be processed is less than the preset moving displacement, the validity state of the target vehicle identification is set to invalid, and the target vehicle identification is deleted, wherein N is an integer greater than or equal to 1.

11. The method according to claim 1, characterized in that: The method further comprises: In the process of tracking the target vehicle, if it is detected that the target vehicle performs a preset action in a preset capture area, the target vehicle is captured.

12. The method according to claim 11, characterized in that The preset capture area includes a convex polygonal effective capture area; the preset action includes the moving displacement of the target vehicle reaching the preset capture displacement.

13. The method according to claim 11, characterized in that After capturing the target, the method further includes: Performing license plate recognition on the captured result of the target vehicle to obtain license plate recognition information of the target vehicle; Storing the license plate recognition information of the target vehicle in the license plate recognition queue of the target vehicle; Selecting the best license plate recognition information from the license plate recognition queue as the license plate recognition result of the target vehicle; The capture result of the target vehicle and the license plate recognition result of the target vehicle are reported.

14. The method according to claim 1, characterized in that The first vehicle part corresponds to a first part identifier, and the second vehicle part corresponds to a second part identifier; the method further includes: If the detection frame of the part to be processed matches, setting the first part identifier corresponding to the part to be processed as the first part identifier corresponding to the candidate part detection frame corresponding to the first vehicle part; If the to-be-processed part detection frame does not match, a new first part identifier is allocated for the first part identifier corresponding to the to-be-processed part detection frame.

15. The method according to claim 1, characterized in that The detecting of the first vehicle part on the current video frame of the preset video to obtain a detection frame of the part to be processed includes: Detecting the first vehicle part on the video frame of the preset video using the detection model to obtain a detection result of the detection frame of the part to be processed; wherein the detection result includes coordinate frame information and confidence level; If the confidence is greater than a preset confidence threshold, the current video frame is used as a starting video frame of the target vehicle.

16. A computer device, characterized in that: The invention comprises a memory and a processor coupled to each other, wherein the memory stores program data, and the processor is used to execute the program data to implement the steps of the method according to any one of claims 1 to 15.

17. A storage device, characterized in that: Program data that can be executed by a processor are stored, and the program data is used to implement the steps of the method according to any one of claims 1 to 15.

Images