Target tracking device and method for road monitoring video

Abstract

The embodiment of the present application provides a device and method for target tracking of road surveillance video. The method comprises: performing target detection on real-time surveillance video of a road area of ​​interest to obtain detection information of one or more targets; matching the detection information of the one or more targets with tracking information for target tracking; updating the tracking information according to the matching result; and updating the road information of the area of ​​interest according to the detection information and / or the tracking information. In this way, fast target tracking can be achieved, and the scale change and / or occlusion problems can be solved for real-time road surveillance video.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of video surveillance technology. Background Art

[0002] Video surveillance systems are widely used in life. For example, cameras can be used to monitor roads and detect various traffic events. In order to detect events, the configuration information of video surveillance can generally be pre-set; for example, the region of interest (ROI), lane width, lane direction, etc.

[0003] In traffic surveillance videos, the size of the detection target (or object, such as a vehicle) decreases from near to far, and obstacles or other objects may block the detection target. Scale change and occlusion are one of the problems of traditional object tracking.

[0004] Traffic analysis based on artificial intelligence (AI) technology is a hot research field that can detect traffic incidents, measure vehicle speeds, and calculate traffic flow. Almost all of these applications are based on some basic modules, such as objection detection, multi-target tracking, event judgment, etc. Tracking of traffic targets is one of the most important parts.

[0005] It should be noted that the above introduction to the technical background is only for the convenience of providing a clear and complete description of the technical solutions of the present application and for the convenience of understanding by those skilled in the art. It cannot be considered that the above technical solutions are well known to those skilled in the art simply because they are described in the background technology section of the present application. Summary of the invention

[0006] The inventors found that the latest tracking method based on deep learning can solve some problems of scale change and occlusion, but it requires a lot of calculations and also requires prior training for application scenarios.

[0007] For a real-time road monitoring system, there are multiple modules that need to work together, including scene understanding, object detection, tracking, etc.; therefore, the processing time of each module should be balanced. Usually, the detection module will take the most time, so the tracking module requires less time. Tracking methods based on deep learning are difficult to apply to real-time road monitoring systems.

[0008] In response to at least one of the above technical problems, an embodiment of the present application provides a target tracking device and method for road surveillance video, which solves the scale change and / or occlusion problems for real-time road surveillance video.

[0009] According to a first aspect of an embodiment of the present application, a target tracking device for a road monitoring video is provided, comprising:

[0010] A target detection unit performs target detection on the real-time monitoring video of the road area of ​​interest to obtain detection information of one or more targets;

[0011] a tracking matching unit, which matches the detection information of the one or more targets with the tracking information for target tracking;

[0012] a tracking update unit, which updates the tracking information according to the matching result; and

[0013] A road updating unit is configured to update the road information of the area of ​​interest according to the detection information and / or the tracking information.

[0014] According to a second aspect of an embodiment of the present application, a method for tracking a target in a road surveillance video is provided, comprising:

[0015] Perform target detection on real-time surveillance video of the road area of ​​interest to obtain detection information of one or more targets;

[0016] matching the detection information of the one or more targets with tracking information for target tracking;

[0017] Updating the tracking information according to the matching results; and

[0018] The road information of the area of ​​interest is updated according to the detection information and / or the tracking information.

[0019] One of the beneficial effects of the embodiments of the present application is that: the detection information of one or more targets is matched with the tracking information for target tracking, and the tracking information is updated according to the matching result; and the road information of the area of ​​interest is updated according to the detection information and / or the tracking information. Thus, fast target tracking can be achieved, and the scale change and / or occlusion problems can be solved for real-time road monitoring videos.

[0020] With reference to the following description and accompanying drawings, the specific implementation of the embodiment of the present application is disclosed in detail, indicating the way in which the principle of the embodiment of the present application can be adopted. It should be understood that the implementation of the present application is not limited in scope. Within the scope of the spirit and clauses of the attached claims, the implementation of the present application includes many changes, modifications and equivalents.

[0021] Features described and / or illustrated with respect to one embodiment may be used in the same or similar manner in one or more other embodiments, combined with features in other embodiments, or substituted for features in other embodiments.

[0022] It should be emphasized that the term “include / comprises” when used herein refers to the presence of features, integers, steps or components, but does not exclude the presence or addition of one or more other features, integers, steps or components. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The included drawings are used to provide a further understanding of the embodiments of the present application, which constitute a part of the specification, are used to illustrate the implementation methods of the present application, and together with the text description, explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. In the drawings:

[0024] Figure 1 is a schematic diagram of a target tracking method for a road monitoring video according to an embodiment of the present application;

[0025] Figure 2 is an example diagram of updating lane width according to an embodiment of the present application;

[0026] Figure 3 is an example diagram of parameters for tracking matching in an embodiment of the present application;

[0027] Figure 4 It is a schematic diagram of a method for processing certain tracking information that is not matched to detection information according to an embodiment of the present application;

[0028] Figure 5 is an example diagram of the current frame in an embodiment of the present application;

[0029] Figure 6 is an example diagram of an input image for template matching in an embodiment of the present application;

[0030] Figure 7 is an example diagram of a template image for performing template matching in an embodiment of the present application;

[0031] Figure 8 is a schematic diagram of a target tracking device for a road monitoring video according to an embodiment of the present application;

[0032] Fig. 9 It is a schematic diagram of an electronic device according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] With reference to the accompanying drawings, the foregoing and other features of the embodiments of the present application will become apparent through the following description. In the description and the accompanying drawings, specific embodiments of the present application are specifically disclosed, which show some embodiments in which the principles of the embodiments of the present application can be adopted. It should be understood that the present application is not limited to the described embodiments. On the contrary, the embodiments of the present application include all modifications, variations and equivalents that fall within the scope of the attached claims.

[0034] In the embodiments of the present application, the terms "first", "second", etc. are used to distinguish different elements in terms of title, but do not indicate the spatial arrangement or temporal order of these elements, etc., and these elements should not be limited by these terms. The term "and / or" includes any one and all combinations of one or more of the associated listed terms. The terms "comprising", "including", "having", etc. refer to the presence of the stated features, elements, components or components, but do not exclude the presence or addition of one or more other features, elements, components or components.

[0035] In the embodiments of the present application, the singular forms "a", "the", etc. include plural forms and should be broadly understood as "a kind" or "a type" rather than being limited to the meaning of "one"; in addition, the term "said" should be understood to include both singular and plural forms, unless the context clearly indicates otherwise. In addition, the term "according to" should be understood as "at least in part according to...", and the term "based on" should be understood as "at least in part based on...", unless the context clearly indicates otherwise.

[0036] In the embodiments of the present application, traffic events and road monitoring are taken as examples for illustration. A video image including multiple frames can be obtained through a camera. The camera can be a PTZ camera for intelligent transportation video monitoring, which continuously shoots the road monitoring area. The present application is not limited to this.

[0037] Embodiments of the first aspect

[0038] The present invention provides a method for tracking a target in a road monitoring video. Figure 1 is a schematic diagram of a target tracking method for a road monitoring video according to an embodiment of the present application. Figure 1 As shown, the method includes:

[0039] 101, performing target detection on a real-time monitoring video of a road region of interest (ROI) to obtain detection information of one or more targets;

[0040] 102, matching detection information of one or more targets with tracking information for target tracking;

[0041] 103, updating the tracking information according to the matching result; and

[0042] 104 , updating the road information of the area of ​​interest according to the detection information and / or tracking information.

[0043] like Figure 1 As shown, optionally, the method may also include:

[0044] 105, determining whether it is in the initialization stage;

[0045] When in the initialization phase, the road information of the area of ​​interest is updated according to the detection information (104); when not in the initialization phase, the detection information of the one or more targets is matched with the tracking information for target tracking (102).

[0046] It is worth noting that the above Figure 1 The embodiments of the present application are only schematically described, but the present application is not limited thereto. For example, the execution order of the various operations can be appropriately adjusted, and other operations can be added or some operations can be reduced. Those skilled in the art can make appropriate modifications based on the above content, and are not limited to the above appended Figure 1 Records of.

[0047] In some embodiments, target detection and tracking can be performed based on the video sequence of the camera and the configuration file. The configuration file may include the ROI area, and the road information may include lane width and lane direction, which may be represented by a matrix.

[0048] For example, two matrices can be used to save the lane width value and lane direction value (driving angle) for each Y position in each ROI, which are called lane_width_mat and lane_dir_mat respectively. For example, if there are 2 ROIs and the image height is 360 (pixels), the size of these two matrices is 2*360.

[0049] In some embodiments, the method, model or detector of target detection can be selected according to the computing power of the hardware configuration, for example, the target can be detected by machine learning or deep learning model. In the initial stage, the range of detection can be adjusted by the ROI area, and the target detection module will output the detection information (including rectangle, type and attributes) of each detected target to the tracking module. The detection information detected by the detection module is referred to as blob information below.

[0050] For example, you can use a pruned YOLO V3 model as a detector and train it using samples from surveillance cameras. The model can detect seven types of objects, including cars, trucks, buses, vans, motorcycles, bicycles, and people. For more information on how to perform object detection, you can also refer to related technologies.

[0051] In some embodiments, a tracker can be used to store tracking information for target tracking. For example, each vehicle tracker has a buffer in a container, and the purpose of tracking is to update the tracker buffer container and delete the tracker that leaves the ROI area from the container.

[0052] For example, a storage space can be allocated for each newly detected vehicle. A newly detected vehicle means that the vehicle does not match any buffer block. For example, the data structure of the buffer is shown in Table 1, and the detected vehicle rectangle is allocated to "track_rect" in this new buffer. Other information in the buffer will be set to 0 and updated in later steps.

[0053] Table 1

[0054]

[0055] In the embodiments of the present application, road information can be used to solve the scale change and occlusion problems. For example, the lane width value can be used to adjust the distance threshold for tracking; if a fixed threshold is used for tracking and matching, the tracking accuracy will decrease when the vehicle reaches a long-distance area or a close-distance area. The lane width value can also be used to adjust the scale before template matching when the tracker does not match the detected blob.

[0056] In some embodiments, the rectangular width of the detection information is multiplied by a predetermined ratio value to update the road width within the predetermined height range corresponding to the detection information; and / or the vector angle between multiple frames is calculated based on the midpoint of the bottom edge of the detection information to update the road direction within the predetermined height range corresponding to the detection information.

[0057] For example, in the initialization phase, the lane width is updated after the target detection module; after the initialization phase, the lane width is updated after the tracking module. The lane width can refer to the width of the "car" rectangle. The actual lane width is usually 3.5m to 4m, and the car width is 1.6m to 1.8m. Therefore, it can be assumed that the lane width is 1.5 times the car width (predetermined ratio value), where 1.5 can be changed to other values, and the present application is not limited to this.

[0058] Figure 22 is an example diagram of updating lane width in an embodiment of the present application, showing the case where the lane width value is adaptively updated according to the detection information. The road monitoring area includes ROI 1 and ROI 2; the distance between line 201 and line 202 along the lane is the adaptive lane width at each Y position.

[0059] like Figure 2 As shown, for example, when a car Blob 203 is detected, the Y coordinate of its bottom is y, and its width is w in ROI 1, then the lane width value between y-10 and y+10 in the first row of lane_width_mat is updated to 1.5*w. The offset of 10 pixels is to ensure that the update can cover the ROI area from top to bottom after a period of time.

[0060] For another example, the lane direction update is after the tracking module and can be used to remove false matches in the tracking module. If the target has been tracked for a while, the driving angle (0°-360°) is stable. The angle between the two trackers can be calculated with the middle point of the bottom edge as the reference point, and the frame duration is t (e.g. 5).

[0061] For example, for a target, if it is tracked for 5 frames, the reference point of the current i-th frame is (x i ,y i ), the reference point of frame i-5 is (x i-5 ,y i-5 ), the vector angle between these two points will be updated to lane_dir_mat. Similar to the lane width update, the upper and lower areas of the current Y position (for example, between y-10 and y+10) can also be updated.

[0062] In some embodiments, it is determined whether certain detection information and certain tracking information can be matched. Specifically, it can be determined whether certain detection information and certain tracking information can be matched when at least one or any combination of the following conditions is met:

[0063] The target corresponding to the detection information and the target corresponding to the tracking information are located in the same region of interest;

[0064] The distance between the middle point of the bottom edge of the detection information and the middle point of the bottom edge of the tracking information is less than the lane width corresponding to the tracking information multiplied by a first predetermined coefficient;

[0065] The difference between the rectangular area of ​​the detection information and the rectangular area of ​​the tracking information is less than the smaller area of ​​the two multiplied by a second predetermined coefficient;

[0066] The angle between the matching vector direction of the detection information and the lane driving direction corresponding to the tracking information is smaller than a predetermined angle.

[0067] For example, since real-time road videos are monitored, this means that more than 10 frames can be processed in one second, so the moving distance of the object cannot be too long. Each detected blob can be compared with the buffer tracker based on the distance, area and history information of the tracker itself.

[0068] For example, if the detected blob is blob[i], the buffer tracker is tracker[j], the lane width value of tracker[j] is lane_wj, and the driving angle of tracker[j] is anglej, then the matching condition is:

[0069] 1) The corresponding targets are in the same ROI;

[0070] 2) The distance between the middle points of the two bottom edges should be less than 0.5*lane_wj;

[0071] 3) The difference between the area values ​​of the two rectangles should be less than 0.3 times the smaller area;

[0072] 4) The angle between the lane driving direction of tracker [j] and the matching vector should be less than 45°.

[0073] It is worth noting that the above matching methods and conditions are specific examples of the present application, but the present application is not limited thereto.

[0074] Figure 3 is an example diagram of parameters used for tracking matching in an embodiment of the present application, such as Figure 3 As shown, the dotted line under tracker[j] is the historical track, and the lane driving direction under tracker[j] is vector direction 301, which is the direction angle value of the Y coordinate of the bottom midpoint of tracker[j] in lane_dir_mat; the angle between the bottom midpoint of blob[i] and tracker[j] (also called the matching vector direction, i.e. the direction of vector 302) can also be calculated. The angle of condition 4) above is the angle between vector 301 and vector 302, which should be less than 45°.

[0075] In some embodiments, when a certain tracking information corresponds to one or more detection information, the best matching detection information is selected to update the tracking information; when a certain detection information corresponds to one or more tracking information, the detection information is used to update the best matching tracking information; when a certain detection information does not match the tracking information, the detection information is used as a new target.

[0076] For example, after matching each detected blob and tracker, a matrix called match_mat is obtained. If N blobs and M trackers are detected in the current frame, the size of the matching matrix is ​​N*M. The tracking container will be updated according to the following situations:

[0077] For a tracker [j], there are one or more detection clusters matching it, i.e.

[0078]

[0079] The best matching one is selected to update tracker[j], and the other matching flags on the jth column will be set to 0.

[0080] For a detected blob[i], there are one or more matching trackers, i.e.

[0081]

[0082] The best matching tracker is selected, blob[i] is used to update it, and the other matching flags on the i-th row are set to 0.

[0083] If no tracker matches the detected blob[i], that is

[0084]

[0085] The detection blob can then be treated as a new object entering the ROI, which is then initialized and pushed into the tracking container.

[0086] In some embodiments, when certain tracking information does not match the detection information, prediction information is constructed for the tracking information or the tracking information is removed.

[0087] For example, if no blob [j] matching the tracker is detected, i.e.

[0088]

[0089] This situation includes two possibilities: one is that the target leaves the ROI; the other is that the detection module temporarily misses the target. For the second situation, the embodiment of the present application can find the rectangle according to the template matching method, or perform a prediction operation according to historical information.

[0090] Figure 4 FIG. 1 is a schematic diagram of a processing method for a certain tracking information that does not match the detection information according to an embodiment of the present application. Figure 4 As shown, for a certain tracking information that does not match the detection information, the processing method includes:

[0091] 401, determine whether the number of template matching frames (matching-cnt) is less than a first threshold (threshold 1); if yes, execute 402, if not, execute 405.

[0092] For example, the template matching frame number (matching-cnt) may be frame_cnt in Table 1 minus predict_cnt and vehicle_cnt.

[0093] 402, construct an input image using the current frame and construct a template image using the forward frame (eg, the previous frame).

[0094] In some embodiments, a rectangle may be found in the current frame to construct an input image for template matching, wherein a Kalman filter may be applied to predict the displacement vector (dx, dy) between frames.

[0095] For example, the middle point of the bottom edge of the rectangle is used as the reference point, and the displacement of the reference point between frames is (dx, dy). In order to initialize the Kalman filter, there can be 4 state value dimensions, namely dx, dy, dx', dy'. The transition matrix A is defined as follows:

[0096]

[0097] During the update process of the tracking module, the Kalman filter will update the prediction state based on the measurement results. If the prediction function is called with the measurement matrix of the previous frame, it will return the prediction result of the current frame, for example, the position of the predicted reference point is predict_point.

[0098] In some embodiments, a prediction point of the current frame is calculated based on a Kalman filter to obtain a corresponding first lane width; a corresponding second lane width is obtained based on a bottom middle point of the prediction information; a lane width ratio is calculated based on the first lane width and the second lane width; and the input image is obtained from the current frame based on the lane width ratio, and / or the template image is obtained from the forward frame based on the lane width ratio.

[0099] Figure 5 is an example of the current frame of the embodiment of the present application. For example, first determine the range of the input image from the current frame. "predict_point" is the predicted bottom edge middle point, which comes from the prediction of the Kalman filter, such as Figure 5 As shown in 501 in FIG. track_rect is the rectangle of the tracking information in the tracker, such as Figure 5 As shown in 502.

[0100] If the Y coordinate of track_rect is y1, the Y coordinate of predict_point is y2, and the corresponding lane width values ​​are lane_width_mat[y1] and lane_width_mat[y2] respectively, then the ratio of lane widths is lane_width_mat[y2] / lane_width_mat[y1].

[0101] Secondly, the middle point of the bottom edge of "track_rect" can be moved to "predict_point", and the box size of track_rect can be adjusted according to the ratio of the lane width. The new rectangle after the ratio adjustment is called "predict_rect", such as Figure 5 As shown in the box 503 in FIG. 1 ; then the range of this rectangle is enlarged by 0.1 times to reduce the error, forming a matching range (matching range), as shown in FIG. Figure 5 As shown in the range box 504 in FIG.

[0102] Then, the image within the matching range in the current frame can be cropped to obtain an input image for template matching.

[0103] Figure 6 This is an example diagram of an input image for template matching in an embodiment of the present application.

[0104] In some embodiments, a template image for template matching may be constructed based on the previous frame. Since the size of a vehicle may vary greatly between frames, the template image may also be adjusted based on the ratio of lane width.

[0105] Figure 7 is an example diagram of a template image for performing template matching in an embodiment of the present application. Figure 7 As shown, the left image is a small image in the previous frame, and the right image is an image after resizing it, for example, multiplied by the above lane width ratio, and the resized image is used as a template image.

[0106] like Figure 4 As shown, the processing method also includes:

[0107] 403 : Perform template matching on the input image and the template image.

[0108] In some embodiments, if the matching range is W×H and the blob template is w×h, a gray image of each blob can be used as a template. The matching function slides in the background image in a W×H manner, compares w×h overlapping color blocks, and saves the comparison results in mat, the size of which is (W-w+1)×(H-h+1).

[0109] For example, the normalized correlation coefficient method can be used for template matching, as shown in the following formula. I represents the image (buffered background image within the matching range), T is the template (gray image in the blob area), and R is the result matrix. The image and template are normalized before calculating the similarity. This operation can avoid errors caused by changes in lighting. The R value ranges from -1 to 1. 1 means that the image is the same as the template, -1 means the opposite, and 0 means that they have no linear relationship.

[0110]

[0111]

[0112]

[0113] It is worth noting that the above is only a schematic illustration of template matching, and the present application is not limited thereto. For the specific content of template matching, reference may also be made to related technologies.

[0114] like Figure 4 As shown, the processing method also includes:

[0115] 404 , determining whether the matching degree of the template matching is greater than a second threshold (threshold 2), and if so, executing 405 ; if not, executing 406 .

[0116] 405 , updating the tracking information according to the input image serving as the prediction information.

[0117] In some embodiments, after applying template matching to the above template image and input image, a result matrix R and a corresponding matching rectangle can be obtained. The maximum value in R is the final matching degree match_degree (between -1 and 1). The corresponding rectangle of match_degree is the matching rectangle match_rect.

[0118] In some embodiments, a threshold of the matching degree can be set to determine whether it is a valid matching result. For example, the threshold is 0.7; if the matching degree is greater than 0.7, it means that the template matching is valid, and the process will go to 405 of updating the tracking information; otherwise, it means that the template matching fails, and the process will go to 406 of performing the prediction operation.

[0119] In the embodiment of the present application, when the detection information does not match the tracking information and the template matching process also fails, the target still cannot be judged as a departing object because the target may be temporarily blocked by obstacles or other objects. Therefore, the prediction operation can be used to continue tracking for several frames.

[0120] like Figure 4As shown, the processing method also includes:

[0121] 406 , determine whether the predicted frame number (predict-cnt) is less than a third threshold (threshold 3) and whether the total frame number (frame-cnt) is less than a fourth threshold (threshold 4); if yes, execute 407 , if not, execute 411 .

[0122] 407 , construct a prediction image as the prediction information using the current frame.

[0123] For example, the above-mentioned input image may be used as the predicted image, but the present application is not limited thereto.

[0124] like Figure 4 As shown, the processing method also includes:

[0125] 408, determine whether the predicted image is valid; if the predicted image is valid, execute 409; if the predicted image is invalid, execute 410.

[0126] 409: Update the tracking information according to the predicted image.

[0127] 410, remove the tracking information.

[0128] In some embodiments, the prediction operation cannot be performed all the time, and predict_cnt saves the number of consecutive predicted frames. If predict_cnt is above the threshold, the predicted rectangle will not be used, and this tracker will perform a removal operation. After the predicted rectangle is updated to track_rect, predict_cnt will be increased by 1. Alternatively, if track_rect is updated by a detected blob or template matching result, predict_cnt will be set to 0.

[0129] It is worth noting that the above Figure 4 The embodiments of the present application are only schematically described, but the present application is not limited thereto. For example, the execution order of the various operations can be appropriately adjusted, and other operations can be added or some operations can be reduced. Those skilled in the art can make appropriate modifications based on the above content, and are not limited to the above appended Figure 4 Records of.

[0130] In some embodiments, the tracker is updated based on the results of the blob obtained by tracking matching, or the blob obtained by template matching, or the blob obtained by prediction operation. track_rect will be updated with a rectangle of one of the above 3 blobs, frame_cnt will be incremented by 1, and the bottom midpoint of this new rectangle will be used as a metric to update kalman_filter. If the rectangle comes from tracking matching, vehicle_cnt will be incremented by 1; if the rectangle comes from prediction operation, predict_cnt will be incremented by 1.

[0131] In some embodiments, if the subsequent modules need the track information, the tracker structure can also add a container to save the track points. Then it can be used to judge the traffic events. In the traffic analysis system, the tracking results will be used for traffic target counting, congestion detection, reverse detection, lane change detection and illegal parking detection, etc.

[0132] The above only describes the steps or processes related to the present application, but the present application is not limited thereto. The target tracking method may also include other steps or processes, and reference may be made to the prior art for the specific contents of these steps or processes. In addition, the above only uses the above formulas as examples to exemplify the embodiments of the present application, but the present application is not limited to these formulas, and these formulas may be appropriately modified, and the implementation methods of these modifications shall all be included within the scope of the embodiments of the present application.

[0133] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0134] As can be seen from the above embodiments, the detection information of one or more targets is matched with the tracking information for target tracking, and the tracking information is updated according to the matching result; and the road information of the area of ​​interest is updated according to the detection information and / or the tracking information. In this way, fast target tracking can be achieved, and the scale change and / or occlusion problems can be solved for real-time road monitoring videos.

[0135] Embodiments of the second aspect

[0136] An embodiment of the present application provides a target tracking device for a road surveillance video, and the contents that are the same as those in the embodiment of the first aspect are not repeated here.

[0137] Figure 8 is a schematic diagram of a target tracking device for a road monitoring video according to an embodiment of the present application. Figure 8 As shown, the target tracking device 800 of the road monitoring video includes:

[0138] The target detection unit 801 performs target detection on the real-time monitoring video of the road area of ​​interest to obtain detection information of one or more targets;

[0139] A tracking matching unit 802 matches the detection information of the one or more targets with the tracking information for target tracking;

[0140] A tracking update unit 803, which updates the tracking information according to the matching result; and

[0141] The road updating unit 804 updates the road information of the area of ​​interest according to the detection information and / or the tracking information.

[0142] In some embodiments, Figure 8 As shown, the target tracking device 800 of the road monitoring video may also include:

[0143] An initialization determination unit 805, which determines whether it is in the initialization stage;

[0144] When in the initialization stage, the road updating unit 804 updates the road information of the area of ​​interest according to the detection information; when not in the initialization stage, the tracking matching unit 802 matches the detection information of the one or more targets with the tracking information used for target tracking.

[0145] In some embodiments, the road updating unit 804 is used to:

[0146] Multiplying the rectangular width of the detection information by a predetermined ratio value to update the road width within a predetermined height range corresponding to the detection information; and / or

[0147] The vector angle between multiple frames is calculated according to the midpoint of the bottom edge of the detection information, and the road direction within the predetermined height range corresponding to the detection information is updated.

[0148] In some embodiments, the tracking matching unit 802 is used to determine that a certain detection information and a certain tracking information can be matched when at least one or any combination of the following conditions is met:

[0149] The target corresponding to the detection information and the target corresponding to the tracking information are located in the same region of interest;

[0150] The distance between the middle point of the bottom edge of the detection information and the middle point of the bottom edge of the tracking information is less than the lane width corresponding to the tracking information multiplied by a first predetermined coefficient;

[0151] The difference between the rectangular area of ​​the detection information and the rectangular area of ​​the tracking information is less than the smaller area of ​​the two multiplied by a second predetermined coefficient;

[0152] The angle between the matching vector direction of the detection information and the lane driving direction corresponding to the tracking information is smaller than a predetermined angle.

[0153] In some embodiments, the tracking update unit 803 is used to: when a certain tracking information corresponds to one or more detection information, select the best matching detection information to update the tracking information; when a certain detection information corresponds to one or more tracking information, use the detection information to update the best matching tracking information; and

[0154] The tracking update unit 803 is also used to: when a certain detection information does not match the tracking information, use the detection information as a new target; when a certain tracking information does not match the detection information, construct prediction information for the tracking information or remove the tracking information.

[0155] In some embodiments, the tracking update unit 803 is used to:

[0156] When the number of template matching frames is less than a first threshold, constructing an input image using the current frame and constructing a template image using the forward frame;

[0157] performing template matching on the input image and the template image; and

[0158] When the degree of template matching is greater than a second threshold, the tracking information is updated according to the input image serving as the prediction information.

[0159] In some embodiments, the tracking update unit 803 is further configured to:

[0160] Calculate the prediction point of the current frame according to the Kalman filter to obtain the corresponding first lane width;

[0161] Obtaining a corresponding second lane width according to the middle point of the bottom edge of the prediction information;

[0162] calculating a ratio of lane widths based on the first lane width and the second lane width; and

[0163] The input image is obtained from the current frame according to the ratio of the lane width, and / or the template image is obtained from the forward frame according to the ratio of the lane width.

[0164] In some embodiments, the tracking update unit 803 is further configured to:

[0165] When the number of template matching frames is greater than or equal to the first threshold, or the matching degree of the template matching is less than or equal to the second threshold, determining whether the number of predicted frames is less than a third threshold and whether the total number of frames is less than a fourth threshold;

[0166] In a case where the number of predicted frames is less than the third threshold and the total number of frames is less than the fourth threshold, constructing a predicted image as the prediction information using the current frame; and

[0167] When the predicted image is valid, the tracking information is updated according to the predicted image.

[0168] In some embodiments, the tracking update unit 803 is further configured to:

[0169] When the number of predicted frames is greater than or equal to the third threshold, or the total number of frames is greater than or equal to the fourth threshold, or the predicted image is invalid, the tracking information is removed.

[0170] To keep it simple, Figure 8 The connection relationship or signal direction between various components or modules is only exemplified, but it should be clear to those skilled in the art that various related technologies such as bus connection can be used. The above-mentioned components or modules can be implemented by hardware facilities such as processors and memories; the embodiments of the present application are not limited to this.

[0171] The above embodiments are merely exemplary of the embodiments of the present application, but the present application is not limited thereto, and appropriate modifications may be made based on the above embodiments. For example, the above embodiments may be used alone, or one or more of the above embodiments may be combined.

[0172] As can be seen from the above embodiments, the detection information of one or more targets is matched with the tracking information for target tracking, and the tracking information is updated according to the matching result; and the road information of the area of ​​interest is updated according to the detection information and / or the tracking information. In this way, fast target tracking can be achieved, and the scale change and / or occlusion problems can be solved for real-time road monitoring videos.

[0173] Embodiments of the third aspect

[0174] The embodiment of the present application provides an electronic device, including the target tracking device for road monitoring video as described in the embodiment of the second aspect, the contents of which are incorporated herein. The electronic device can be, for example, a computer, a server, a workstation, a laptop computer, a smart phone, etc., but the embodiment of the present application is not limited thereto.

[0175] Fig. 9 Schematic diagram of an electronic device according to an embodiment of the present application. Fig. 9 As shown, the electronic device 900 may include: a processor (e.g., a central processing unit CPU) 910 and a memory 920; the memory 920 is coupled to the central processing unit 910. The memory 920 may store various data; in addition, it may store a program 921 for information processing, and execute the program 921 under the control of the processor 910.

[0176] In some embodiments, the functions of the target tracking device 800 for road surveillance video are integrated into the processor 910. The processor 910 is configured to implement the target tracking method for road surveillance video as described in the embodiment of the first aspect.

[0177] In some embodiments, the target tracking device 800 for road surveillance video is configured separately from the processor 910. For example, the target tracking device 800 for road surveillance video can be configured as a chip connected to the processor 910, and the functions of the target tracking device 800 for road surveillance video are realized through the control of the processor 910.

[0178] In some embodiments, the processor 910 is configured to perform the following control: perform target detection on real-time surveillance video of a road area of ​​interest to obtain detection information of one or more targets; match the detection information of the one or more targets with tracking information for target tracking; update the tracking information based on the matching results; and update the road information of the area of ​​interest based on the detection information and / or the tracking information.

[0179] In some embodiments, the processor 910 is also configured to perform the following control: determine whether it is in the initialization stage; if it is in the initialization stage, update the road information of the area of ​​interest according to the detection information; if it is not in the initialization stage, match the detection information of the one or more targets with the tracking information for target tracking.

[0180] In some embodiments, the processor 910 is also configured to perform the following control: multiplying the rectangular width of the detection information by a predetermined ratio value, and updating the road width within the predetermined height range corresponding to the detection information; and / or calculating the vector angle between multiple frames based on the midpoint of the bottom edge of the detection information, and updating the road direction within the predetermined height range corresponding to the detection information.

[0181] In some embodiments, the processor 910 is further configured to perform the following control: determining that certain detection information and certain tracking information can be matched when at least one or any combination of the following conditions is met:

[0182] The target corresponding to the detection information and the target corresponding to the tracking information are located in the same region of interest;

[0183] The distance between the middle point of the bottom edge of the detection information and the middle point of the bottom edge of the tracking information is less than the lane width corresponding to the tracking information multiplied by a first predetermined coefficient;

[0184] The difference between the rectangular area of ​​the detection information and the rectangular area of ​​the tracking information is less than the smaller area of ​​the two multiplied by a second predetermined coefficient;

[0185] The angle between the matching vector direction of the detection information and the lane driving direction corresponding to the tracking information is smaller than a predetermined angle.

[0186] In some embodiments, the processor 910 is further configured to perform the following control: when a certain tracking information corresponds to one or more detection information, select the best matching detection information to update the tracking information; when a certain detection information corresponds to one or more tracking information, use the detection information to update the best matching tracking information; and

[0187] When a certain detection information does not match the tracking information, the detection information is used as a new target; when a certain tracking information does not match the detection information, prediction information is constructed for the tracking information or the tracking information is removed.

[0188] In some embodiments, the processor 910 is also configured to perform the following control: when the number of template matching frames is less than a first threshold, use the current frame to construct an input image and use the forward frame to construct a template image; perform template matching on the input image and the template image; and when the matching degree of the template matching is greater than a second threshold, update the tracking information based on the input image as the prediction information.

[0189] In some embodiments, the processor 910 is also configured to perform the following control: calculate the prediction point of the current frame according to the Kalman filter to obtain the corresponding first lane width; obtain the corresponding second lane width according to the middle point of the bottom edge of the prediction information; calculate the lane width ratio according to the first lane width and the second lane width; and obtain the input image from the current frame according to the lane width ratio, and / or obtain the template image from the forward frame according to the lane width ratio.

[0190] In some embodiments, the processor 910 is also configured to perform the following control: when the number of template matching frames is greater than or equal to the first threshold, or the matching degree of the template matching is less than or equal to the second threshold, determine whether the number of predicted frames is less than a third threshold and whether the total number of frames is less than a fourth threshold; when the number of predicted frames is less than the third threshold and the total number of frames is less than the fourth threshold, use the current frame to construct a predicted image as the prediction information; and when the predicted image is valid, update the tracking information according to the predicted image.

[0191] In some embodiments, the processor 910 is further configured to perform the following control: when the predicted frame number is greater than or equal to the third threshold, or the total frame number is greater than or equal to the fourth threshold, or the predicted image is invalid, the tracking information is removed.

[0192] In addition, if Fig. 9 As shown, the electronic device 900 may also include: an input / output (I / O) device 930 and a display 940, etc.; wherein the functions of the above components are similar to those of the prior art and are not described in detail here. It is worth noting that the electronic device 900 does not necessarily have to include Fig. 9 In addition, the electronic device 900 may also include Fig. 9 For components not shown in the figure, reference may be made to the related art.

[0193] An embodiment of the present application also provides a computer-readable program, wherein when the program is executed in an electronic device, the program enables a computer to execute the target tracking method for road surveillance video as described in the embodiment of the first aspect in the electronic device.

[0194] An embodiment of the present application also provides a storage medium storing a computer-readable program, wherein the computer-readable program enables a computer to execute the target tracking method for road surveillance video as described in the embodiment of the first aspect in an electronic device.

[0195] The above devices and methods of the present application can be implemented by hardware, or by hardware combined with software. The present application relates to such a computer-readable program, which, when executed by a logic component, enables the logic component to implement the above-mentioned devices or components, or enables the logic component to implement the various methods or steps described above. The present application also relates to a storage medium for storing the above program, such as a hard disk, a magnetic disk, an optical disk, a DVD, a flash memory, etc.

[0196] The method / device described in conjunction with the embodiments of the present application may be directly embodied as hardware, a software module executed by a processor, or a combination of the two. For example, one or more of the functional block diagrams shown in the figure and / or one or more combinations of the functional block diagrams may correspond to various software modules of the computer program flow or to various hardware modules. These software modules may correspond to the various steps shown in the figure, respectively. These hardware modules may be implemented by solidifying these software modules, for example, using a field programmable gate array (FPGA).

[0197] The software module may be located in a RAM memory, a flash memory, a ROM memory, an EPROM memory, an EEPROM memory, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. A storage medium may be coupled to a processor so that the processor can read information from the storage medium and write information to the storage medium; or the storage medium may be an integral part of the processor. The processor and the storage medium may be located in an ASIC. The software module may be stored in a memory of a mobile terminal or in a memory card that can be inserted into the mobile terminal. For example, if a device (such as a mobile terminal) uses a large-capacity MEGA-SIM card or a large-capacity flash memory device, the software module may be stored in the MEGA-SIM card or the large-capacity flash memory device.

[0198] For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can be implemented as a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic device, discrete hardware component or any appropriate combination thereof for performing the functions described in the present application. For one or more of the functional blocks described in the drawings and / or one or more combinations of functional blocks, it can also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in communication with a DSP, or any other such configuration.

[0199] The present application is described above in conjunction with specific implementation methods, but it should be clear to those skilled in the art that these descriptions are exemplary and are not intended to limit the scope of protection of the present application. Those skilled in the art can make various modifications and variations to the present application based on the principles of the present application, and these modifications and variations are also within the scope of the present application.

[0200] Regarding the implementation methods including the above embodiments, the following additional notes are also disclosed:

[0201] Note 1: A target tracking method for road monitoring video, comprising:

[0202] Perform target detection on real-time surveillance video of the road area of ​​interest to obtain detection information of one or more targets;

[0203] matching the detection information of the one or more targets with tracking information for target tracking;

[0204] updating the tracking information according to the matching result; and

[0205] The road information of the area of ​​interest is updated according to the detection information and / or the tracking information.

[0206] Note 2: The method according to Note 1, wherein the method further comprises:

[0207] Determine whether it is in the initialization phase;

[0208] When in the initialization stage, the road information of the area of ​​interest is updated according to the detection information; when not in the initialization stage, the detection information of the one or more targets is matched with the tracking information for target tracking.

[0209] Note 3. The method according to Note 1 or 2, wherein:

[0210] Multiplying the rectangular width of the detection information by a predetermined ratio value to update the road width within a predetermined height range corresponding to the detection information; and / or

[0211] The vector angle between multiple frames is calculated according to the midpoint of the bottom edge of the detection information, and the road direction within the predetermined height range corresponding to the detection information is updated.

[0212] Note 4. The method according to any one of Notes 1 to 3, wherein it is determined that a certain detection information and a certain tracking information can be matched when at least one or any combination of the following conditions is met:

[0213] The target corresponding to the detection information and the target corresponding to the tracking information are located in the same region of interest;

[0214] The distance between the middle point of the bottom edge of the detection information and the middle point of the bottom edge of the tracking information is less than the lane width corresponding to the tracking information multiplied by a first predetermined coefficient;

[0215] The difference between the rectangular area of ​​the detection information and the rectangular area of ​​the tracking information is less than the smaller area of ​​the two multiplied by a second predetermined coefficient;

[0216] The angle between the detection information matching vector direction and the lane driving direction corresponding to the tracking information is smaller than a predetermined angle.

[0217] Note 5. The method according to any one of Notes 1 to 4, wherein:

[0218] In the case where a certain tracking information corresponds to one or more detection information, selecting the most matching detection information to update the tracking information; in the case where a certain detection information corresponds to one or more tracking information, using the detection information to update the most matching tracking information; and

[0219] When a certain detection information does not match the tracking information, the detection information is used as a new target; when a certain tracking information does not match the detection information, prediction information is constructed for the tracking information or the tracking information is removed.

[0220] Note 6: The method according to Note 5, wherein the method further comprises:

[0221] When the number of template matching frames is less than a first threshold, constructing an input image using the current frame and constructing a template image using the forward frame;

[0222] performing template matching on the input image and the template image; and

[0223] When the degree of template matching is greater than a second threshold, the tracking information is updated according to the input image serving as the prediction information.

[0224] Note 7: The method according to Note 6, wherein the method further comprises:

[0225] Calculate the prediction point of the current frame according to the Kalman filter to obtain the corresponding first lane width;

[0226] Obtaining a corresponding second lane width according to the middle point of the bottom edge of the prediction information;

[0227] calculating a ratio of lane widths based on the first lane width and the second lane width; and

[0228] The input image is obtained from the current frame according to the ratio of the lane width, and / or the template image is obtained from the forward frame according to the ratio of the lane width.

[0229] Note 8: The method according to Note 6 or 7, wherein the method further comprises:

[0230] When the number of template matching frames is greater than or equal to the first threshold, or the matching degree of the template matching is less than or equal to the second threshold, determining whether the number of predicted frames is less than a third threshold and whether the total number of frames is less than a fourth threshold;

[0231] In a case where the number of predicted frames is less than the third threshold and the total number of frames is less than the fourth threshold, constructing a predicted image as the prediction information using the current frame; and

[0232] When the predicted image is valid, the tracking information is updated according to the predicted image.

[0233] Note 9. The method according to Note 8, wherein the method further comprises:

[0234] When the number of predicted frames is greater than or equal to the third threshold, or the total number of frames is greater than or equal to the fourth threshold, or the predicted image is invalid, the tracking information is removed.

[0235] Note 10: A target tracking device for road monitoring video, comprising:

[0236] A target detection unit performs target detection on the real-time monitoring video of the road area of ​​interest to obtain detection information of one or more targets;

[0237] a tracking matching unit, which matches the detection information of the one or more targets with the tracking information for target tracking;

[0238] a tracking update unit, which updates the tracking information according to the matching result; and

[0239] A road updating unit is configured to update the road information of the area of ​​interest according to the detection information and / or the tracking information.

[0240] Note 11. The device according to Note 10, wherein the device further comprises:

[0241] an initialization determination unit, which determines whether it is in the initialization stage;

[0242] When in the initialization stage, the road updating unit updates the road information of the area of ​​interest according to the detection information; when not in the initialization stage, the tracking matching unit matches the detection information of the one or more targets with the tracking information used for target tracking.

[0243] Note 12. The device according to Note 10 or 11, wherein the road updating unit is used for:

[0244] Multiplying the rectangular width of the detection information by a predetermined ratio value to update the road width within a predetermined height range corresponding to the detection information; and / or

[0245] The vector angle between multiple frames is calculated according to the midpoint of the bottom edge of the detection information, and the road direction within the predetermined height range corresponding to the detection information is updated.

[0246] Note 13. The device according to any one of Notes 10 to 12, wherein the tracking and matching unit is used to determine that a certain detection information and a certain tracking information can be matched when at least one or any combination of the following conditions is met:

[0247] The target corresponding to the detection information and the target corresponding to the tracking information are located in the same region of interest;

[0248] The distance between the middle point of the bottom edge of the detection information and the middle point of the bottom edge of the tracking information is less than the lane width corresponding to the tracking information multiplied by a first predetermined coefficient;

[0249] The difference between the rectangular area of ​​the detection information and the rectangular area of ​​the tracking information is less than the smaller area of ​​the two multiplied by a second predetermined coefficient;

[0250] The angle between the matching vector direction of the detection information and the lane driving direction corresponding to the tracking information is smaller than a predetermined angle.

[0251] Note 14. The device according to any one of Notes 10 to 13, wherein:

[0252] The tracking update unit is used to: when a certain tracking information corresponds to one or more detection information, select the most matching detection information to update the tracking information; when a certain detection information corresponds to one or more tracking information, use the detection information to update the most matching tracking information; and

[0253] The tracking update unit is also used to: when a certain detection information does not match the tracking information, use the detection information as a new target; when a certain tracking information does not match the detection information, construct prediction information for the tracking information or remove the tracking information.

[0254] Note 15. The device according to Note 14, wherein the tracking and updating unit is used for:

[0255] When the number of template matching frames is less than a first threshold, constructing an input image using the current frame and constructing a template image using the forward frame;

[0256] performing template matching on the input image and the template image; and

[0257] When the degree of template matching is greater than a second threshold, the tracking information is updated according to the input image serving as the prediction information.

[0258] Note 16. The device according to Note 15, wherein the tracking and updating unit is further used for:

[0259] Calculate the prediction point of the current frame according to the Kalman filter to obtain the corresponding first lane width;

[0260] Obtaining a corresponding second lane width according to the middle point of the bottom edge of the prediction information;

[0261] calculating a ratio of lane widths based on the first lane width and the second lane width; and

[0262] The input image is obtained from the current frame according to the ratio of the lane width, and / or the template image is obtained from the forward frame according to the ratio of the lane width.

[0263] Note 17. The device according to Note 15 or 16, wherein the tracking and updating unit is further used for:

[0264] When the number of template matching frames is greater than or equal to the first threshold, or the matching degree of the template matching is less than or equal to the second threshold, determining whether the number of predicted frames is less than a third threshold and whether the total number of frames is less than a fourth threshold;

[0265] In a case where the number of predicted frames is less than the third threshold and the total number of frames is less than the fourth threshold, constructing a predicted image as the prediction information using the current frame; and

[0266] When the predicted image is valid, the tracking information is updated according to the predicted image.

[0267] Note 18. The device according to Note 17, wherein the tracking and updating unit is further used for:

[0268] When the number of predicted frames is greater than or equal to the third threshold, or the total number of frames is greater than or equal to the fourth threshold, or the predicted image is invalid, the tracking information is removed.

[0269] Note 19. An electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to implement the target tracking method for road surveillance video as described in any one of Notes 1 to 9.

Claims

1. A target tracking device for road monitoring video, characterized in that: The device comprises: A target detection unit performs target detection on the real-time monitoring video of the road area of ​​interest to obtain detection information of one or more targets; an initialization determination unit, which determines whether it is in the initialization stage; a tracking matching unit, which matches the detection information of the one or more targets with the tracking information for target tracking when not in the initialization phase; a tracking update unit, which updates the tracking information according to the matching result; and A road updating unit, which, when in the initialization stage, updates the road information of the area of ​​interest according to the detection information and / or the tracking information, wherein the road information includes road width and lane direction.

2. The device according to claim 1, wherein: The road updating unit is used for: Multiplying the rectangular width of the detection information by a predetermined ratio value to update the road width within a predetermined height range corresponding to the detection information; and / or The vector angle between multiple frames is calculated according to the midpoint of the bottom edge of the detection information, and the road direction within the predetermined height range corresponding to the detection information is updated.

3. The device according to claim 1, wherein: The tracking and matching unit is used to determine that a certain detection information and a certain tracking information can be matched when at least one or any combination of the following conditions is met: The target corresponding to the detection information and the target corresponding to the tracking information are located in the same region of interest; The distance between the middle point of the bottom edge of the detection information and the middle point of the bottom edge of the tracking information is less than the lane width corresponding to the tracking information multiplied by a first predetermined coefficient; The difference between the rectangular area of ​​the detection information and the rectangular area of ​​the tracking information is less than the smaller area of ​​the two multiplied by a second predetermined coefficient; The angle between the matching vector direction of the detection information and the lane driving direction corresponding to the tracking information is smaller than a predetermined angle.

4. The device according to claim 1, wherein: The tracking update unit is used to: when a certain tracking information corresponds to one or more detection information, select the most matching detection information to update the tracking information; In the case where a certain detection information corresponds to one or more tracking information, the detection information is used to update the best matching tracking information; as well as The tracking update unit is further used to: when a certain detection information does not match the tracking information, use the detection information as a new target; When certain tracking information does not match the detection information, prediction information is constructed for the tracking information or the tracking information is removed.

5. The device according to claim 4, wherein: The tracking update unit is used for: When the number of template matching frames is less than a first threshold, constructing an input image using the current frame and constructing a template image using the forward frame; Performing template matching on the input image and the template image; as well as When the degree of template matching is greater than a second threshold, the tracking information is updated according to the input image serving as the prediction information.

6. The device according to claim 5, wherein: The tracking update unit is also used for: Calculate the prediction point of the current frame according to the Kalman filter to obtain the corresponding first lane width; Obtaining a corresponding second lane width according to the middle point of the bottom edge of the prediction information; calculating a ratio of lane widths based on the first lane width and the second lane width; as well as The input image is obtained from the current frame according to the ratio of the lane width, and / or the template image is obtained from the forward frame according to the ratio of the lane width.

7. The device according to claim 5, wherein: The tracking update unit is also used for: When the number of template matching frames is greater than or equal to the first threshold, or the matching degree of the template matching is less than or equal to the second threshold, determining whether the number of predicted frames is less than a third threshold and whether the total number of frames is less than a fourth threshold; When the number of predicted frames is less than the third threshold and the total number of frames is less than the fourth threshold, constructing a predicted image as the prediction information using the current frame; as well as When the predicted image is valid, the tracking information is updated according to the predicted image.

8. The device according to claim 7, wherein: The tracking update unit is also used for: When the number of predicted frames is greater than or equal to the third threshold, or the total number of frames is greater than or equal to the fourth threshold, or the predicted image is invalid, the tracking information is removed.

9. A target tracking method for road monitoring video, characterized in that: The method comprises: Perform target detection on real-time surveillance video of the road area of ​​interest to obtain detection information of one or more targets; Determine whether it is in the initialization phase; In a case where the initialization phase is not in progress, the detection information of the one or more targets is matched with tracking information for target tracking; Updating the tracking information according to the matching results; and In the case of the initialization stage, the road information of the region of interest is updated according to the detection information and / or the tracking information, wherein the road information includes road width and lane direction.

Images