Camera PTZ automatic tracking method and system based on target detection

By using the object detection technology combined with PTZ camera and Yolo model at the work site, the precise tracking and judgment of suspected violation targets is achieved, and the drift and loss of visual target tracking in complex environments is solved, and the tracking accuracy and real-time performance are improved. It makes up for the shortcomings of fixed cameras such as blind spots in monitoring and large computing volume.

CN118247481BActive Publication Date: 2024-12-06STATE GRID GANSU ELECTRIC POWER RESEARCH INSTITUTE
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410174035.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-02-07
Publication Date
2024-12-06
Estimated Expiration
2044-02-07

AI Technical Summary

Technical Problem

The prior art has problems of drift and loss in visual target tracking at the work site. Especially in complex environments, it is difficult for fixed cameras to achieve high-precision and real-time tracking, and there are problems such as blind spots in monitoring and large amounts of computing.

Method used

Using the PTZ automatic tracking method based on object detection, the target detection model is trained using Yolo model to conduct real-time object detection and identification of the work site. Through the automatic tracking and zoom function of the PTZ camera, combined with deflection and deviation correction algorithm, the suspected violation of targets is achieved. Accurate tracking and judgment.

Benefits of technology

It improves the accuracy and real-time performance of visual target tracking, reduces the amount of computing, enhances robustness, avoids the monitoring blind spots and target loss problems of fixed cameras, and realizes all-round automatic tracking and efficient violation detection.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118247481B_ABST
    Figure CN118247481B_ABST
Patent Text Reader

Abstract

The present invention belongs to the technical field of visual target tracking, and discloses a camera PTZ automatic tracking method and system based on target detection. The method includes: collecting historical video data of the work site as a data sample set; obtaining a target detection model based on Yolo model training; based on a PTZ camera, using the target detection model to perform real-time target detection and obtain target confidence, and dividing the target into noise targets, suspected illegal targets and normal targets; ignoring noise targets, making illegal judgments and early warning broadcasts for normal targets, performing PTZ automatic tracking and identification and zooming for suspected illegal targets, and synchronously transmitting the tracking result data of the suspected illegal targets to the target detection model for identification and judgment, until it is determined whether the suspected illegal target is illegal, and the tracking is terminated. The present invention can meet the requirements of visual target tracking accuracy and complex work site application requirements, and the method is simple and efficient, with good real-time and robustness.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of visual target tracking, and in particular relates to a camera PTZ automatic tracking method and system based on target detection. Background Art

[0002] The power grid is difficult to manage as a whole due to its complex environment, vast territory, numerous operation points, strong uncertainty in time and location, complex equipment types, complex operators and frequent operations. The traditional form of manual inspections by operation supervisors at the operation site cannot timely perceive safety issues at the operation site. There are few on-site supervisors and single management methods, and it is impossible to promptly discover, stop and correct violations at the operation site. It is urgent to improve the level of safety management and control through digital technology.

[0003] The existing anti-violation intelligent diagnosis device at the work site uses the PTZ camera to conduct autonomous inspections and visual target tracking on the work site, and uses the TTS delay-free sound and light alarm module to promptly stop the violation of regulations by the personnel at the work site and prevent major accidents. Visual target tracking is a very important key technology in the identification of violation behaviors. Its task is to predict the size and position of the target object in the subsequent frames given the size and position of the target object in the initial frame of a certain video sequence, so as to obtain the motion trajectory of the target in the entire video sequence; however, in the actual tracking and prediction process, due to the changes in the appearance of the target object, such as changes in perspective, posture, scale, etc., as well as the external noise brought by the environment in which the target object is located, such as: occlusion noise, lighting effects, background noise, etc., the tracking process is very prone to drift and loss.

[0004] Therefore, it is necessary to develop a visual target tracking method that can meet the accuracy requirements of visual target tracking and the application requirements of complex work sites.

[0005] Chinese patent CN116703985A discloses a cross-lens multi-target tracking method, device and medium based on skeleton similarity, the method comprises: reading multiple camera data streams, storing two of the videos in a storage queue at the same time, extracting the streams from the storage queue frame by frame, and obtaining the first picture and the second picture taken by the two cameras at the same time; performing head positioning on the first picture and the second picture to obtain appearance features and center coordinates, and performing skeleton extraction to obtain the skeleton posture of the joint points, using the appearance features and center coordinates as the input of the target tracking model, and performing ID matching of the target in the same area; when it is detected that the first picture enters the second picture, the first picture and the second picture are respectively located; the first picture and the second picture are respectively located; the second ... After the overlapping areas of the screen are found, the skeleton posture similarity and the center coordinate Euclidean distance are calculated, and a linear weighted combination of the two is used to achieve cross-region target ID matching when switching areas; this patent improves tracking accuracy and robustness, but the monitoring system of this patent uses a fixed camera, which is fixed and unchanged. The moving trajectory of the target is tracked by adding a frame to the target. There are monitoring blind spots, and the tracked target is easily blocked and lost by obstacles. When it deals with the real-time tracking problem of complex cross-border scenes in large-scale monitoring systems, there are problems such as huge amount of calculations generated in the target tracking process, poor real-time performance, and unstable tracking. Summary of the invention

[0006] The present invention aims to solve the technical problems existing in the background technology and provide a camera PTZ automatic tracking method and system based on target detection, which can meet the accuracy requirements of visual target tracking and the application requirements of complex work sites, and has the characteristics of simple method, high efficiency, small amount of calculation, good real-time and robust tracking, etc.

[0007] In order to achieve the above technical objectives, the present invention adopts the following technical solutions:

[0008] A camera PTZ automatic tracking method based on target detection, the method comprising the following steps:

[0009] Step S1: Collect and obtain historical video data of the work site as a data sample set;

[0010] Step S2: Using the data sample set, obtain the target detection model based on Yolo model training;

[0011] Step S3: Based on the PTZ camera, use the target detection model to perform real-time target detection on the work site and obtain the target confidence; according to the target confidence, the target is divided into noise targets, suspected violation targets and normal targets;

[0012] Step S4: The target detection model ignores noise targets, makes violation judgments and issues warnings for normal targets, performs PTZ automatic tracking, identification and zooming on suspected illegal targets, and synchronously transmits the tracking result data of suspected illegal targets to the target detection model for identification and judgment, until it is determined whether the suspected illegal target is illegal, and the tracking is terminated.

[0013] Furthermore, in the process of obtaining the target detection model based on Yolo model training, when the number of training samples of a certain target is insufficient, the number of training samples of the certain target is increased by data enhancement.

[0014] Furthermore, the data enhancement method includes: scaling, translating, rotating, randomly adjusting hue / saturation / brightness, fusing, adding noise, cropping or a combination of operations on the image of the training sample, and saving the operated image as a newly added training sample.

[0015] Furthermore, in step S4, the suspected illegal target is automatically tracked and identified by PTZ and zoomed, and the tracking result data of the suspected illegal target is synchronously transmitted to the target detection model for identification and judgment, until it is determined whether the suspected illegal target is illegal, and the tracking is terminated, which specifically includes:

[0016] Step S41: locking the suspected illegal target through the built-in PTZ automatic tracking module of the PTZ camera, and automatically controlling the PTZ pan-tilt deflection to track the target;

[0017] Step S42: after the deflection is completed, target correction is performed to place the identified suspected target at the center of the lens of the PTZ camera;

[0018] Step S43: After the target deviation correction is completed, the zoom factor is calculated according to the size of the suspected target in the image, the PTZ camera captures real-time video and uploads it to the target detection model, and steps S3-S4 are repeated until it is confirmed whether the suspected illegal target is in violation and an early warning is broadcast.

[0019] Furthermore, the step S41 specifically includes:

[0020] Step S411: Calculate the current relative offset of the center point of the target frame at the current moment relative to the center point of the current lens image according to the lens image size of the PTZ camera and the target frame size of the current suspected violation target obtained according to the target detection model;

[0021] Step S412: Calculate the next moment relative offset of the center point of the target frame detected at the next moment relative to the center point of the lens image, and calculate the similarity S between the next moment relative offset and the current relative offset according to the cosine similarity formula. AB ;

[0022]

[0023] Among them, A is the current relative offset vector; B is the relative offset vector at the next moment; Cosθ represents the cosine value of the angle between the relative offset vectors of the previous and next times; ||A|| and ||B|| represent the magnitudes of vectors A and B;

[0024] Step S413: Determine whether to control the PTZ pan / tilt to deflect according to the similarity threshold S0:

[0025] If the similarity S between the relative offsets of the previous and next times AB ≥ S0, then discard the relative offset at the next moment, do not deflect, and transfer to step S412 to recalculate the updated relative offset and similarity at the next moment;

[0026] If the similarity S between the relative offsets of the previous and next times AB < S0, then control the PTZ pan / tilt to deflect according to the relative offset at the next moment.

[0027] Furthermore, the specific steps of step S411 include:

[0028] First, obtain the coordinate values (x1, y1), (x2, y2) of the two diagonal points of the target box in the current camera frame through the target detection model, where (x1, y1) is the coordinate of the upper left corner and (x2, y2) is the coordinate of the lower right corner;

[0029] Then, encode and calculate the width x w and height y h of the target box, x w = x2 - x1, y h, = y2 - y1;

[0030] Next, encode and calculate the center point coordinates (real_x, real_y) of the target box, where real_x = x1 + x w / 2; real_y = y1 + y h, / 2;

[0031] Finally, encode and calculate the normalized offset (x_offset, y_offset) of the center point coordinates (real_x, real_y) of the target box relative to the center point coordinates (targe.shape[x] / 2, targe.shape[y] / 2) of the current image, and the calculation formula is:

[0032]

[0033]

[0034] In the above formula, x_offset is the normalized offset of the center point of the target frame relative to the center point of the current lens image in the x direction, y_offset is the normalized offset of the center point of the target frame relative to the center point of the current lens image in the y direction, real_x is the x-axis coordinate of the center point of the target frame, real_y is the y-axis coordinate of the center point of the target frame, target.shape[x] is the width of the current lens image along the x-axis, target.shape[y] is the length of the current lens image along the y-axis, target.shape[x] / 2 is the x-axis coordinate of the center point of the current lens image, and target.shape[y] / 2 is the y-axis coordinate of the center point of the current lens image.

[0035] Furthermore, the step S42 specifically includes:

[0036] After the deflection is completed, the relative offset and relative offset distance between the center of the deflected target frame and the center of the lens image are calculated, and the relative offset distance is compared with the set distance threshold to determine whether to perform target correction:

[0037] If the relative offset distance is less than the set distance threshold, no target correction is required;

[0038] Otherwise, target correction is performed according to the relative offset between the center of the deflected target frame and the center of the lens image, and the PTZ pan / tilt is controlled to deflect so that the suspected illegal target is placed in the center of the lens of the PTZ camera.

[0039] Furthermore, in step S42, the target correction formula f is as follows:

[0040]

[0041] In the above formula, dx is the distance between the center of the target frame and the center of the lens image in the x-axis direction after deflection, dy is the distance between the center of the target frame and the center of the lens image in the y-axis direction after deflection, uncertainty[x] is the distance threshold in the x-axis direction, and uncertainty[y] is the distance threshold in the y-axis direction;

[0042] If the distance between the center of the deflected target frame and the center of the lens image in the x direction and the y direction is less than the distance thresholds uncertainty[x] and uncertainty[y] in the corresponding directions, no target correction is required. Otherwise, target correction is required based on the relative offset between the center of the deflected target frame and the center of the lens image, and the suspected illegal target is placed at the center of the PTZ camera's lens.

[0043] Furthermore, in step S2, the Yolo model adopts Yolov5 model, Yolov7 model or Yolov8 model.

[0044] At the same time, the present invention also provides a camera PTZ automatic tracking system based on target detection, which is used to implement the method described in any one of the above items, and the system includes:

[0045] PTZ camera: used to capture video data of the work site in real time and transmit it to the target detection module, with a built-in PTZ automatic tracking module;

[0046] Target detection module: It has a built-in target detection model based on Yolo model training, which is used for real-time target detection and calculation of the confidence of the detected target. It also divides the target into noise targets, suspected illegal targets and normal targets according to the confidence of the detected target. It ignores the noise targets, judges the normal targets as illegal and transmits the judgment results to the early warning module, and transmits the suspected illegal targets to the PTZ automatic tracking module.

[0047] PTZ automatic tracking module: automatically tracks and identifies suspected illegal targets and zooms in, and synchronously transmits the tracking result data to the target detection model until the target detection model confirms whether the suspected illegal target has violated the regulations and terminates the tracking;

[0048] Early warning module: Provide early warning and notification for traffic violations.

[0049] Compared with the prior art, the beneficial effects of the present invention are:

[0050] (1) The PTZ automatic tracking method of the camera based on target detection provided by the present invention performs real-time detection of the work site through the target detection model, divides the target into noise target, suspected violation target and normal target according to the confidence of the detected target, ignores the noise target, makes violation judgment and early warning broadcast for the normal target, automatically tracks and identifies the suspected violation target and zooms, and synchronously transmits the tracking result data to the target detection model until the target detection model determines whether the suspected violation target has any violation behavior, and terminates the tracking; at the same time, the method of the present invention performs real-time detection of the work site and tracking of suspected targets based on the PTZ pan-tilt. The PTZ camera is a camera with PTZ pan-tilt control, which has the functions of omnidirectional movement, tilt and automatic zoom, can meet the accuracy requirements of visual target tracking and the application requirements of the work site, and the suspected tracking target is always in the center of the lens of the PTZ camera, which makes up for the shortcomings of the fixed camera such as narrow monitoring field of view, monitoring blind spot, and easy obstruction and loss of the target, and has the characteristics of automatic tracking with little interference from the external environment, simple and efficient method, small amount of automatic tracking calculation, good real-time and robust tracking, etc.

[0051] (2) In the PTZ automatic tracking process of the present invention, the PTZ camera is deflected by calculating the offset. During the deflection, the offset similarity is calculated and a similarity threshold is set to determine whether the PTZ camera will perform the next deflection. This method can filter out a large amount of offset data with high repetition generated by the PTZ camera during the deflection process, thereby avoiding the camera shaking and unstable tracking caused by the instantaneous and frequent deflection of the camera. At the same time, after the PTZ camera deflects once, the target correction is immediately performed. Through the target correction, the center of the target image after the deflection coincides with the center of the lens image, so that the center of the target image is always located at the center of the lens image, thereby improving the real-time and high-precision requirements of the PTZ camera automatic tracking.

[0052] (3) The method of the present invention is not only applicable to work sites in the power field, but also applicable to digital and intelligent control devices for work sites in other fields such as railways and construction, and has good versatility. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 It is a flow chart of a camera PTZ automatic tracking method based on target detection according to an embodiment of the present invention;

[0054] Figure 2 A flowchart of PTZ automatic tracking and target correction according to an embodiment of the present invention;

[0055] Figure 3 The figure is a schematic diagram of the structure of a camera PTZ automatic tracking system based on target detection according to an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0057] Combination Figure 1 As shown, an embodiment of the present invention provides a camera PTZ automatic tracking method based on target detection, the method comprising the following steps:

[0058] Step S1: Collect and obtain historical video data of the work site as a data sample set.

[0059] Step S2: Using the data sample set, obtain the target detection model based on Yolo model training.

[0060] The Yolo model may adopt the Yolov5 model, the Yolov7 model or the Yolov8 model, and Yolov5 is preferred in the embodiment of the present invention.

[0061] The Yolov5 model is an industrial-grade preset model, which has excellent performance in industrial deployment, algorithm stability, model accuracy, and small weight file size. In terms of small target recognition, the detection accuracy and detection effect of the model can meet the monitoring and recognition requirements of the work site of the embodiment of the present invention.

[0062] In the process of obtaining the target detection model based on Yolo model training, when the number of training samples of a certain target is insufficient, the number of training samples of a certain target can be increased by data enhancement. The data enhancement method includes: scaling, translating, rotating, randomly adjusting hue / saturation / brightness, fusing, adding noise, cropping and other operations or a combination of operations on the image of the training sample, and saving the operated image as a newly added training sample.

[0063] Step S3: Based on the PTZ camera, the target detection model is used to perform real-time target detection on the work site, and the targets are divided into noise targets, suspected violation targets and normal targets according to the confidence of the detected targets;

[0064] Step S4: The target detection model ignores noise targets, makes violation judgments and issues warnings for normal targets, performs PTZ automatic tracking, identification and zooming on suspected illegal targets, and synchronously transmits the tracking result data of suspected illegal targets to the target detection model for identification and judgment, until it is determined whether the suspected illegal target is illegal, and the tracking is terminated.

[0065] The method of the present invention performs real-time detection and tracking of suspected targets at the work site based on the PTZ pan-tilt. The PTZ camera is a camera with PTZ pan-tilt control, and has the functions of omnidirectional movement, tilt and automatic zoom. It can meet the accuracy requirements of visual target tracking and the application requirements of the work site, and the suspected tracking target is always in the lens center of the PTZ camera, which makes up for the shortcomings of fixed cameras such as narrow monitoring field of view, monitoring blind spots, and easy obstruction and loss of targets. The method has the characteristics of automatic tracking with little interference from the external environment, simple and efficient method, small amount of automatic tracking calculation, good real-time and robust tracking, etc.

[0066] Among them, combined Figure 2 As shown, in step S4, the suspected illegal target is automatically tracked and identified by PTZ and zoomed, and the tracking result data of the suspected illegal target is synchronously transmitted to the target detection model for identification and judgment, until it is determined whether the suspected illegal target is illegal, and the tracking is terminated, which specifically includes:

[0067] Step S41: locking the suspected illegal target through the built-in PTZ automatic tracking module of the PTZ camera, and automatically controlling the PTZ pan-tilt deflection to track the target;

[0068] Step S42: after the deflection is completed, target correction is performed to place the identified suspected target at the center of the lens of the PTZ camera;

[0069] Step S43: After the target deviation correction is completed, the zoom factor is calculated according to the size of the suspected target in the image, the PTZ camera shoots real-time video and uploads it to the target detection model, and steps S3-S4 are repeated until it is confirmed whether the suspected illegal target is illegal and an early warning is broadcast.

[0070] When the PTZ automatic tracking module automatically tracks and identifies suspected illegal targets, it controls the PTZ camera to deflect and track the suspected illegal targets. Sometimes the suspected illegal targets are not in the center of the PTZ camera's lens after deflection. At this time, the target can be corrected after each deflection to be located in the center of the lens, so as to ensure the accuracy of subsequent offset data based on relative displacement processing, thereby ensuring the real-time and accuracy of subsequent automatic tracking.

[0071] Wherein, the step S41 specifically includes:

[0072] Step S411: Calculate the current relative offset of the center point of the target frame at the current moment relative to the center point of the current lens image according to the lens image size of the PTZ camera and the target frame size of the current suspected violation target obtained according to the target detection model; the specific calculation process is as follows:

[0073] First, the target detection model is used to obtain the coordinate values ​​(x1, y1) and (x2, y2) of the two diagonal points of the target frame in the current lens frame, where (x1, y1) is the coordinate of the upper left corner and (x2, y2) is the coordinate of the lower right corner.

[0074] Then, the code calculates the width x of the target box w and height y h , x w =x2-x1,y h , =y2-y1;

[0075] Next, the code calculates the center point coordinates of the target frame (real_x, real_y), where real_x = x1 + x w / 2; real_y=y1+y h, / 2;

[0076] Finally, the encoding calculates the normalized relative offset (x_offset, y_offset) of the center point coordinates (real_x, real_y) of the target box relative to the center point coordinates (target.shape[x] / 2, target.shape[y] / 2) of the current image. The calculation formula is:

[0077]

[0078]

[0079] In the above formulas (1)-(2), x_offset is the normalized offset of the center point of the target frame relative to the center point of the current lens image in the x direction, y_offset is the normalized offset of the center point of the target frame relative to the center point of the current lens image in the y direction, real_x is the x-axis coordinate of the center point of the target frame, real_y is the y-axis coordinate of the center point of the target frame, target.shape[x] is the width of the current lens image along the x-axis, target.shape[y] is the length of the current lens image along the y-axis, target.shape[x] / 2 is the x-axis coordinate of the center point of the current lens image, and target.shape[y] / 2 is the y-axis coordinate of the center point of the current lens image.

[0080] The above offset is normalized to a range of 0-1, so that the PTZ camera protocol can be called to give the PTZ pan / tilt a valid offset, thereby controlling the PTZ pan / tilt to perform effective deflection.

[0081] Step S412: Calculate the next moment relative offset of the center point of the target frame relative to the center point of the lens image, and calculate the similarity S between the next moment relative offset and the current relative offset according to the cosine similarity formula. AB ;

[0082]

[0083] In the above formula (3), A is the current relative offset vector; B is the relative offset vector at the next moment; Cosθ represents the cosine value of the angle between the two relative offset vectors; ||A||, ||B|| represent the modulus lengths of the A and B vectors;

[0084] Step S413: Determine whether to control the PTZ pan-tilt deflection according to the similarity threshold S0:

[0085] If the similarity S between the two relative offsets AB ≥S0, the relative offset at the next moment is discarded, no deflection is performed, and the process goes to step S412 to recalculate the updated relative offset at the next moment and the similarity;

[0086] If the similarity S between the relative offsets in two consecutive times AB <S0, then control the PTZ pan-tilt to deflect according to the relative offset at the next moment.

[0087] Since the deflection speed of the PTZ camera is much slower than the data processing speed, by filtering out the relatively high similarity offsets, it is possible to avoid the situation where the camera loses the target frame due to excessive deflection, as well as the camera jittering and shaking frequently and being unable to zoom in to identify the target frame caused by frequent deflection.

[0088] In the step S42, after the deflection is completed, calculate the relative offset and relative offset distance between the center of the deflected target frame and the center of the lens image, and compare the relative offset distance with the set distance threshold to determine whether to perform target rectification:

[0089] If the relative offset distance is less than the set distance threshold, there is no need to perform target rectification; otherwise, perform target rectification according to the relative offset between the center of the deflected target frame and the center of the lens image, and control the PTZ pan-tilt to deflect so that the suspected illegal target is placed at the center of the lens of the PTZ camera.

[0090] Among them, the target rectification formula f is as follows:

[0091]

[0092] In the above formula (4), dx is the distance between the center of the deflected target frame and the center of the lens image in the x-axis direction, dy is the distance between the center of the deflected target frame and the center of the lens image in the y-axis direction, uncertainty[x] is the distance threshold in the x-axis direction, and uncertainty[y] is the distance threshold in the y-axis direction.

[0093] If the distances between the center of the deflected target frame and the center of the lens image in the x-direction and y-direction are both less than the distance thresholds uncertainty[x] and uncertainty[y] in the corresponding directions, it means that the suspected illegal target after deflection is located at the center position of the lens image, and there is no need to perform target rectification; otherwise, it means that the suspected illegal target is not at the center position of the lens image, and target rectification needs to be performed according to the relative offset between the center of the deflected target frame and the center of the lens image, and place the suspected illegal target at the center of the lens of the PTZ camera.

[0094] Meanwhile, the embodiment of the present invention also provides a PTZ automatic tracking system for a camera based on target detection, as Figure 3 shown, the system includes:

[0095] PTZ camera: used to capture video data of the work site in real time and transmit it to the target detection module, with a built-in PTZ automatic tracking module;

[0096] Target detection module: It has a built-in target detection model based on Yolo model training, which is used for real-time target detection and calculation of the confidence of the detected target. It also divides the target into noise targets, suspected illegal targets and normal targets according to the confidence of the detected target. It ignores the noise targets, judges the normal targets as illegal and transmits the judgment results to the early warning module, and transmits the suspected illegal targets to the PTZ automatic tracking module.

[0097] PTZ automatic tracking module: automatically tracks and identifies suspected illegal targets and zooms in, and synchronously transmits the tracking result data to the target detection model until the target detection model confirms whether the suspected illegal target has violated the regulations and terminates the tracking;

[0098] Early warning module: Provide early warning and notification for traffic violations.

[0099] The above description is only an embodiment of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement and improvement made within the application scope of the present invention shall be included in the protection scope of the present invention.

Claims

1. A camera PTZ automatic tracking method based on target detection, characterized in that: The method comprises the following steps: Step S1: Collect and obtain historical video data of the work site as a data sample set; Step S2: Using the data sample set, obtain the target detection model based on Yolo model training; Step S3: Based on the PTZ camera, use the target detection model to perform real-time target detection on the work site and obtain target confidence; Targets are divided into noise targets, suspected illegal targets and normal targets according to target confidence; Step S4: The target detection model ignores the noise target, judges the normal target as illegal and issues warning, automatically tracks and identifies the suspected illegal target with PTZ and zooms, and synchronously transmits the tracking result data of the suspected illegal target to the target detection model for identification and judgment, until it is determined whether the suspected illegal target is illegal, and the tracking is terminated; In the step S4, the suspected illegal target is automatically tracked and identified by PTZ and zoomed, and the tracking result data of the suspected illegal target is synchronously transmitted to the target detection model for identification and judgment, until it is determined whether the suspected illegal target is illegal, and the tracking is terminated, which specifically includes: Step S41: locking the suspected illegal target through the built-in PTZ automatic tracking module of the PTZ camera, and automatically controlling the PTZ pan-tilt deflection to track the target; Step S42: after the deflection is completed, target correction is performed to place the identified suspected target at the center of the lens of the PTZ camera; Step S43: After the target deviation correction is completed, the zoom factor is calculated according to the size of the suspected target in the image, the PTZ camera shoots real-time video and uploads it to the target detection model, and steps S3-S4 are repeated until it is confirmed whether the suspected illegal target is illegal and an early warning is broadcast; The step S41 specifically includes: Step S411: Calculate the current relative offset of the center point of the target frame at the current moment relative to the center point of the current lens image according to the lens image size of the PTZ camera and the target frame size of the current suspected violation target obtained according to the target detection model; Step S412: Calculate the next moment relative offset of the center point of the target frame detected at the next moment relative to the center point of the lens image, and calculate the similarity S between the next moment relative offset and the current relative offset according to the cosine similarity formula. AB ; Where A is the current relative offset vector; B is the relative offset vector at the next moment; Cosθ represents the cosine value of the angle between the two relative offset vectors; ||A||, ||B|| represent the modulus lengths of the A and B vectors; Step S413: Determine whether to control the PTZ pan-tilt deflection according to the similarity threshold S0: If the similarity S between the two relative offsets AB ≥S0, the relative offset at the next moment is discarded, no deflection is performed, and the process goes to step S412 to recalculate the updated relative offset at the next moment and the similarity; If the similarity S between the relative offsets in the previous and next times AB <S0, then control the PTZ pan-tilt to deflect according to the relative offset at the next moment; The step S411 specifically includes: First, the target detection model is used to obtain the coordinate values ​​(x1, y1) and (x2, y2) of the two diagonal points of the target frame in the current lens frame, where (x1, y1) is the coordinate of the upper left corner and (x2, y2) is the coordinate of the lower right corner. Then, the code calculates the width x of the target box w and height y h , x w =x2-x1,y h =y2-y1; Next, the code calculates the center point coordinates of the target frame (real_x, real_y), where real_x = x1 + x w / 2; real_y=y1+y h / 2; Finally, the encoding calculates the normalized offset (x_offset, y_offset) of the center point coordinates (real_x, real_y) of the target box relative to the center point coordinates (target.shape[x] / 2, target.shape[y] / 2) of the current image. The calculation formula is: In the above formula, x_offset is the normalized offset of the center point of the target frame relative to the center point of the current lens image in the x direction, y_offset is the normalized offset of the center point of the target frame relative to the center point of the current lens image in the y direction, real_x is the x-axis coordinate of the center point of the target frame, real_y is the y-axis coordinate of the center point of the target frame, target.shape[x] is the width of the current lens image along the x-axis, target.shape[y] is the length of the current lens image along the y-axis, target.shape[x] / 2 is the x-axis coordinate of the center point of the current lens image, and target.shape[y] / 2 is the y-axis coordinate of the center point of the current lens image.

2. The method according to claim 1, characterized in that In the process of obtaining the target detection model based on Yolo model training, when the number of training samples for a certain target is insufficient, the number of training samples for a certain target is increased through data enhancement.

3. The method according to claim 2, characterized in that The data enhancement method includes: performing one or a combination of operations including scaling, translating, rotating, randomly adjusting hue / saturation / brightness, fusing, adding noise, and cropping on the image of the training sample, and saving the operated image as a newly added training sample.

4. The method according to claim 1, characterized in that: The step S42 specifically includes: After the deflection is completed, the relative offset and relative offset distance between the center of the deflected target frame and the center of the lens image are calculated, and the relative offset distance is compared with the set distance threshold to determine whether to perform target correction: If the relative offset distance is less than the set distance threshold, no target correction is required; Otherwise, target correction is performed according to the relative offset between the center of the deflected target frame and the center of the lens image, and the PTZ pan / tilt is controlled to deflect so that the suspected illegal target is placed in the center of the lens of the PTZ camera.

5. The method according to claim 4, characterized in that In step S42, the target correction formula f is as follows: In the above formula, dx is the distance between the center of the target frame and the center of the lens image in the x-axis direction after deflection, dy is the distance between the center of the target frame and the center of the lens image in the y-axis direction after deflection, uncertainty[x] is the distance threshold in the x-axis direction, and uncertainty[y] is the distance threshold in the y-axis direction; If the distance between the center of the deflected target frame and the center of the lens image in the x direction and the y direction is less than the distance thresholds uncertainty[x] and uncertainty[y] in the corresponding directions, no target correction is required. Otherwise, target correction is required based on the relative offset between the center of the deflected target frame and the center of the lens image, and the suspected illegal target is placed at the center of the PTZ camera's lens.

6. The method according to claim 1, characterized in that In step S2, the Yolo model adopts the Yolov5 model, the Yolov7 model or the Yolov8 model.

7. A camera PTZ automatic tracking system based on target detection, used to implement the method according to any one of claims 1 to 6, characterized in that: The system comprises: PTZ camera: used to capture video data of the work site in real time and transmit it to the target detection module, with a built-in PTZ automatic tracking module; Target detection module: It has a built-in target detection model based on Yolo model training, which is used for real-time target detection and calculation of the confidence of the detected target. It also divides the target into noise targets, suspected illegal targets and normal targets according to the confidence of the detected target. It ignores the noise targets, judges the normal targets as illegal and transmits the judgment results to the early warning module, and transmits the suspected illegal targets to the PTZ automatic tracking module. PTZ automatic tracking module: automatically tracks and identifies suspected illegal targets and zooms in, and synchronously transmits the tracking result data to the target detection model until the target detection model confirms whether the suspected illegal target has violated the regulations and terminates the tracking; Early warning module: Provide early warning and notification for traffic violations.