A multi-dimensional cooperative remote monitoring real-time target detection and tracking method and system

Abstract

The application discloses a kind of multi-dimensional cooperative real-time target detection tracking method and system of remote monitoring, which comprises the following steps: obtaining environmental image data and carrying out multi-stage image data preprocessing to obtain preprocessed environmental image data;Based on the improved YOLO target detection model, target detection processing is carried out and correlation matching is carried out to obtain the motion trajectory of numbered target;The motion trajectory of numbered target is dynamically cached and analyzed by target trigger mechanism to construct motion trajectory event picture;The number of targets in the motion trajectory event picture is judged, and target tracking is carried out according to the judgment result to realize real-time target detection tracking.The application can effectively improve the tracking robustness in nonlinear motion scene, reduce the position prediction error, and ultimately improve the accuracy of target tracking.The application can be widely applied in the field of Internet of Things artificial intelligence technology as a kind of multi-dimensional cooperative real-time target detection tracking method and system of remote monitoring.

Description

Technical Field

[0001] This invention relates to the field of Internet of Things and artificial intelligence technology, and in particular to a real-time target detection and tracking method and system for multi-dimensional collaborative remote monitoring. Background Technology

[0002] With the rapid development of IoT, AI, and edge computing technologies, target detection technology has been widely applied in various fields. However, the current technological system still has significant shortcomings. Traditional solutions rely excessively on cloud-based data processing, resulting in limited system scenarios, high response latency, and heavy network bandwidth pressure, making it difficult to meet real-time requirements. Furthermore, existing technologies are mostly limited to single detection functions, failing to improve capabilities in dynamic tracking and control, remote monitoring and interaction, and data value mining, leading to fragmented system functions and limited application scenarios. Specifically, monitoring equipment generally suffers from technical bottlenecks such as image quality limited by a fixed viewing angle, insufficient terminal computing power to support real-time processing needs, limited effective data value mining, and a lack of remote interaction interfaces. Summary of the Invention

[0003] To address the aforementioned technical problems, the present invention aims to provide a real-time target detection and tracking method and system for multi-dimensional collaborative remote monitoring, which can effectively improve tracking robustness in nonlinear motion scenarios, reduce position prediction errors, and ultimately improve target tracking accuracy.

[0004] The first technical solution adopted in this invention is: a real-time target detection and tracking method for multi-dimensional collaborative remote monitoring, comprising the following steps:

[0005] Acquire environmental image data and perform multi-level image data preprocessing to obtain preprocessed environmental image data;

[0006] Based on the improved YOLO target detection model, target detection and correlation matching are performed on the preprocessed environmental image data to obtain the motion trajectory of the numbered target;

[0007] The motion trajectory of numbered targets is dynamically cached and analyzed through a target triggering mechanism to construct a motion trajectory event screen;

[0008] The system determines the number of targets in the motion trajectory event image and tracks the targets based on the determination results, achieving real-time target detection and tracking.

[0009] Furthermore, the step of acquiring environmental image data and performing multi-level image data preprocessing to obtain preprocessed environmental image data specifically includes:

[0010] Construct a thread pool architecture based on priority scheduling, wherein the thread pool architecture includes high-level parallel threads and low-level threads;

[0011] The environment image data is obtained by using a high-level parallel thread and written to a buffer queue by a low-level thread to obtain the cached environment image data.

[0012] The cached environmental image data is preprocessed by scaling and color correction, and then transferred to the preprocessed data queue to obtain the preprocessed environmental image data.

[0013] Furthermore, the step of performing target detection processing and association matching on the preprocessed environmental image data based on the improved YOLO target detection model to obtain the motion trajectory of the numbered target specifically includes:

[0014] Based on the YOLO object detection model, INT8 quantization is performed and several detection head outputs are added to construct an improved YOLO object detection model;

[0015] Based on the backbone network module of the improved YOLO target detection model, feature extraction is performed on the preprocessed environmental image data to obtain environmental image feature data.

[0016] The neck network module based on the improved YOLO target detection model performs multi-scale fusion of environmental image feature data to obtain fused environmental image feature data.

[0017] The detection head module based on the improved YOLO target detection model performs target detection on the fused environmental image feature data to obtain target detection results, which include category, location and confidence information.

[0018] A multi-target correlation tracking algorithm based on adaptive motion modeling is used to correlate and match target detection results to obtain the motion trajectory of numbered targets.

[0019] Furthermore, the step of the multi-target association tracking algorithm based on adaptive motion modeling performing association matching on the target detection results to obtain the motion trajectory of the numbered targets specifically includes:

[0020] Initialize the trajectory and establish a trajectory sequence, which includes an active trajectory, an inactive trajectory, and a lost-track trajectory;

[0021] Based on the confidence information of the target detection results, high-resolution target detection boxes and low-resolution target detection boxes are determined;

[0022] A prediction model for nonlinear motion trajectories is constructed based on adaptive extended Kalman filtering, and the active trajectory is predicted to obtain the adaptive extended Kalman filter prediction frame.

[0023] The Hungarian algorithm is used to match and associate the adaptive extended Kalman filter prediction frames with high-resolution target detection boxes. If the association fails, the high-resolution target detection boxes are loaded into the inactive trajectory. If the association succeeds, the high-resolution target detection boxes are loaded into the active trajectory.

[0024] The Hungarian algorithm is used to match low-scoring target detection boxes with inactive trajectories. If the association fails, the low-scoring target detection boxes are loaded into the lost trajectory. If the association succeeds, the low-scoring target detection boxes are loaded into the active trajectory.

[0025] The Hungarian algorithm is used to match and associate inactive trajectories with lost trajectories. If the association fails, the confidence level is reduced. If the confidence level after reduction is greater than the preset threshold, the current lost trajectory is retained. If the confidence level after reduction is less than the preset threshold, the current lost trajectory is deleted. If the association is successful, the inactive trajectory is loaded into the active trajectory.

[0026] Anomaly detection is performed on the final activated trajectory, and the motion trajectory of the target with a number is output.

[0027] Furthermore, the step of dynamically caching and analyzing the motion trajectory of numbered targets through a target triggering mechanism to construct a motion trajectory event screen specifically includes:

[0028] Perform valid target detection on the motion trajectory of the numbered target. If a valid target is found, record the current timestamp and store it in the video buffer queue.

[0029] The effective frame sequence in the recording buffer queue is compressed into an H.264 standard bitstream using the FFmpeg tool and then packaged into an MP4 video file. The effective frame sequence represents the pre-recorded frames buffered before the start of recording and the subsequent real-time frames.

[0030] Generate independent log files and statistical reports based on MP4 video files;

[0031] Incremental learning processing and analysis are performed on independent log files and statistical reports, and combined retrieval is carried out to construct motion trajectory event images.

[0032] Furthermore, the step of determining the number of targets in the motion trajectory event image and tracking targets based on the determination result to achieve real-time target detection and tracking specifically includes:

[0033] The number of targets in the motion trajectory event image is determined, and the determination result is obtained;

[0034] If the number of targets in the judgment result is greater than the preset threshold, the motion trajectory event screen is corrected to obtain the corrected motion trajectory event screen.

[0035] Target tracking is performed based on the corrected motion trajectory event footage;

[0036] If the number of targets in the judgment result is less than the preset threshold, target tracking is performed on the motion trajectory event screen to achieve real-time target detection and tracking.

[0037] Furthermore, the step of correcting the motion trajectory event image if the number of targets in the judgment result is greater than a preset threshold, to obtain a corrected motion trajectory event image, specifically includes:

[0038] If the number of targets in the judgment result is greater than the preset threshold, all targets are set to form a data point set, and there exists a hypothetical data point whose total distance to the center coordinates of other data points is the minimum.

[0039] Search for hypothetical data points, allowing the exclusion of some data points that are too far out of the loop;

[0040] The objective function of minimizing the total distance is used to iteratively search and obtain the corrected center coordinates of several targets.

[0041] Calculate the distance between the corrected center coordinates of several targets and the center of the screen until a preset termination condition is met. Update the total distance value, determine the remaining data points as targets, and obtain the corrected motion trajectory event screen.

[0042] Furthermore, the step of tracking the motion trajectory event image to achieve real-time target detection and tracking if the number of targets in the judgment result is less than a preset threshold specifically includes:

[0043] If the number of targets in the judgment result is less than the preset threshold, calculate the distance deviation between the center coordinates of the first target and the center of the screen.

[0044] The distance deviation value is processed by median filtering and cubic spline interpolation to obtain the interpolated and filtered distance deviation value.

[0045] Based on the PID fuzzy controller, the distance deviation between the interpolated and filtered target deviation and the center of the image is fuzzified, fuzzy inferred, and sharpened, and a PWM control signal is output.

[0046] The PWM control signal drives the gimbal motor to move closer to the target, enabling real-time target detection and tracking.

[0047] Furthermore, it also includes:

[0048] The message queue telemetry transmission protocol enables command transmission between the end-to-cloud and end-to-end.

[0049] By sending JSON-formatted content and entering preset interface commands and specific parameters in the command field, you can obtain the device's IP address, directly control the PTZ's reset and four-way movement, and switch the device's operating mode.

[0050] A real-time video transmission channel is built based on a real-time streaming protocol.

[0051] The second technical solution adopted in this invention is: a real-time target detection and tracking system for multi-dimensional collaborative remote monitoring, comprising:

[0052] The first module is used to acquire environmental image data and perform multi-level image data preprocessing to obtain preprocessed environmental image data.

[0053] The second module is used to perform target detection processing and correlation matching on the preprocessed environmental image data based on the improved YOLO target detection model to obtain the motion trajectory of the numbered target.

[0054] The third module is used to dynamically cache and analyze the motion trajectory of numbered targets through a target triggering mechanism, and to construct a motion trajectory event screen.

[0055] The fourth module is used to determine the number of targets in the motion trajectory event screen and to track targets based on the determination results, thereby achieving real-time target detection and tracking.

[0056] The beneficial effects of the method and system of this invention are as follows: This invention acquires environmental image data and performs multi-level image data preprocessing, and then performs target detection processing and correlation matching on the preprocessed environmental image data based on an improved YOLO target detection model. Relying on CPU-NPU multi-threaded pipeline processing, it does not rely on the cloud and quickly and stably keeps the target in the center of the screen, effectively improving the tracking robustness in nonlinear motion scenes, reducing position prediction errors, increasing the success rate of identity recovery after occlusion or missed detection, and reducing the trajectory drift rate caused by false detection or sudden noise. Furthermore, it dynamically caches and analyzes the motion trajectory of numbered targets through a target triggering mechanism, and finally judges the number of targets in the motion trajectory event screen and performs target tracking based on the judgment result, thereby improving the accuracy of target tracking. Attached Figure Description

[0057] Figure 1 This is a flowchart of the steps of a real-time target detection and tracking method for multi-dimensional collaborative remote monitoring according to the present invention;

[0058] Figure 2 This is a structural block diagram of a real-time target detection and tracking system for multi-dimensional collaborative remote monitoring according to the present invention;

[0059] Figure 3This is a schematic diagram of the system framework for multi-dimensional collaborative remote monitoring provided in a specific embodiment of the present invention;

[0060] Figure 4 This is a schematic diagram of a real-time target detection and tracking method provided in a specific embodiment of the present invention;

[0061] Figure 5 This is a schematic diagram of the input and output structure of the target detection model provided in a specific embodiment of the present invention;

[0062] Figure 6 This is a flowchart illustrating the multi-target correlation tracking algorithm provided in a specific embodiment of the present invention;

[0063] Figure 7 This is a schematic diagram of the data mining process provided in a specific embodiment of the present invention;

[0064] Figure 8 This is a schematic diagram of the control flow provided in a specific embodiment of the present invention. Detailed Implementation

[0065] The present invention will now be described in further detail with reference to the accompanying drawings and specific embodiments. The step numbers in the following embodiments are only for ease of explanation and do not limit the order of the steps. The execution order of each step in the embodiments can be adapted according to the understanding of those skilled in the art.

[0066] Reference Figure 1 This invention provides a real-time target detection and tracking method for multi-dimensional collaborative remote monitoring, the method comprising the following steps:

[0067] S100: Acquire environmental image data and perform multi-level image data preprocessing to obtain preprocessed environmental image data;

[0068] S110. Construct a thread pool architecture based on priority scheduling, wherein the thread pool architecture includes high-level parallel threads and low-level threads;

[0069] First, it needs to be explained that, as Figure 3As shown, the system includes an image acquisition module for acquiring image data; an edge computing module for image processing, model inference, image encoding, and video synthesis; a motion control module for calculating motion trajectories and outputting PWM control signals; a gimbal for mounting the image acquisition module, which adjusts the camera's viewing angle through its own rotation; an external storage module for storing valid files, including videos, log files, and statistical reports; and a remote communication module for remote data exchange with servers or other devices. The image acquisition module is connected to the edge computing module. The remote communication module is connected to both the edge computing module and the motion control module, forming two bidirectional data transmission channels. The motion control module is connected to the edge computing module and receives control commands from it. The gimbal is connected to the motion control module and receives its control commands to track the target object. The edge computing module is connected to the external storage module. The image acquisition module transmits the acquired image data to the edge computing module. The image acquisition module is mounted on the gimbal, and the gimbal's rotation adjusts the camera's viewing angle to center the target in the image.

[0070] S120. Obtain environmental image data through a high-level parallel thread, and write the environmental image data into a buffer queue through a low-level thread to obtain cached environmental image data.

[0071] Specifically, the image acquisition module is fixedly installed at the top of the system and continuously acquires environmental image data through a camera, forming a continuous video stream in time sequence. To optimize resource utilization, the system constructs a thread pool architecture based on priority scheduling: six high-level parallel threads are used for ADC image acquisition, and two low-level threads are used to write the acquired raw image data into the buffer queue Raw_Queue.

[0072] S130. Perform preprocessing operations such as size scaling and color correction on the cached environmental image data in sequence, and transfer it to the preprocessed data queue to obtain the preprocessed environmental image data.

[0073] Specifically, after image data is extracted from the Raw_Queue buffer, it first undergoes multi-stage preprocessing before being transferred to the Pre_Queue preprocessed data queue. Preprocessing operations include size scaling (uniformly adjusting the input resolution to 640×640×3) and color correction (converting BGR format to RGB format, followed by brightness optimization through histogram equalization). Subsequently, the edge computing module inputs the data from the Pre_Queue preprocessed data queue into the improved YOLO object detection model.

[0074] S200. Based on the improved YOLO target detection model, target detection processing is performed on the preprocessed environmental image data and correlation matching is performed to obtain the motion trajectory of the numbered target;

[0075] Specifically, based on the YOLO target detection model, INT8 quantization is performed and several detection head outputs are added to construct an improved YOLO target detection model. Based on the backbone network module of the improved YOLO target detection model, features are extracted from the preprocessed environmental image data to obtain environmental image feature data. Based on the neck network module of the improved YOLO target detection model, multi-scale fusion of the environmental image feature data is performed to obtain fused environmental image feature data. Based on the detection head module of the improved YOLO target detection model, target detection is performed on the fused environmental image feature data to obtain target detection results, which include category, location, and confidence information. A multi-target association tracking algorithm based on adaptive motion modeling is used to associate and match the target detection results to obtain the motion trajectory of numbered targets.

[0076] In this embodiment, preprocessed image tensor data is first acquired; then, based on edge computing requirements, the YOLO model is improved to obtain the improved YOLO model. The improvements are as follows:

[0077] 1) Model Structure Adjustment: The post-processing module of the original model was removed, and the number of output heads was expanded from 6 to 9. Three new output heads were added to calculate the sum of confidence scores for small, medium, and large targets, respectively, for threshold filtering in fast post-processing. The input-output structure of the model is as follows: Figure 5 As shown.

[0078] 2) Model quantization and conversion: The pre-trained model is quantized with INT8 (FP16→INT8) to compress the model size and improve inference speed; then the model format is converted to adapt to the edge computing module to ensure hardware compatibility and improve computational efficiency.

[0079] The image data is further input into the improved YOLO model for inference. Feature extraction is performed through a backbone network composed of several modules such as C3k2 and C2PSA to obtain three-level features: 80×80×C1, 40×40×C2, and 20×20×C3. The three-level feature data are then fused at multiple scales through a neck network based on the PANet structure to obtain fused three-level feature data: 80×80×C4, 40×40×C5, and 20×20×C6. Finally, the fused features are input into a lightweight classification detection head through the head network to output the target detection results.

[0080] Given the performance limitations of edge devices, the edge computing module employs a strategy of multi-threaded concurrent collaboration between the CPU and NPU. Specifically, the CPU is primarily responsible for preprocessing and post-processing, while the NPU handles most of the inference computations, improving overall throughput through multi-threading and pipelined design.

[0081] Finally, after the model inference results, the output detection head results of the model undergo post-processing steps (coordinate transformation, confidence filtering, non-maximum suppression, classification focus loss) to extract the target detection results: category, location and confidence information: [classes,boxes(x,y,h,w),scores].

[0082] Further, the trajectory is initialized, and a trajectory sequence is established, including the active trajectory, the inactive trajectory, and the lost-track trajectory. Based on the confidence information of the target detection results, high-resolution target detection boxes and low-resolution target detection boxes are determined. A prediction model for nonlinear motion trajectories is constructed based on adaptive extended Kalman filtering to predict the active trajectory, obtaining the adaptive extended Kalman filter prediction frame. The adaptive extended Kalman filter prediction frame and the high-resolution target detection box are matched and associated using the Hungarian algorithm. If the association fails, the high-resolution target detection box is loaded into the inactive trajectory; if the association succeeds, the high-resolution target detection box is loaded into the active trajectory. The Hungarian algorithm is used to match low-scoring target detection boxes with inactive trajectories. If the matching fails, the low-scoring target detection box is loaded into the lost trajectory; if the matching succeeds, the low-scoring target detection box is loaded into the active trajectory. The Hungarian algorithm is also used to match and associate inactive trajectories with lost trajectories. If the matching fails, confidence decay is performed. If the decayed confidence value is greater than a preset threshold, the current lost trajectory is retained; if the decayed confidence value is less than the preset threshold, the current lost trajectory is deleted. If the matching succeeds, the inactive trajectory is loaded into the active trajectory. Anomaly detection is performed on the final active trajectory, and the motion trajectory of the target with a number is output.

[0083] In this embodiment, after obtaining the target detection results, they are input into a multi-target association tracking algorithm for association matching of the detection results, generating the number of each target and the motion trajectory between consecutive images. In dynamic tracking scenarios, targets may undergo nonlinear motions such as acceleration or sudden changes in direction, and when the camera viewpoint is adjusted, the target's motion trajectory in the image coordinate system will also exhibit significant nonlinear characteristics. This embodiment of the invention proposes a multi-target association tracking algorithm based on adaptive motion modeling, such as... Figure 6 As shown, the specific content is as follows:

[0084] 1) Initialize the trajectory and establish the trajectory sequence: active trajectory, inactive trajectory, and lost trajectory;

[0085] 2) Input the target detection results and divide them into high-scoring boxes D based on the confidence level.high and low-resolution frame D low ;

[0086] 3) Use Adaptive Extended Kalman Filter (AEKF) to construct a prediction model for nonlinear motion trajectories, predict the next frame of the activated trajectory, and dynamically adjust the process noise covariance matrix Q and the observation noise covariance matrix R.

[0087] 4) First association matching: Associating the current high-scoring bounding box D using the Hungarian algorithm. high If the current frame does not match the AEKF predicted frame of the previous frame, it will be loaded into the inactive trajectory; otherwise, it will be loaded into the active trajectory and the trajectory will be updated.

[0088] 5) Second association matching: Associating the current low-scoring bounding box D using the Hungarian algorithm. low If no match is found between the current frame and the inactive trajectory, the current frame will be loaded into the lost trajectory; otherwise, the current frame will be loaded into the active trajectory and the trajectory will be updated.

[0089] 6) The nth association matching: Inactive trajectories and lost-track trajectories are associated using the Hungarian algorithm. If no match is found, the confidence score of the current frame is reduced (scores). k =β×scores k-1 (β is the attenuation coefficient). If the confidence level is greater than the preset threshold LOSS_THRESHOLD, the frame will still be kept in the lost track trajectory; otherwise, the current frame and the accumulated lost track trajectory will be deleted. If a match is successfully found, the current frame will be loaded into the active trajectory and the trajectory will be updated.

[0090] 7) When the association matching is successful, the result is subjected to abnormal trajectory detection. If the acceleration change exceeds the preset threshold for consecutive MAX_A2ERR frames, it is determined to be an abnormal trajectory and the trajectory is reset. If there is no abnormality, the target number and output trajectory are generated.

[0091] This algorithm effectively improves tracking robustness in nonlinear motion scenarios, reduces position prediction error, increases the success rate of identity recovery after occlusion or missed detection, and reduces trajectory drift rate caused by false detection or sudden noise.

[0092] S300: Dynamically cache and analyze the motion trajectory of numbered targets through a target triggering mechanism to construct a motion trajectory event screen;

[0093] Specifically, effective target detection is performed on the motion trajectory of the numbered target. If an effective target is found, the current timestamp is recorded and stored in the video recording buffer queue. The effective frame sequence in the video recording buffer queue is compressed into an H.264 standard bitstream using FFmpeg and encapsulated to generate an MP4 video file. The effective frame sequence represents the pre-recorded frames buffered before the recording start time and the subsequent real-time frames. Independent log files and statistical reports are generated based on the MP4 video files. Incremental learning processing and analysis are performed on the independent log files and statistical reports, and combined retrieval is performed to construct the motion trajectory event screen.

[0094] In this embodiment, when a valid target is detected for the first time, the following steps are performed:

[0095] 1) Record the current timestamp. If the system is not in recording mode, start the video recording process, mark the pre-recorded frames buffered before the recording start time and the subsequent real-time frames as valid frames, and store them in the recording buffer queue Record_Queue.

[0096] 2) Continue the above operations as long as the target persists and the stopping condition is not triggered;

[0097] 3) When any stopping condition is met (including the target disappearing continuously for a longer period than a preset threshold (e.g., 5 seconds), or the cumulative duration of the cache reaches the upper limit (e.g., 10 minutes)), the cached valid frame sequence is compressed into an H.264 standard bitstream using the FFmpeg tool, encapsulated into an MP4 video file, and then stored in an external storage module. At the same time, the Record_Queue is cleared and other cached resources are released.

[0098] Furthermore, each MP4 file is associated with an independent log file and statistical report. The log file contains basic attribute information (creation time of the video file and log file, storage path, and checksum), as well as detailed data of each detection during the video recording process (detection timestamp, target category label, quantity statistics, and coordinate area). The statistical report is automatically generated in JSON format and includes: creation time; frequency statistics: the cumulative number of times each target category is detected within the video period; peak statistics: the maximum number of the same type of target present simultaneously in a single frame; and duration: the time interval from the first appearance of the target to its last disappearance.

[0099] Furthermore, the stored data can support the following extended applications: building incremental learning datasets: driving model retraining by filtering low-confidence detection samples in logs to achieve iterative optimization of the model; supporting behavioral analysis of specific targets, providing a structured data source for the construction of industry knowledge graphs; building databases and combining conditional retrieval: realizing compound queries based on time range, target type, and quantity thresholds through preset scripts.

[0100] S400: Determines the number of targets in the motion trajectory event screen and performs target tracking based on the determination result, thereby achieving real-time target detection and tracking.

[0101] Specifically, the number of targets in the motion trajectory event image is judged to obtain a judgment result; if the number of targets in the judgment result is greater than a preset threshold, the motion trajectory event image is corrected to obtain a corrected motion trajectory event image; target tracking is performed based on the corrected motion trajectory event image; if the number of targets in the judgment result is less than the preset threshold, target tracking is performed on the motion trajectory event image to achieve real-time target detection and tracking.

[0102] If the number of targets in the judgment result is greater than a preset threshold, all targets are set into a data point set, and there exists a hypothetical data point whose total distance to the center coordinates of the other data points is the smallest. The hypothetical data point is searched, and some data points that are too far out of the group are allowed to be excluded. The search is iteratively performed according to the objective function of minimizing the total distance value to obtain the corrected center coordinates of several targets. The distance between the corrected center coordinates of several targets and the center of the screen is calculated until the preset termination condition is met, the total distance value is updated, the remaining data points are determined as targets, and the corrected motion trajectory event screen is obtained.

[0103] If the number of targets in the judgment result is less than a preset threshold, calculate the distance deviation between the center coordinates of the first target and the center of the screen; perform median filtering and cubic spline interpolation on the distance deviation to obtain the interpolated and filtered distance deviation; based on the PID fuzzy controller, perform fuzzification, fuzzy inference and sharpening processing on the distance deviation between the interpolated and filtered target and the center of the screen, and output a PWM control signal; drive the gimbal motor to move closer to the target according to the PWM control signal to realize real-time target detection and tracking.

[0104] First, if multiple targets appear in the image, calculate the correction center for each target. The calculation process for the correction center is as follows:

[0105] 1) Basic assumptions, symbol definitions, and objective function;

[0106] Since the number of targets captured and detected is limited, that is, the total number of targets has an upper limit n;

[0107] Let the center coordinates of each target be a point set: P (t) ={P1(x1, y1), ..., P n (x n y n )}, P i (x i y i ) represents the original center coordinates of the target point, and its set is P. (i), i = 1, 2, ..., n;

[0108] Suppose there exists a point The total distance S to other target center points is minimized, allowing

[0109] Find point C * During the process, it is permissible to exclude P. (t) Medium to L max A target that is too far removed from the group.

[0110] Objective function:

[0111]

[0112] Its representation point C * Excluding at most L max After 1 point, the remaining point set Minimum total distance S to C min .

[0113] 2) Initialization;

[0114] Set the current point set P (t) ;

[0115] Initial center

[0116] Initial total distance

[0117] The cumulative number of excluded points is L = t;

[0118] The number of iterations, k.

[0119] 3) Set termination conditions;

[0120] The iteration process stops when the following conditions are met:

[0121] Exclusion point reaches the limit: L≥L max ;

[0122] Total distance convergence: |S (k) -S (k-1) |<ε, where ε is a minterm.

[0123] 4) Update and iterate;

[0124] For the k-th iteration (k>1):

[0125] Calculate the distance from each point to C (k) Distance:

[0126] Select the point furthest away as the candidate outlier:

[0127] Exclude this point: L = L + 1, update the point set:

[0128] Based on the updated point set, the computation center is:

[0129] Total distance updated:

[0130] Furthermore, if there are few targets in the image, the target numbered 1 will be tracked, and the center coordinates of that target will be used as the center coordinates of the whole.

[0131] Furthermore, calculate the center coordinates [X] of the target. k Y k [T and the center point of the image [X]] half Y half Deviation value of T:

[0132]

[0133] Where, ΔX k Let ΔY be the pixel deviation in the horizontal direction between the center of the target in the k-th frame and the center of the image captured by the camera. k denoted as the vertical pixel deviation between the target center and the image center captured by the camera in the k-th frame.

[0134] After receiving the deviation data from the edge computing module, the control module loads new data into a finite-sized circular queue structure and uses median filtering and cubic spline interpolation to make the data more continuous and smooth. The motion control module uses a fuzzy incremental PID control algorithm to generate PWM signals to drive the gimbal.

[0135] PID control parameter K x =[K p K i K d The update is performed in the fuzzy controller, which includes three parts: fuzzification, fuzzy inference (the fuzzy rule base will be denoted by M hereafter), and defuzzification. Details are as follows:

[0136] Fuzzification: The target deviation e(k) and the rate of change of deviation Δe(k) are mapped to a fuzzy set {negative large NB, negative medium NM, zero ZO, positive medium PM, positive large PB} through triangular membership functions. Triangular membership functions:

[0137]

[0138] Among them, K x For K p K i K d A general term, K representsx In the membership degree of the rule base, c is the center point of the membership function, and w is the half width of the scaled bottom edge.

[0139] Fuzzy reasoning: Establish a fuzzy rule base M with 25 rules, in the following form:

[0140] R i :IF e(k)=A AND Δe(k)=B THENΔK p =C, ΔK i =D, ΔK d =E

[0141] Where A, B, C, D, and E represent an item in the fuzzy rule base M, and their specific contents are as follows:

[0142]

[0143] Clarification:

[0144]

[0145] Where ΔK x It is ΔK p ΔK i ΔK d The collective term, representing the clarified K x The increment; K represents x In the fuzzy rule base, the unfuzzy value defined by the τth rule (corresponding to the constants represented by PB, PM, ZO, NM, and NB).

[0146] Parameter update:

[0147]

[0148] Furthermore, the incremental PID formula is: Δu(k)=K p Δe(k)+K i e(k)+K d Δ 2 e(k).

[0149] Where Δu(k) is the increment of the output PWM signal, and e(k) is the pixel deviation of the k-th frame.

[0150] Furthermore, the PID output is used to apply upper and lower limit amplitudes, which are then passed through a low-pass filter before being used to output a PWM signal to the gimbal motor.

[0151] Furthermore, when the target's correction center is located near the center of the image, not responding to the gimbal can result in a more stable effect.

[0152] Furthermore, as the gimbal adjusts the camera's horizontal and vertical angles, the camera gradually approaches the target, ultimately ensuring the target remains centered in the frame. Through these operations, a smooth and rapid real-time tracking effect can be achieved.

[0153] Finally, this embodiment of the invention also proposes a communication method and interface implementation scheme for remote streaming and remote operation, specifically including:

[0154] 1) The remote communication module can access the Internet through the widely used WiFi technology, taking advantage of its convenient wireless connection characteristics to meet the needs of mobile devices or other scenarios where wiring is inconvenient; on the other hand, it can also access the Internet through the RJ45 interface, relying on the stability and high speed of the wired network to ensure the reliability of data transmission, thereby achieving a stable connection in complex network environments. Whether in urban environments with a lot of signal interference or remote areas with limited network coverage, it can ensure that the device can connect to the Internet normally.

[0155] 2) The MQTT protocol (Message Queuing Telemetry Transport) is lightweight and low-overhead. The system deploys an MQTT client service, which establishes a connection with the MQTT server (Broker) to realize the transmission of instructions between the end-to-cloud and end-to-end (such as IP acquisition, PTZ control, mode switching, etc.). It also supports QoS1 level message reliability guarantee to ensure that important instructions can be delivered accurately even when the network fluctuates, avoiding device operation errors or functional abnormalities due to instruction loss.

[0156] Furthermore, after a device subscribes to a topic and starts the MQTT service, the user sends JSON-formatted content to that topic. This can be done on multiple devices; the device ID field allows for specific adjustments by specifying device numbers; and the command field allows for obtaining the device's IP address, directly controlling the PTZ's reset and four-way movement, switching device operating modes (restart or shutdown, hibernation, automatic patrol, manual control), and other functions by entering preset interface commands and specific parameters.

[0157] 3) An optimized RTSP (Real-Time Stream Protocol) video streaming design constructs a highly robust real-time video transmission channel. Specifically, an RTSP server is deployed on the device side, supporting 1080P@30fps high-definition video streaming under the H.264 encoding standard. An adaptive bitrate adjustment mechanism dynamically matches network bandwidth (adjustment range 500Kbps-8Mbps), maintaining an end-to-end transmission latency of <300ms, allowing access via web servers and streaming media servers. Furthermore, the SRTP (Secure Real-time Transport Protocol) protocol is used for encrypted video stream transmission, combined with a two-way authentication mechanism using device identity certificates to prevent unauthorized access.

[0158] In summary, such as Figure 4 As shown, this embodiment of the invention first acquires video streams in real time through an image acquisition module; after preprocessing, the video frames are input into an improved target detection model for inference; the inference results are post-processed to obtain detection results; and an improved multi-target association tracking algorithm generates object numbers and motion trajectories; the system performs data value mining on valid frames, encodes them, and transfers them to an external storage module; the position deviation is calculated, and the motion control module performs filtering, interpolation, and fuzzy incremental PID calculations on the deviation data, outputting control signals to drive the gimbal to track in real time. The remote communication module adopts an MQTT-RTSP (Message Queuing Telemetry Transport Protocol-Real-Time Streaming Protocol) layered protocol architecture, thereby forming a multi-dimensional collaborative remote monitoring system. This invention has the following significance: it constructs a closed-loop physical tracking system at the edge, eliminating the need for cloud processing and exhibiting high robustness; it emphasizes data value mining and can be used for various purposes such as model iteration and research on specific target behavior; and its layered communication architecture provides a convenient remote human-computer interaction interface.

[0159] Therefore, the embodiments of the present invention have the following improvements compared to the prior art:

[0160] 1) An edge-based physical tracking closed-loop system was constructed. Through an improved target detection model, an improved multi-target correlation tracking algorithm, and adaptive gimbal tracking technology, relying on CPU-NPU multi-threaded pipeline processing of data, the system can quickly and stably keep the target centered in the frame without relying on the cloud.

[0161] 2) Emphasis on data value mining. The target-triggered mechanism automatically records video and automatically collects various valid data, generating logs and statistical reports. The saved data is highly effective and can be used for subsequent development based on the needs of different scenarios, such as subsequent model iteration, behavioral research on specified targets, and data organization and querying.

[0162] 3) Layered communication architecture. It integrates MQTT and RTSP protocols, and has the functions of remote control command interface and real-time encrypted remote transmission of images, providing convenient interface methods for servers and users to call.

[0163] Reference Figure 2 A real-time target detection and tracking system for multi-dimensional collaborative remote monitoring, comprising:

[0164] The first module 201 is used to acquire environmental image data and perform multi-level image data preprocessing to obtain preprocessed environmental image data.

[0165] The second module 202 is used to perform target detection processing and correlation matching on the preprocessed environmental image data based on the improved YOLO target detection model to obtain the motion trajectory of the numbered target;

[0166] The third module 203 is used to dynamically cache and analyze the motion trajectory of numbered targets through a target triggering mechanism, and construct a motion trajectory event screen.

[0167] The fourth module 204 is used to determine the number of targets in the motion trajectory event screen and to perform target tracking based on the determination result, thereby realizing real-time target detection and tracking.

[0168] The content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0169] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of this application.

Claims

1. A multi-dimensional cooperative remote monitoring real-time target detection and tracking method, characterized in that, The method comprises the following steps: obtaining environment image data and performing multi-level image data preprocessing to obtain preprocessed environment image data; performing target detection processing on the preprocessed environment image data based on an improved YOLO target detection model and performing correlation matching to obtain the motion trajectory of the numbered target; performing dynamic caching and analysis on the motion trajectory of the numbered target through a target triggering mechanism to construct a motion trajectory event picture; judging the number of targets in the motion trajectory event picture and performing target tracking according to the judgment result to realize real-time target detection tracking.

2. The real-time target detection and tracking method of multi-dimensional cooperative remote monitoring according to claim 1, characterized in that, The step of obtaining environment image data and performing multi-level image data preprocessing to obtain preprocessed environment image data specifically comprises: constructing a thread pool architecture based on priority scheduling, wherein the thread pool architecture comprises high-level parallel threads and low-level threads; obtaining environment image data through the high-level parallel threads and writing the environment image data into a buffer queue through the low-level threads to obtain cached environment image data; performing preprocessing operations of size scaling and color correction on the cached environment image data in sequence and storing them into a preprocessing data queue to obtain preprocessed environment image data.

3. The real-time target detection and tracking method of multi-dimensional cooperative remote monitoring according to claim 2, characterized in that, The step of performing target detection processing on the preprocessed environment image data based on the improved YOLO target detection model and performing correlation matching to obtain the motion trajectory of the numbered target specifically comprises: constructing an improved YOLO target detection model based on the YOLO target detection model, INT8 quantization and a plurality of detection head output ends; performing feature extraction on the preprocessed environment image data based on the backbone network module of the improved YOLO target detection model to obtain environment image feature data; performing multi-scale fusion on the environment image feature data based on the neck network module of the improved YOLO target detection model to obtain fused environment image feature data; performing target detection on the fused environment image feature data based on the detection head module of the improved YOLO target detection model to obtain target detection results, wherein the target detection results comprise category, position and confidence information; performing correlation matching on the target detection results based on a multi-target correlation tracking algorithm based on adaptive motion modeling to obtain the motion trajectory of the numbered target.

4. The real-time target detection and tracking method of multi-dimensional cooperative remote monitoring according to claim 3, characterized in that, The step of performing correlation matching on the target detection results based on the multi-target correlation tracking algorithm based on adaptive motion modeling to obtain the motion trajectory of the numbered target specifically comprises: initializing a trajectory and establishing a trajectory sequence, wherein the trajectory sequence comprises activated trajectories, unactivated trajectories and lost trajectories; determining high-score target detection boxes and low-score target detection boxes according to the confidence information of the target detection results; constructing a prediction model of a nonlinear motion trajectory based on an adaptive extended Kalman filter to predict the activated trajectories to obtain an adaptive extended Kalman filter prediction frame; performing matching correlation on the adaptive extended Kalman filter prediction frame and the high-score target detection boxes through the Hungarian algorithm, and if the correlation fails, the high-score target detection boxes are loaded into the unactivated trajectories, and if the correlation succeeds, the high-score target detection boxes are loaded into the activated trajectories; The low-score target detection frame is matched with the inactivated track by the Hungarian algorithm, and if the matching fails, the low-score target detection frame is loaded into the lost track, and if the matching succeeds, the low-score target detection frame is loaded into the activated track; The inactivated track is matched with the lost track by the Hungarian algorithm, and if the matching fails, the confidence is decayed, if the decayed confidence is greater than a preset threshold, the current lost track is retained, if the decayed confidence is less than the preset threshold, the current lost track is deleted, and if the matching succeeds, the inactivated track is loaded into the activated track; The final activated track is detected for abnormal track, and a motion track of a numbered target is output.

5. The multi-dimensional cooperative remote monitoring real-time target detection and tracking method according to claim 4, characterized in that, The motion track of the numbered target is detected for effective target, and if there is an effective target, a current time stamp is recorded and stored in a video cache queue. The effective frame sequence in the video cache queue is compressed into an H.264 standard code stream by the FFmpeg tool, and an MP4 video file is generated by encapsulation, and the effective frame sequence represents pre-recording frames buffered before a recording starting time and subsequent real-time frames. An independent log file and a statistical report are generated according to the MP4 video file. The independent log file and the statistical report are processed and analyzed by incremental learning, and are combined for retrieval, and a motion track event picture is constructed. The number of targets in the motion track event picture is judged, and target tracking is performed according to a judgment result, and real-time target detection and tracking are realized.

6. The multi-dimensional cooperative remote monitoring real-time target detection and tracking method according to claim 5, characterized in that, The number of targets in the motion track event picture is judged, and a judgment result is obtained. If the number of targets in the judgment result is greater than a preset threshold, the motion track event picture is modified, and a modified motion track event picture is obtained. Target tracking is performed based on the modified motion track event picture. If the number of targets in the judgment result is less than the preset threshold, target tracking is performed on the motion track event picture, and real-time target detection and tracking are realized. If the number of targets in the judgment result is greater than the preset threshold, all targets are set as a data point set, and there is a hypothesis data point to other data points with a minimum total distance value of center coordinates.

7. The real-time target detection and tracking method of multi-dimensional cooperative remote monitoring according to claim 6, characterized in that, The hypothesis data point is searched, and some outlying data points are excluded. The total distance value is iteratively searched according to a minimum target function, and a plurality of target modified center coordinates are obtained. The distance between the plurality of target modified center coordinates and the center of the picture is calculated until a preset termination condition is met, the total distance value is updated, the remaining data points are determined as targets, and the modified motion track event picture is obtained. If the number of targets in the judgment result is less than the preset threshold, target tracking is performed on the motion track event picture, and real-time target detection and tracking are realized. ​ 8. The real-time target detection and tracking method of multi-dimensional cooperative remote monitoring according to claim 7, characterized in that, ​ If the target quantity in the judgment result is less than a preset threshold, a distance deviation value of a center coordinate of a first target from a center of the picture is calculated; The distance deviation value is subjected to median filtering and cubic spline interpolation processing to obtain an interpolated filtered distance deviation value; The interpolated filtered target deviation and the distance deviation value from the center of the picture are subjected to fuzzification, fuzzy reasoning and defuzzification processing based on a PID fuzzy controller to output a PWM control signal; The gimbal motor is driven to approach the target based on the PWM control signal to realize real-time target detection and tracking.

9. The real-time target detection and tracking method of multi-dimensional cooperative remote monitoring according to claim 8, characterized in that, Further comprising: Implementing instruction transmission between the end-cloud and end-end through the message queue telemetry transmission protocol; Through sending the content in JSON format, the IP address of the device, the reset and four-way movement of the gimbal directly controlled by the device, and the switching of the running mode of the device can be obtained after inputting the preset interface instruction and specific parameters in the command item; A real-time video transmission channel is constructed based on the real-time streaming protocol.

10. A multi-dimensional coordinated remote monitoring real-time target detection and tracking system, characterized in that, Comprising the following modules: A first module for obtaining environmental image data and performing multi-level image data preprocessing to obtain preprocessed environmental image data; A second module for performing target detection processing on the preprocessed environmental image data based on an improved YOLO target detection model and performing correlation matching to obtain the motion trajectory of the numbered target; A third module for performing dynamic caching and analysis on the motion trajectory of the numbered target through a target triggering mechanism to construct a motion trajectory event picture; A fourth module for judging the target quantity in the motion trajectory event picture and performing target tracking according to the judgment result to realize real-time target detection and tracking.

Images