Throwing behavior detection method and system based on adaptive tracking and residual optimization

Abstract

The invention discloses a throwing behavior detection method and system based on adaptive tracking and residual optimization, and relates to the technical field of deep learning and computer vision, and the method comprises the steps: determining a tool detection frame of an operation tool image through a YOLOv8 tool target detection model; outputting a detection track point sequence of the tool track object based on the tool detection frame through a multi-source adaptive confusion perception tracker; performing cubic B-spline interpolation on the detection track point sequence of which the number of track points is greater than or equal to the track point threshold value to determine a spline interpolation track point sequence; after a spline interpolation track point sequence and a detection track point sequence are adopted to construct a residual sequence, gating cyclic residual optimization is carried out to determine a smooth optimization track point sequence; and curve fitting is carried out on the detection track point sequence and the smooth optimization track point sequence with the track point number smaller than the track point threshold value, and a throwing behavior detection result is determined according to the goodness of fit of curve fitting. According to the scheme, the throwing behavior detection and recognition precision is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of deep learning and computer vision, and particularly relates to a throwing behavior detection method and system based on adaptive tracking and residual optimization. BACKGROUND

[0002] With the rapid development of industrial production and construction fields, the wide use of tools and instruments significantly improves work efficiency, but also puts higher requirements on safety management. Especially in high-altitude operation or complex operation environment, if workers throw tools and instruments in violation of regulations, it may cause serious safety accidents such as personnel casualties, equipment damage or production interruption. Therefore, how to efficiently and accurately detect and identify the illegal behavior of throwing tools and instruments has become a technical problem to be solved in industrial safety management.

[0003] In recent years, with the progress of computer vision and artificial intelligence technology, target detection and tracking algorithms have been widely used in behavior analysis field, providing a solution path for automatic identification of illegal behavior. The existing method considers judging the throwing behavior of tools and instruments by tracking the dynamic trajectory of tools and instruments in real time and judging whether it meets the characteristics of throwing behavior. However, the trajectory data of tools and instruments in motion may be missing or noisy due to detection errors, occlusion or light changes, and the fitted trajectory of the thrown object is prone to have the limitation of insufficient trajectory continuity, resulting in poor accuracy of throwing behavior detection and identification. SUMMARY

[0004] The present application provides a throwing behavior detection method and system based on adaptive tracking and residual optimization, which solves the technical problem of poor accuracy of throwing behavior detection and identification caused by the limitation of insufficient trajectory continuity of the fitted trajectory of the thrown object in the existing throwing behavior detection method.

[0005] The present application provides a throwing behavior detection method and system based on adaptive tracking and residual optimization, which solves the technical problem of poor accuracy of throwing behavior detection and identification caused by the limitation of insufficient trajectory continuity of the fitted trajectory of the thrown object in the existing throwing behavior detection method.

[0006] The YOLOv8 tool and instrument target detection model is used for target detection on the current frame of the working tool and instrument image to determine the tool and instrument detection frame.

[0007] Based on each tool and instrument detection frame, the multi-source adaptive confusion perception tracker is used for trajectory tracking of the tool and instrument trajectory object by fusing occlusion perception screening, cost adaptive weighted synthesis and confusion perception decoupling association, and the detection trajectory point sequence of the tool and instrument trajectory object is output.

[0008] The third order B-spline interpolation is used to determine the corresponding spline interpolation trajectory point sequence for the detection trajectory point sequence with the number of trajectory points greater than or equal to the trajectory point threshold.

[0009] By using the spline interpolation trajectory point sequence and the corresponding detection trajectory point sequence, a residual sequence is constructed, and then gated cyclic residual optimization is performed to determine the smooth optimized trajectory point sequence.

[0010] For the detected trajectory point sequence with fewer than the trajectory point threshold and the smoothed optimized trajectory point sequence, the goodness of fit of the curve fitting is determined, and the throwing behavior detection result is determined based on the goodness of fit.

[0011] Furthermore, based on each of the tool detection frames, the trajectory tracking of the tool trajectory object is performed by fusing occlusion perception filtering, cost adaptive weighted synthesis, and confusion perception decoupling association through a multi-source adaptive confusion perception tracker, outputting a sequence of detected trajectory points of the tool trajectory object, including:

[0012] Candidate detection boxes are constructed using the detection boxes of each tool and the associated detection appearance feature vectors, and a candidate set is formed.

[0013] Occlusion-aware non-maximum suppression is performed on the candidate set to divide it into a primary candidate set and a backup candidate set;

[0014] Based on the Kalman filter, predict the trajectory prediction box of the tool trajectory object in the current frame in the trajectory dictionary, and determine the predicted appearance feature vector of each trajectory prediction box.

[0015] The motion cost and appearance cost are calculated by using the main candidate detection boxes and their respective detection appearance feature vectors of the main candidate set, and the trajectory prediction boxes and their respective prediction appearance feature vectors, and then weighted and summed according to adaptive weights to determine the basic comprehensive cost.

[0016] For the main candidate detection boxes that meet the confusion criteria, an confusion penalty term is introduced into the corresponding basic comprehensive cost to determine the confusion comprehensive cost;

[0017] The main candidate detection box and the trajectory prediction box are matched in one stage by the Hungarian algorithm based on the basic comprehensive cost or the confusion comprehensive cost, and the successfully matched box pairs, unmatched main candidate detection boxes and unmatched trajectory prediction boxes are determined.

[0018] If the unmatched trajectory prediction box is not empty, then the unmatched trajectory prediction box is repaired and associated in two stages based on the occlusion comprehensive cost in combination with the backup candidate set, and the unmatched backup candidate detection box is determined and the successfully matched box pair and the unmatched trajectory prediction box are updated.

[0019] Update the trajectory points in the trajectory dictionary according to the coordinates of the center point of the detection box in each successfully matched box pair;

[0020] In the trajectory dictionary, create tool trajectory objects for unmatched primary candidate detection boxes and unmatched backup candidate detection boxes, and update the trajectory points according to the coordinates of the center point of the corresponding detection box.

[0021] Update the number of lost frames of the tool trajectory object to which the unmatched trajectory prediction box belongs in the trajectory dictionary, and delete the tool trajectory object whose number of lost frames meets the frame number threshold in the trajectory dictionary;

[0022] Output the sequence of detected trajectory points for each tool trajectory object in the trajectory dictionary.

[0023] Further, the step of performing occlusion-aware non-maximum suppression on the candidate set and dividing it into a primary candidate set and a backup candidate set includes:

[0024] The candidate detection boxes in the candidate set are filtered by confidence threshold to determine high-confidence candidate detection boxes;

[0025] Select the candidate detection box with the highest confidence as the main box, and calculate the main box IoU value between the main box and the other high-confidence candidate detection boxes.

[0026] The high-confidence candidate detection boxes whose main frame IoU value is less than or equal to the perceived IoU threshold, along with the main frame, are moved into the main candidate set as main candidate detection boxes.

[0027] The high-confidence candidate detection boxes whose main frame IoU value is greater than the perceptual IoU threshold and greater than the occlusion threshold are moved into the backup candidate set as backup candidate detection boxes.

[0028] Further, if the unmatched trajectory prediction box is not empty, then based on the occlusion comprehensive cost, a two-stage repair association is performed on the unmatched trajectory prediction box in conjunction with the backup candidate set to determine the unmatched backup candidate detection box and update the successfully matched box pair and the unmatched trajectory prediction box, including:

[0029] If the unmatched trajectory prediction box is not empty, then calculate the occlusion comprehensive cost with an occlusion penalty term for each unmatched trajectory prediction box and the backup candidate detection box of the backup candidate set.

[0030] The Hungarian algorithm is used to match unmatched trajectory prediction boxes with backup candidate detection boxes based on the comprehensive cost of each occlusion, to determine unmatched backup candidate detection boxes and update successfully matched box pairs and unmatched trajectory prediction boxes.

[0031] The process of determining the overall cost of occlusion includes:

[0032]

[0033]

[0034] In the formula, To cover the overall costs, For the price of sports, For the sake of appearance, For tool trajectory objects With detection box This pair of pairings has an occlusion penalty. For the repair correlation weight of motion cost, The repair association weight is the cost of appearance. To repair the correlation weight of the occlusion penalty item, For the Sigmoid function, This is the Sigmoid steepness coefficient. For the first The trajectory prediction box of the first tool trajectory object and the first IoU value between each detection box For the occlusion threshold, It is the numerical stability constant. For the first Confidence of each detection box, The confidence level of the main frame. This is the appearance adjustment factor. For appearance similarity.

[0035] Further, the step of constructing a residual sequence using spline interpolation trajectory point sequences and corresponding detection trajectory point sequences, followed by gated cyclic residual optimization to determine a smoothed optimized trajectory point sequence, includes:

[0036] The trajectory residuals are determined by calculating the difference between the trajectory points one by one using the associated spline interpolation trajectory point sequence and the detection trajectory point sequence, and the corresponding trajectory residual sequence is formed.

[0037] Each trajectory residual sequence is transformed to a uniform scale to determine the standardized residual sequence.

[0038] Calculate the first-order and second-order differences of the trajectory residuals in each of the trajectory residual sequences;

[0039] The standardized residuals in each of the standardized residual sequences are concatenated with the associated first-order and second-order differences to form a time-series input vector, thus forming the corresponding time-series residual sequence.

[0040] The corrected residual sequence of each of the time-series residual sequences is output through the gated loop unit;

[0041] Average smoothing is performed on the corrected residual sequence that does not meet the residual jump detection condition to determine the smoothed corrected residual sequence;

[0042] For the corrected residual sequence that meets the residual jump detection condition, after eliminating abnormal jumps and updating the corrected residual sequence, average smoothing is performed to determine the smoothed corrected residual sequence.

[0043] Based on the smoothed correction residual sequences, the associated spline interpolation trajectory point sequences are compensated and corrected to output a smoothed optimized trajectory point sequence.

[0044] Furthermore, the category labels of the tool detection boxes include "tool in hand" and "tool in the air"; the process of constructing candidate detection boxes using each of the tool detection boxes and the associated detection appearance feature vectors, and forming a candidate set, includes:

[0045] Based on the category labels of each tool detection box, category filtering is performed, and tool detection boxes with the category label "tool in the air" are determined as target tool detection boxes;

[0046] The detection boxes of each target tool and the corresponding extracted detection appearance feature vectors are used to construct candidate detection boxes;

[0047] A candidate set is formed using the candidate detection boxes described above.

[0048] Furthermore, the curve fitting includes parabolic fitting and linear fitting.

[0049] Furthermore, the process of determining the adaptive weights includes:

[0050]

[0051] In the formula, As the weight of the cost of movement, For the Sigmoid function, For location prediction residuals, The coordinates of the center point of the detection box. The coordinates of the center point of the trajectory prediction box. This represents the velocity change of the tool trajectory object between adjacent frames. It is the L2 norm. The weights for the location prediction residuals, As the weight of the change in velocity, The weight of appearance cost.

[0052] Furthermore, the conditions for easy confusion include an IoU value between the main candidate detection box and the trajectory prediction box that is greater than the confusion IoU threshold and an appearance similarity that is greater than the appearance threshold.

[0053] The process of determining the overall cost of confusion includes:

[0054]

[0055]

[0056] In the formula, To obscure the overall cost, Based on comprehensive costs, For the first The trajectory object of the tool and the first Obfuscation penalty between detection boxes The penalty coefficient is... For the Sigmoid function, This is the Sigmoid steepness coefficient. For the first The trajectory prediction box of the first tool trajectory object and the first IoU value between each detection box To obfuscation threshold, For appearance similarity.

[0057] A second aspect of the present invention provides a throwing behavior detection system based on adaptive tracking and residual optimization, comprising:

[0058] The target detection module is used to perform target detection on the working tool image of the current frame using the YOLOv8 tool target detection model, and determine the tool detection box;

[0059] The target tracking module is used to track the trajectory of the tool trajectory object based on each of the tool detection boxes by fusing occlusion perception filtering, cost adaptive weighted synthesis and confusion perception decoupling association through a multi-source adaptive confusion perception tracker, and outputs the detection trajectory point sequence of the tool trajectory object;

[0060] The trajectory point interpolation module is used to determine the corresponding spline interpolated trajectory point sequence by cubic B-spline interpolation for detected trajectory point sequences where the number of trajectory points is greater than or equal to the trajectory point threshold.

[0061] The trajectory optimization and smoothing module is used to construct a residual sequence by using spline interpolation trajectory point sequence and corresponding detection trajectory point sequence, and then perform gated cyclic residual optimization to determine the smoothed optimized trajectory point sequence.

[0062] The throwing behavior analysis module is used to determine the goodness of fit of the curve fitting for the detected trajectory point sequence with fewer than a trajectory point threshold and the smoothed optimized trajectory point sequence, and to determine the throwing behavior detection result based on the goodness of fit.

[0063] As can be seen from the above technical solutions, the present invention has the following advantages:

[0064] The above-described solution of the present invention provides a throwing behavior detection method based on adaptive tracking and residual optimization, comprising: performing target detection on the working tool image of the current frame using a YOLOv8 tool target detection model to determine tool detection boxes; based on each tool detection box, performing trajectory tracking of the tool trajectory object by fusing occlusion perception filtering, cost adaptive weighted synthesis, and confusion perception decoupling association through a multi-source adaptive confusion perception tracker, and outputting a sequence of detected trajectory points of the tool trajectory object; using cubic B-spline interpolation to determine the corresponding spline interpolated trajectory point sequence for the detected trajectory point sequence with a number of trajectory points greater than or equal to a trajectory point threshold; using the spline interpolated trajectory point sequence and the corresponding detected trajectory point sequence to construct a residual sequence and then performing gated cyclic residual optimization to determine a smoothed optimized trajectory point sequence; determining the goodness of fit of the curve fitting for the detected trajectory point sequence with a number of trajectory points less than the trajectory point threshold and the smoothed optimized trajectory point sequence, and determining the throwing behavior detection result based on the goodness of fit. Based on the above scheme, a multi-source adaptive confusion perception tracker is constructed to achieve stable tracking and cross-frame trajectory recording in complex scenarios. Cubic B-spline interpolation is used to supplement trajectory points, and a gated recurrent unit is introduced for residual learning and noise correction, which effectively reduces trajectory noise interference and ensures the smoothness and continuity of the trajectory. Finally, the goodness of fit of the trajectory curve fitting is used to identify whether there is throwing behavior, which improves the accuracy of detecting throwing tools in complex industrial scenarios. Attached Figure Description

[0065] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0066] Figure 1 The flowchart illustrates the steps of a throwing behavior detection method based on adaptive tracking and residual optimization provided in Embodiment 1 of the present invention.

[0067] Figure 2 This is a schematic diagram illustrating the labeling of tool category tags provided in Embodiment 1 of the present invention;

[0068] Figure 3 The flowchart of the MACA-Tracker algorithm provided in Embodiment 1 of the present invention is shown below;

[0069] Figure 4 This is a schematic diagram of the gated cyclic residual optimization process provided in Embodiment 1 of the present invention;

[0070] Figure 5 This is a comparison chart of the optimization effects provided in Embodiment 1 of the present invention;

[0071] Figure 6 This is a structural block diagram of a throwing behavior detection system based on adaptive tracking and residual optimization provided in Embodiment 2 of the present invention. Detailed Implementation

[0072] This invention provides a throwing behavior detection method and system based on adaptive tracking and residual optimization, which addresses the limitation of existing throwing behavior detection methods in that the fitted trajectory of the thrown object is prone to insufficient trajectory continuity, resulting in poor throwing behavior detection and recognition accuracy.

[0073] To make the objectives, features, and advantages of this invention more apparent and understandable, the technical solutions of the embodiments of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of this invention, and not all embodiments. Based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0074] Please see Figure 1 The present invention provides a throwing behavior detection method based on adaptive tracking and residual optimization, comprising:

[0075] Step 101: Perform target detection on the tool image of the current frame using the YOLOv8 tool target detection model to determine the tool detection box.

[0076] It should be noted that this embodiment uses a field tool training dataset to train the YOLOv8 network. After training, a YOLOv8 tool target detection model is obtained. Each frame of the work tool image captured in the field operation scene is input into the YOLOv8 tool target detection model for target detection, determining the detection results of each tool trajectory object. A tool trajectory object refers to each tool being treated as an independent trajectory tracking object. The tools in the image are identified through target detection and marked in the form of bounding boxes. The obtained detection results are called tool detection boxes. Each tool detection box carries at least the detection box position (X-axis coordinates of the upper left corner of the detection box). The Y-axis coordinate of the top left corner of the detection box Detection box width Detection frame height ), confidence level and category labels Information such as...

[0077] Understandably, in the YOLOv8 tool and equipment target detection model, non-maximum suppression (NMS) is applied to the original detection results output by the model to initially remove redundant boxes with high overlap (IoU), ensuring that each real target retains at least one optimal detection box. Based on the target detection confidence threshold (e.g., 0.3), low-confidence original detection boxes are filtered out, and only tool and equipment detection boxes with a confidence level not lower than the preset threshold are retained to reduce false detections.

[0078] In one specific embodiment of this example, the category labels of the tool detection box include "tool in hand" and "tool in the air".

[0079] It should be noted that the LabelImg annotation tool was used to perform dual-state annotation and data augmentation on the training tool images collected from the on-site monitoring, constructing an on-site tool training dataset; the category labels of the on-site tool training dataset include "tool in hand" and "tool in the air" (e.g., ...). Figure 2 As shown in the figure, dual-state annotation avoids misjudging the trajectory of a worker holding a tool as a parabolic curve when the worker is working normally, thus preventing it from being misidentified as throwing the tool. By combining dual-state annotation with the high-precision detection capabilities of YOLOv8, the normal use and throwing states can be effectively distinguished from the throwing state from the source, which helps to reduce the risk of misjudging the state of the tool. At the same time, the data augmentation used during training can include mosaic augmentation, rotation, translation, scaling and HSV augmentation to enhance data diversity and simulate the state of the tool under different perspectives and different lighting conditions.

[0080] Step 102: Based on the detection frames of each tool, track the trajectory of the tool trajectory object by fusing occlusion perception filtering, cost adaptive weighted synthesis and confusion perception decoupling association through a multi-source adaptive confusion perception tracker, and output the detection trajectory point sequence of the tool trajectory object.

[0081] Occlusion perception and filtering refers to the perception and filtering of situations where a tool is occluded by other tools.

[0082] Cost adaptive weighted synthesis refers to the adaptive adjustment of weights during the weighted synthesis operation when calculating the matching cost.

[0083] Confusion perception decoupling association refers to constructing a matching cost separately for targets that are prone to confusion and mismatch during perception and then performing matching association.

[0084] A detection trajectory point sequence refers to a sequence of positional information of points representing the location of the detection box (called detection trajectory points) extracted from the detection box. A detection trajectory point sequence includes the positional information of multiple detection trajectory points; typically, the center point of the detection box can be used as the detection trajectory point.

[0085] It should be noted that this embodiment constructs a multi-source adaptive confusion-aware tracker (MACA-Tracker) based on the DeepSORT algorithm framework to track the detection results. Stable cross-frame matching is achieved through occlusion-aware filtering, cost adaptive weighted synthesis and confusion-aware decoupling association, and the detection trajectory point sequence is output.

[0086] In one specific implementation of this embodiment, such as Figure 3 As shown, step 102 includes the following sub-steps:

[0087] S11. Construct candidate detection boxes using the detection boxes of each tool and the associated detection appearance feature vectors, and form a candidate set.

[0088] In a more specific embodiment of this example, sub-step S11 includes:

[0089] S1101. Perform category filtering based on the category labels of each tool detection box, and determine the tool detection box with the category label "tool in the air" as the target tool detection box.

[0090] S1102. Construct candidate detection boxes from the detection boxes of each target tool and the corresponding extracted detection appearance feature vectors;

[0091] S1103. A candidate set is formed using each candidate detection box.

[0092] It should be noted that after receiving the detection results from the YOLOv8 tool and equipment target detection model, a candidate set is constructed. , For the first Each candidate bounding box in the candidate set carries at least its location (X-axis coordinate of the top-left corner of the bounding box). The Y-axis coordinate of the top left corner of the detection box Detection box width Detection frame height ), confidence level Category tags and detection of appearance feature vectors These candidate detection boxes serve as input to the multi-source adaptive confusion-aware tracker for subsequent matching and association. It can be understood that the detection appearance feature vector is the appearance feature of the tool trajectory object in the tool detection box. It can be obtained by extracting the appearance of the image of the area where the tool detection box is located through a preset appearance feature extraction network. For details, please refer to the existing technology, which will not be elaborated here.

[0093] Furthermore, if the YOLOv8 tool target detection model uses dual-state labeled category tags during training, the output detection results include tool detection boxes with the category tag "tool in hand" and the category tag "tool in the air". First, it is necessary to filter according to the category tag, and only retain the tool detection boxes with the category "tool in the air" as target tool detection boxes, and then construct a candidate set.

[0094] S12. Perform occlusion-aware nonmaximum suppression on the candidate set and divide it into a primary candidate set and a backup candidate set.

[0095] In a more specific embodiment of this example, sub-step S12 includes:

[0096] S1201. Filter the candidate detection boxes in the candidate set using a confidence threshold to determine high-confidence candidate detection boxes;

[0097] S1202. Select the candidate detection box with the highest confidence as the main box, and calculate the main box IoU value between the high confidence candidate detection boxes other than the main box and the main box;

[0098] S1203: Move the high-confidence candidate detection boxes whose main frame IoU value is less than or equal to the perceived IoU threshold, along with the main frame, into the main candidate set as the main candidate detection boxes;

[0099] S1204. Move high-confidence candidate detection boxes whose main frame IoU value is greater than the perceptual IoU threshold and greater than the occlusion threshold into the backup candidate set as backup candidate detection boxes.

[0100] It should be noted that when performing occlusion-aware nonmaximum suppression (ONMS) on the candidate set, a lower confidence limit is first set. By filtering out low-confidence candidate bounding boxes, a high-confidence candidate bounding box set is selected from the candidate set. Sort the high-confidence candidate detection boxes from highest to lowest confidence, determine the highest-confidence candidate detection box as the main box, and calculate the IoU value between the main box and the other high-confidence candidate detection boxes to obtain the main box IoU value; set a perceptual IoU threshold to determine whether the overlap is too high. To suppress candidate boxes with excessive overlap, if the IoU value of the main box is less than or equal to the perceptual IoU threshold, it is moved to the main candidate set. Simultaneously, the main bounding box is also moved into the main candidate set. This retains high-confidence candidate bounding boxes that do not significantly overlap within the main candidate set. Furthermore, if the IoU value of the main bounding box is greater than... If the high-confidence candidate detection box is suppressed by the main box, an occlusion threshold is set to perceive the degree of occlusion. For a suppressed box, if its IoU value with the main box is less than or equal to... If the IoU value of the main frame is greater than 1, it will be directly removed. If it is determined to be "obscured by the main frame" rather than redundant, it will be moved to the backup candidate set. The primary candidate set and the backup candidate set are used for the subsequent first-stage direct matching and the second-stage occlusion repair matching, respectively.

[0101] S13. Based on the Kalman filter, predict the trajectory prediction box of the tool trajectory object in the trajectory dictionary in the current frame, and determine the predicted appearance feature vector of each trajectory prediction box.

[0102] Kalman filtering can predict the most likely position and velocity of a target in the current frame based on the target's historical motion state; for details, please refer to existing technologies.

[0103] A trajectory dictionary refers to a structured storage of trajectory items for each tool trajectory object, using the ID assigned to the tool trajectory object as the key. The trajectory item can include information such as the detection box position, appearance feature vector, confidence level, status flag, number of lost frames, and trajectory point sequence.

[0104] The trajectory prediction box refers to the bounding box in the current frame where the predicted tool trajectory object will appear, which corresponds to the tool detection box.

[0105] The predicted appearance feature vector refers to the appearance features of the tool trajectory object in the trajectory prediction box, which corresponds to the detected appearance feature vector.

[0106] It should be noted that this implementation uses a dictionary structure to manage all tool trajectory objects, with ID as the index, to achieve fast query, update and deletion; the Kalman filter algorithm is used to predict the trajectory prediction box of the tool trajectory object stored in the trajectory dictionary in the current frame, and the predicted appearance feature vector corresponding to the trajectory prediction box is determined.

[0107] S14. Using the main candidate detection boxes and their respective detection appearance feature vectors from the main candidate set, and the prediction boxes and their respective prediction appearance feature vectors from each trajectory prediction box, calculate the motion cost and appearance cost, and sum them according to adaptive weights to determine the basic comprehensive cost.

[0108] It should be noted that when performing trajectory matching, for any tool trajectory object With the main candidate set The main candidate detection box in Motion cost and appearance cost are calculated separately and weighted by adaptive weights to obtain the basic comprehensive cost, which is used for trajectory matching with candidates. The process of determining the basic comprehensive cost includes:

[0109]

[0110] In the formula, Based on comprehensive costs, As the weight of the cost of movement, For the price of sports, As a weight for appearance cost, For the price of sports, For the first A tool trajectory object For the first One detection box;

[0111] The process of determining the cost of motion includes:

[0112]

[0113] In the formula, The coordinates of the center point of the detection box. , The coordinates of the center point of the trajectory prediction box. For transpose, The trajectory prediction error covariance matrix;

[0114] The process of determining the appearance cost includes:

[0115]

[0116] In the formula, For the first The appearance feature vector of a tool trajectory object. For the first The appearance feature vector of each detection box The numerator is the L2 norm; the numerator is the inner product of the features, and the denominator is the magnitude of the eigenvector.

[0117] In a more specific implementation of this embodiment, the process of determining the adaptive weights includes:

[0118]

[0119] In the formula, For the Sigmoid function, For location prediction residuals, This represents the velocity change of the tool trajectory object between adjacent frames. The weights for the location prediction residuals, This represents the weight of the change in velocity.

[0120] S15. For the main candidate detection boxes that meet the confusion condition, introduce a confusion penalty term into the basic comprehensive cost to determine the confusion comprehensive cost.

[0121] It should be noted that, based on the basic comprehensive cost Confusion-aware decoupling association is used to match trajectories with candidates. Since multiple targets overlap or have similar appearances, mismatches are likely to occur. Therefore, a condition is set to determine whether the detection box and the prediction box are likely to be confused, namely the confusion condition. For the main candidate detection box that meets the confusion condition, a confusion penalty term is introduced on the basis of the basic comprehensive cost to improve the discriminativeness and reduce mismatches.

[0122] In a more specific implementation of this embodiment, the confusion criteria include an IoU value between the main candidate detection box and the trajectory prediction box that is greater than a confusion IoU threshold and an appearance similarity value that is greater than an appearance threshold. This applies to any tool trajectory object. With the main candidate detection boxes in the main candidate set When the IoU value between the two is greater than the preset obfuscation IoU threshold Furthermore, if the appearance similarity exceeds a preset appearance threshold, the main candidate detection box is deemed likely to be confused with the trajectory prediction box, thus meeting the confusion condition. It is understandable that appearance similarity can be determined by measuring the similarity distance between the detected appearance feature vector and the predicted appearance feature vector; more specifically, it can also be determined by combining appearance cost.

[0123] In a more specific implementation of this embodiment, the process for determining the obfuscation cost includes:

[0124]

[0125] In the formula, To obscure the overall cost, Based on comprehensive costs, For the first The trajectory object of the tool and the first Obfuscation penalty between detection boxes This is the penalty coefficient;

[0126] The process for determining the confusion penalty item includes:

[0127]

[0128] In the formula, For the Sigmoid function, This is the Sigmoid steepness coefficient. For the first The trajectory prediction box of the first tool trajectory object and the first IoU value between each detection box The obfuscation threshold is set (e.g., 0.5). For appearance similarity;

[0129] The process of determining appearance similarity includes:

[0130]

[0131] S16. Using the Hungarian algorithm, a one-stage matching process is performed between the main candidate detection box and the trajectory prediction box based on each basic comprehensive cost or each confusion comprehensive cost to determine the successfully matched box pairs, the unmatched main candidate detection boxes, and the unmatched trajectory prediction boxes.

[0132] It should be noted that for main candidate detection boxes that meet the easy-to-confusion condition, the confusion comprehensive cost is used to match them using the Hungarian algorithm. For other main candidate detection boxes that do not meet the easy-to-confusion condition, the corresponding basic comprehensive cost is used to match them using the Hungarian algorithm. This completes the first-stage matching based on the main candidate set, and determines the successfully matched box pairs and the unmatched unmatched main candidate detection boxes and unmatched trajectory prediction boxes. For the specific principle of the Hungarian algorithm, please refer to the existing technology.

[0133] S17. If the unmatched trajectory prediction box is not empty, then combine the backup candidate set with the occlusion comprehensive cost to perform a two-stage repair association on the unmatched trajectory prediction box, determine the unmatched backup candidate detection box and update the successfully matched box pair and the unmatched trajectory prediction box.

[0134] It should be noted that if there are unmatched trajectory prediction boxes after the first stage of matching, a second stage of repair association is performed with the backup candidate set to determine new successfully matched box pairs, new unmatched trajectory prediction boxes, and unmatched backup candidate detection boxes that were not successfully matched.

[0135] In a more specific embodiment of this example, sub-step S17 includes:

[0136] S1701. If the unmatched trajectory prediction box is not empty, calculate the occlusion comprehensive cost with occlusion penalty term introduced for each unmatched trajectory prediction box and the backup candidate detection box of the backup candidate set.

[0137] S1702. Using the Hungarian algorithm, based on the comprehensive cost of each occlusion, match the unmatched trajectory prediction boxes with the backup candidate detection boxes, determine the unmatched backup candidate detection boxes, and update the successfully matched box pairs and the unmatched trajectory prediction boxes.

[0138] It should be noted that for unmatched trajectory prediction boxes that fail to match, in order to avoid trajectory interruption due to occlusion, the occlusion comprehensive cost with an occlusion penalty term is used to determine the matching result through the Hungarian algorithm.

[0139] Specifically, the process of determining the overall cost of occlusion includes:

[0140]

[0141] In the formula, To cover the overall costs, For the price of sports, For the sake of appearance, For tool trajectory objects With detection box This pair of matching relationships has an occlusion penalty. For the repair correlation weight of motion cost, The repair association weight is the cost of appearance. Repair the associated weights for the occlusion penalty item; , , ;

[0142] The process for determining the occlusion penalty includes:

[0143]

[0144] In the formula, For the Sigmoid function, This is the Sigmoid steepness coefficient. For the first The trajectory prediction box of the first tool trajectory object and the first IoU value between each detection box For the occlusion threshold, It is a numerical stability constant (to prevent division by zero errors). For the first Confidence of each detection box, The confidence level of the main frame. This is the appearance adjustment factor. For appearance similarity.

[0145] S18. Update the trajectory points in the trajectory dictionary according to the coordinates of the center point of the detection box in each successfully matched box pair.

[0146] S19. Create tool trajectory objects for the unmatched primary candidate detection boxes and unmatched backup candidate detection boxes in the trajectory dictionary, and update the trajectory points according to the coordinates of the center point of the corresponding detection box.

[0147] S20. Update the number of lost frames of the tool trajectory object to which the unmatched trajectory prediction box belongs in the trajectory dictionary, and delete the tool trajectory object whose number of lost frames meets the frame number threshold in the trajectory dictionary.

[0148] S21. Output the sequence of detected trajectory points for each tool trajectory object in the trajectory dictionary.

[0149] It should be noted that the tracker completes cross-frame association based on each comprehensive cost matrix. For successfully matched detection boxes, the detection trajectory points of their corresponding trajectories are updated. For unmatched detection boxes, a new trajectory object is created and assigned a unique ID. The trajectory entry is established in the trajectory dictionary using the ID as the key, initializing information such as the detection box position, appearance features, confidence level, status flag, number of lost frames, and trajectory point sequence. In general, for all successfully matched and newly created trajectories (i.e., all "airborne" tools tracked in this frame), the center point coordinates of their detection boxes are calculated, and these center point coordinates, along with the current frame number, are recorded in the trajectory list with the corresponding ID. For existing unmatched trajectories, they are marked as temporarily lost, and their number of lost frames is incremented. When the number of consecutive lost frames of a trajectory exceeds a preset frame threshold (e.g., 30 frames), the trajectory object is deleted from the trajectory dictionary. Based on the trajectory dictionary, a new dictionary entry is created when a new detection fails to match, the trajectory entry information is updated when a detection matches successfully, and the trajectory is automatically deleted when the number of lost frames exceeds the threshold, thus achieving dynamic trajectory management.

[0150] Step 103: For the detected trajectory point sequence with a number of trajectory points greater than or equal to the trajectory point threshold, cubic B-spline interpolation is used to determine the corresponding spline interpolation trajectory point sequence.

[0151] It should be noted that this embodiment considers using cubic B-spline interpolation to supplement trajectory points in order to better preserve the trajectory curvature characteristics when the tool is thrown, and to help solve the technical difficulty of discontinuous trajectory data. Therefore, before performing spline interpolation, the number of trajectory points in each detected trajectory point sequence in the current trajectory dictionary is determined.

[0152] If the number of trajectory points in the dictionary is less than 4, it is determined to be an invalid interpolation condition, and the original discrete trajectory points are returned directly.

[0153] A trajectory parameter model is constructed using cubic B-spline interpolation on all valid detection trajectory point sequences, and the model is expanded using recursive basis functions:

[0154]

[0155] In the formula, A parameterized continuous curve constructed from B-spline basis functions (used to describe the motion trajectory of tools). For control point index, The maximum value of the control point index. The basis functions are p-order B-spline functions (in this embodiment, p=3, i.e., cubic B-spline). For the first control point in the sequence Each control point (containing coordinate information in both the X and Y axes, serving as the basic reference point for constructing the trajectory curve) The normalized parameter variable (with a value range of [0,1], used to traverse the entire trajectory curve) is used to construct a parameterized continuous curve that passes through all valid trajectory points using this formula;

[0156] In the normalized parameter domain Within, an equidistant sampling strategy is used to generate 100 uniformly distributed new parameter nodes to ensure the consistency of trajectory point distribution density;

[0157] Based on the spline function evaluation algorithm, the coordinates of the trajectory points corresponding to the new parameter nodes are calculated using the established parameter model, i.e.:

[0158]

[0159] In the formula, For the first One parameter node, For parameter nodes The corresponding X-axis coordinates of the trajectory points For the first X-axis coordinates of each control point For parameter nodes The corresponding Y-axis coordinates of the trajectory points For the first The Y-axis coordinates of each control point are obtained; by calculating the values ​​of 100 equidistant sampling nodes in the normalized parameter domain, a smooth and continuous motion trajectory consisting of 100 uniformly distributed trajectory points is obtained.

[0160] Step 104: Using the spline interpolation trajectory point sequence and the corresponding detection trajectory point sequence, construct the residual sequence and then perform gated cyclic residual optimization to determine the smoothed optimized trajectory point sequence.

[0161] In one specific embodiment of this example, step 104 includes the following sub-steps:

[0162] The trajectory residuals are determined by calculating the difference between the trajectory points one by one using the associated spline interpolation trajectory point sequence and the detection trajectory point sequence, and the corresponding trajectory residual sequence is formed.

[0163] Each trajectory residual sequence is transformed to a uniform scale to determine the standardized residual sequence;

[0164] Calculate the first and second differences of the trajectory residuals in each trajectory residual sequence;

[0165] The standardized residuals in each standardized residual sequence are concatenated with the associated first-order and second-order differences to form a time-series input vector, thus forming the corresponding time-series residual sequence.

[0166] The corrected residual sequence of each time-series residual sequence is output through the gated loop unit;

[0167] Average smoothing is performed on the corrected residual sequence that does not meet the residual jump detection condition to determine the smoothed corrected residual sequence;

[0168] For the corrected residual sequence that meets the residual jump detection condition, after eliminating abnormal jumps and updating the corrected residual sequence, average smoothing is performed to determine the smoothed corrected residual sequence.

[0169] The associated spline interpolation trajectory point sequence is compensated and corrected based on each smoothed correction residual sequence, and a smoothed optimized trajectory point sequence is output.

[0170] It should be noted that, as Figure 4 As shown, this embodiment proposes a gated cyclic residual optimization mechanism. A residual sequence is constructed from the spline interpolation trajectory point sequence and the detection trajectory point sequence, and then input into the gated cyclic unit for timing correction, outputting the optimized trajectory point sequence.

[0171] First, calculate the spline interpolation trajectory points of the spline interpolation trajectory point sequence. Detection trajectory points with the detection trajectory point sequence The difference is the trajectory residual, from which the trajectory residual sequence of each tool trajectory object is obtained; the process of determining the trajectory residual includes:

[0172]

[0173] In the formula, For a moment trajectory residual, For a moment lateral residuals For a moment Longitudinal residuals For a moment spline interpolation trajectory points, For a moment The detection trajectory points;

[0174] Next, the trajectory residual sequence is converted to a uniform scale to avoid the impact of numerical differences on the training and inference of subsequent neural network models; for example, the Z-score normalization method can be used.

[0175]

[0176] In the formula, For a moment Standardized transverse residuals The mean of the horizontal residuals. The standard deviation of the horizontal residuals. For a moment Standardized longitudinal residuals The mean of the longitudinal residuals. The standard deviation of the longitudinal residuals;

[0177] Then, the first-order and second-order differences are calculated to enhance the dynamic characteristics of the time series. The calculation process includes:

[0178]

[0179] In the formula, For a moment The change in trajectory residuals at time [time] The first difference of the trajectory residual, For a moment trajectory residual, For a moment trajectory residual, For a moment The trajectory residual acceleration characteristics at time t The second difference of the trajectory residual, For a moment The trajectory residual change at time The second difference of the trajectory residual;

[0180] Next, ( , , , The data is concatenated into a temporal input vector and a complete feature sequence is formed. The temporal residual sequence of each tool trajectory object is obtained and input into a gated recurrent unit (GRU) for feature processing. The corrected residual is then output. The corrected residual sequence is used to correct spline trajectory deviations;

[0181] To prevent sudden and abnormal jumps in the model output, residual jump detection conditions are set:

[0182]

[0183] In the formula, For a moment Corrected residuals For a moment Corrected residuals The threshold for abrupt changes is set (e.g., 5 pixels). If the condition is met, aberrant abrupt changes need to be eliminated. In one possible implementation, the process of eliminating aberrant abrupt changes includes adjusting the corrected residual from the next time step. Replace with the corrected residual from the previous time step. Subsequently, for the corrected residual sequences that do not meet the residual jump detection conditions and the corrected residual sequences newly generated after eliminating abnormal jumps, average smoothing is performed to determine the smoothed corrected residual sequence composed of the smoothed corrected residuals; the process of determining the smoothed corrected residuals includes:

[0184]

[0185] In the formula, For a moment Smoothing correction residuals The size of the sliding opening (e.g., 5 frames). For the data frame index within the sliding port, For a moment Corrected residuals;

[0186] The spline interpolation trajectory points are compensated and corrected using smoothing correction residuals, and a sequence of smoothed optimized trajectory points is output. The process of determining the smoothed optimized trajectory points includes:

[0187]

[0188] In the formula, For a moment Smoothed optimized trajectory points.

[0189] Step 105: Determine the goodness of fit of the curve fitting for the detection trajectory point sequence with fewer than the trajectory point threshold and the smoothed optimized trajectory point sequence, and determine the throwing behavior detection result based on the goodness of fit.

[0190] It should be noted that curve fitting is performed on the detection trajectory point sequence that does not meet the spline interpolation optimization and the smoothing optimization trajectory point sequence, and the goodness of fit of the fitted curve trajectory to the parabola or straight line is calculated. Here, the goodness of fit refers to the degree of fit of the regression curve to the observed values.

[0191] To better reflect the actual work scenario, curve fitting includes parabolic fitting and linear fitting; parabolic fitting can use quadratic polynomial fitting to approximate data points, while curve data points... The formulas for fitting a parabola include: , The fitting coefficients can be determined using the least squares method;

[0192] In one implementation, the statistic for measuring goodness of fit is the coefficient of determination (also known as the coefficient of certainty). ,when The closer the value is to 1, the better the fit. The specific calculation process includes:

[0193]

[0194] In the formula, For the first One set of data to be fitted. For the first One fitted data, The mean of all data to be fitted;

[0195] When the fitting result If the value is greater than the set fitting threshold, the trajectory is considered to be either a parabola or a straight line. In other words, the throwing behavior detection result indicates that there may be a violation of throwing tools on site. The trajectory of the tool is marked as "throwing" and the tool's ID is recorded. Otherwise, the trajectory of the tool is marked as "non-throwing".

[0196] Furthermore, the system draws the status (whether it is being thrown) and trajectory of tools from the dictionary in the current video frame. For example, trajectories that conform to parabolic or straight lines are drawn in light blue, while trajectories that do not conform to the throwing status are drawn in white. If no work end signal is received after drawing, the system continues to read the next frame to perform target detection and tracking tasks to determine whether the tool is being thrown. When the tool is detected to be in a "throwing" state, it can be used to automatically generate an alarm signal, record relevant information (such as tool ID, throwing time, trajectory data, and video clips), and notify management personnel to take appropriate measures through a visual interface or alarm device (such as sound prompts and flashing lights).

[0197] In the embodiments of the present invention, see Figure 5As shown, the optimization results of this embodiment are not only reflected in the improvement of trajectory smoothness, but also in the restoration of trajectory continuity. It makes up for the trajectory breakage caused by occlusion, detection loss and other reasons, so that a complete and continuous trajectory line can be formed, thereby more accurately fitting the real motion trajectory. The two originally broken trajectory segments are correctly connected into a continuous trajectory after optimization, which makes it easier to identify that this is the complete motion trajectory of the same tool, so as to more accurately determine whether there is a throwing behavior by the goodness of fit. The throwing behavior detection method based on adaptive tracking and residual optimization in this embodiment has the following aspects: First, by constructing dual-state category labels of "tool in hand" and "tool in the air," the risk of misjudging the tool's state is reduced from the source. Second, a multi-source adaptive confusion-aware tracker is constructed to achieve stable tracking and cross-frame trajectory recording in complex scenes. Specifically, based on occlusion-aware non-maximum suppression, adaptive weighting of motion cost and appearance cost is combined, and a confusion penalty term and a two-stage repair association mechanism are introduced for easily confused scenes to effectively alleviate ID switching and trajectory interruption problems caused by occlusion, intersection, and appearance similarity, thereby improving the tracking robustness in complex scenes. Third, cubic B-spline interpolation is used to supplement... The method constructs a trajectory residual sequence by filling trajectory points, and enhances temporal features through standardization and first / second-order difference. A gated recurrent unit is then introduced for residual learning and denoising correction, effectively reducing trajectory noise interference. Simultaneously, residual jump detection and moving average smoothing are combined to obtain a smooth, optimized trajectory point sequence that balances trend and detail, significantly improving trajectory continuity and noise resistance. Fourthly, the method judges throwing behavior based on the comparison of the goodness of fit and fitting threshold of trajectory curves fitted with straight lines and parabolas, accommodating both typical parabolic motion and near-linear motion scenarios, thus improving the accuracy and stability of throwing behavior recognition. Therefore, this method effectively solves the problems of trajectory discontinuity, easy tracking confusion, and unstable fitting judgment in existing technologies. It improves the accuracy and real-time performance of throwing tool behavior detection in complex industrial scenarios, enabling real-time detection and proactive early warning of illegal throwing tool behavior in industrial settings, providing reliable technical support for the intelligent and automated management of industrial safety.

[0198] Please see Figure 6 The present invention provides a throwing behavior detection system based on adaptive tracking and residual optimization, comprising:

[0199] The target detection module 601 is used to perform target detection on the working tool image of the current frame using the YOLOv8 tool target detection model, and determine the tool detection box;

[0200] The target tracking module 602 is used to track the trajectory of tool trajectory objects based on each tool detection box by fusing occlusion perception filtering, cost adaptive weighted synthesis and confusion perception decoupling association through a multi-source adaptive confusion perception tracker, and outputs the detection trajectory point sequence of the tool trajectory objects;

[0201] The trajectory point interpolation module 603 is used to determine the corresponding spline interpolated trajectory point sequence by cubic B-spline interpolation for the detected trajectory point sequence where the number of trajectory points is greater than or equal to the trajectory point threshold.

[0202] The trajectory optimization and smoothing module 604 is used to construct a residual sequence by using spline interpolation trajectory point sequence and corresponding detection trajectory point sequence, and then perform gated cyclic residual optimization to determine the smoothed optimized trajectory point sequence.

[0203] The throwing behavior analysis module 605 is used to determine the goodness of fit of the curve fitting for the detection trajectory point sequence with fewer than the trajectory point threshold and the smoothed optimized trajectory point sequence, and to determine the throwing behavior detection result based on the goodness of fit.

[0204] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working process of the system and modules described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0205] In the embodiments provided in this application, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0206] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network modules. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0207] Furthermore, the functional modules in the various embodiments of the present invention can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0208] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A throwing behavior detection method based on adaptive tracking and residual optimization, characterized in that, include: The YOLOv8 tool target detection model is used to perform target detection on the tool image of the current frame and determine the tool detection box; Based on the tool detection frames, the trajectory of the tool trajectory object is tracked by a multi-source adaptive confusion perception tracker that integrates occlusion perception filtering, cost adaptive weighted synthesis and confusion perception decoupling association, and outputs the detection trajectory point sequence of the tool trajectory object. For the detected trajectory point sequence with a number of trajectory points greater than or equal to the trajectory point threshold, cubic B-spline interpolation is used to determine the corresponding spline interpolated trajectory point sequence; By using the spline interpolation trajectory point sequence and the corresponding detection trajectory point sequence, a residual sequence is constructed, and then gated cyclic residual optimization is performed to determine the smooth optimized trajectory point sequence. For the detected trajectory point sequence with fewer than the trajectory point threshold and the smoothed optimized trajectory point sequence, the goodness of fit of the curve fitting is determined, and the throwing behavior detection result is determined based on the goodness of fit.

2. The throwing behavior detection method based on adaptive tracking and residual optimization according to claim 1, characterized in that, Based on the detection frames of each tool, the trajectory of the tool trajectory object is tracked by fusing occlusion perception filtering, cost adaptive weighted synthesis, and confusion perception decoupling association through a multi-source adaptive confusion perception tracker, and the detected trajectory point sequence of the tool trajectory object is output, including: Candidate detection boxes are constructed using the detection boxes of each tool and the associated detection appearance feature vectors, and a candidate set is formed. Occlusion-aware non-maximum suppression is performed on the candidate set to divide it into a primary candidate set and a backup candidate set; Based on the Kalman filter, predict the trajectory prediction box of the tool trajectory object in the current frame in the trajectory dictionary, and determine the predicted appearance feature vector of each trajectory prediction box. The motion cost and appearance cost are calculated by using the main candidate detection boxes and their respective detection appearance feature vectors of the main candidate set, and the trajectory prediction boxes and their respective prediction appearance feature vectors, and then weighted and summed according to adaptive weights to determine the basic comprehensive cost. For the main candidate detection boxes that meet the confusion criteria, an confusion penalty term is introduced into the corresponding basic comprehensive cost to determine the confusion comprehensive cost; The main candidate detection box and the trajectory prediction box are matched in one stage by the Hungarian algorithm based on the basic comprehensive cost or the confusion comprehensive cost, and the successfully matched box pairs, unmatched main candidate detection boxes and unmatched trajectory prediction boxes are determined. If the unmatched trajectory prediction box is not empty, then the unmatched trajectory prediction box is repaired and associated in two stages based on the occlusion comprehensive cost in combination with the backup candidate set, and the unmatched backup candidate detection box is determined and the successfully matched box pair and the unmatched trajectory prediction box are updated. Update the trajectory points in the trajectory dictionary according to the coordinates of the center point of the detection box in each successfully matched box pair; In the trajectory dictionary, create tool trajectory objects for unmatched primary candidate detection boxes and unmatched backup candidate detection boxes, and update the trajectory points according to the coordinates of the center point of the corresponding detection box. Update the number of lost frames of the tool trajectory object to which the unmatched trajectory prediction box belongs in the trajectory dictionary, and delete the tool trajectory object whose number of lost frames meets the frame number threshold in the trajectory dictionary; Output the sequence of detected trajectory points for each tool trajectory object in the trajectory dictionary.

3. The throwing behavior detection method based on adaptive tracking and residual optimization according to claim 2, characterized in that, The step of performing occlusion-aware non-maximum suppression on the candidate set and dividing it into a primary candidate set and a backup candidate set includes: The candidate detection boxes in the candidate set are filtered by confidence threshold to determine high-confidence candidate detection boxes; Select the candidate detection box with the highest confidence as the main box, and calculate the main box IoU value between the main box and the other high-confidence candidate detection boxes. The high-confidence candidate detection boxes whose main frame IoU value is less than or equal to the perceived IoU threshold, along with the main frame, are moved into the main candidate set as main candidate detection boxes. The high-confidence candidate detection boxes whose main frame IoU value is greater than the perceptual IoU threshold and greater than the occlusion threshold are moved into the backup candidate set as backup candidate detection boxes.

4. The throwing behavior detection method based on adaptive tracking and residual optimization according to claim 2, characterized in that, If the unmatched trajectory prediction box is not empty, then based on the occlusion comprehensive cost, a two-stage repair association is performed on the unmatched trajectory prediction box in conjunction with the backup candidate set to determine the unmatched backup candidate detection box and update the successfully matched box pairs and the unmatched trajectory prediction box, including: If the unmatched trajectory prediction box is not empty, then calculate the occlusion comprehensive cost with an occlusion penalty term for each unmatched trajectory prediction box and the backup candidate detection box of the backup candidate set. The Hungarian algorithm is used to match unmatched trajectory prediction boxes with backup candidate detection boxes based on the comprehensive cost of each occlusion, to determine unmatched backup candidate detection boxes and update successfully matched box pairs and unmatched trajectory prediction boxes. The process of determining the overall cost of occlusion includes: In the formula, To cover the overall costs, For the price of sports, For the sake of appearance, For tool trajectory objects With detection box This pair of matching relationships has an occlusion penalty. For the repair correlation weight of motion cost, The repair association weight is the cost of appearance. To repair the correlation weight of the occlusion penalty item, For the Sigmoid function, This is the Sigmoid steepness coefficient. For the first The trajectory prediction box of the first tool trajectory object and the first IoU value between each detection box For the occlusion threshold, It is the numerical stability constant. For the first Confidence of each detection box, The confidence level of the main frame. This is the appearance adjustment factor. For appearance similarity.

5. The throwing behavior detection method based on adaptive tracking and residual optimization according to claim 1, characterized in that, The process involves constructing a residual sequence using spline interpolation trajectory point sequences and corresponding detection trajectory point sequences, followed by gated cyclic residual optimization to determine a smoothed optimized trajectory point sequence, including: The trajectory residuals are determined by calculating the difference between the trajectory points one by one using the associated spline interpolation trajectory point sequence and the detection trajectory point sequence, and the corresponding trajectory residual sequence is formed. Each trajectory residual sequence is transformed to a uniform scale to determine the standardized residual sequence. Calculate the first-order and second-order differences of the trajectory residuals in each of the trajectory residual sequences; The standardized residuals in each of the standardized residual sequences are concatenated with the associated first-order and second-order differences to form a time-series input vector, thus forming the corresponding time-series residual sequence. The corrected residual sequence of each of the time-series residual sequences is output through the gated loop unit; Average smoothing is performed on the corrected residual sequence that does not meet the residual jump detection condition to determine the smoothed corrected residual sequence; For the corrected residual sequence that meets the residual jump detection condition, after eliminating abnormal jumps and updating the corrected residual sequence, average smoothing is performed to determine the smoothed corrected residual sequence. Based on the smoothed correction residual sequences, the associated spline interpolation trajectory point sequences are compensated and corrected to output a smoothed optimized trajectory point sequence.

6. The throwing behavior detection method based on adaptive tracking and residual optimization according to claim 2, characterized in that, The tool detection box has category labels for both "tool in hand" and "tool in the air"; The process involves constructing candidate detection boxes using the detection boxes of each tool and the associated detection appearance feature vectors, and forming a candidate set, including: Based on the category labels of each tool detection box, category filtering is performed, and tool detection boxes with the category label "tool in the air" are determined as target tool detection boxes; The detection boxes of each target tool and the corresponding extracted detection appearance feature vectors are used to construct candidate detection boxes; A candidate set is formed using the candidate detection boxes described above.

7. The throwing behavior detection method based on adaptive tracking and residual optimization according to claim 1, characterized in that, The curve fitting includes parabolic fitting and linear fitting.

8. The throwing behavior detection method based on adaptive tracking and residual optimization according to claim 2, characterized in that, The process of determining the adaptive weights includes: In the formula, As the weight of the cost of movement, For the Sigmoid function, For location prediction residuals, The coordinates of the center point of the detection box. The coordinates of the center point of the trajectory prediction box. This represents the velocity change of the tool trajectory object between adjacent frames. It is the L2 norm. The weights for the location prediction residuals, As the weight of the change in velocity, The weight of appearance cost.

9. The throwing behavior detection method based on adaptive tracking and residual optimization according to claim 2, characterized in that, Confusion-prone conditions include an IoU value between the main candidate detection box and the trajectory prediction box that is greater than the confusion IoU threshold and an appearance similarity that is greater than the appearance threshold. The process of determining the overall cost of confusion includes: In the formula, To obscure the overall cost, Based on comprehensive costs, For the first The trajectory object of the tool and the first Obfuscation penalty between detection boxes The penalty coefficient is... For the Sigmoid function, This is the Sigmoid steepness coefficient. For the first The trajectory prediction box of the first tool trajectory object and the first IoU value between each detection box To obfuscation threshold, For appearance similarity.

10. A throwing behavior detection system based on adaptive tracking and residual optimization, characterized in that, include: The target detection module is used to perform target detection on the working tool image of the current frame using the YOLOv8 tool target detection model, and determine the tool detection box; The target tracking module is used to track the trajectory of the tool trajectory object based on each of the tool detection boxes by fusing occlusion perception filtering, cost adaptive weighted synthesis and confusion perception decoupling association through a multi-source adaptive confusion perception tracker, and outputs the detection trajectory point sequence of the tool trajectory object; The trajectory point interpolation module is used to determine the corresponding spline interpolated trajectory point sequence by cubic B-spline interpolation for detected trajectory point sequences where the number of trajectory points is greater than or equal to the trajectory point threshold. The trajectory optimization and smoothing module is used to construct a residual sequence by using spline interpolation trajectory point sequence and corresponding detection trajectory point sequence, and then perform gated cyclic residual optimization to determine the smoothed optimized trajectory point sequence. The throwing behavior analysis module is used to determine the goodness of fit of the curve fitting for the detection trajectory point sequence with a number of trajectory points less than the trajectory point threshold and the smoothed optimized trajectory point sequence, and to determine the throwing behavior detection result based on the goodness of fit.

Images