A method for detecting violations by people at conveyor belt stations based on improved S2ANet and trajectory tracking

Abstract

This invention relates to a method for detecting violations by personnel at conveyor belt stations based on an improved S2ANet and trajectory tracking. The method includes acquiring video and segmenting it into images frame by frame; inputting the images into the S2ANet-CBAM algorithm for conveyor belt target detection to obtain conveyor belt data; inputting the images into the YOLOv5 algorithm for worker target detection to obtain worker data; determining whether the conveyor belt is horizontal by comparing the conveyor belt data and worker data; using a Kalman filter to predict the position information of the worker and conveyor belt in the next frame; and using the Hungarian association algorithm to match and track the trajectories of the worker and conveyor belt. Finally, trajectory correction is performed, and the worker's trajectory and the conveyor belt trajectory are drawn, and the existence of violations is determined. This invention solves the problem of overlapping horizontal target boxes between the conveyor belt and the person in industrial applications due to the large size of the conveyor belt target, avoiding trajectory mismatch problems caused by target occlusion and errors caused by camera shake.

Description

Technical Field

[0001] This invention belongs to the field of behavior detection technology, and more specifically, relates to a method for detecting violations of rules by people at conveyor belt stations based on an improved S2ANet and trajectory tracking. Background Technology

[0002] Conveyor belts, as an effective tool for logistics and transportation, have been widely used in the industrial field, bringing great convenience to people. However, they have also brought many hidden dangers. During operation, workers are prone to unsafe behaviors such as crawling under, crossing, or stepping on conveyor belts in violation of regulations. Once such behaviors occur, they pose a great threat to personal safety, and in severe cases, can even lead to death. Therefore, while publicizing safety regulations, we need to effectively supervise work practices.

[0003] Currently, artificial intelligence (AI) technology is developing rapidly, with various algorithms emerging in endless streams. Many applications already use AI algorithms to solve practical problems, and applying object detection and trajectory tracking algorithms to violation monitoring is also an effective method to avoid security risks. Existing patent literature also discloses related solutions, such as:

[0004] Chinese patent document CN111814594A discloses a method, apparatus, device, and storage medium for identifying logistics violations, including: S1: acquiring several images of logistics violations; S2: annotating the logistics violations appearing in the images and converting them into an image dataset in TFRecord format; S3: training a YOLOv3 model pre-converted into a TensorFlow fixed model using the image dataset based on the TensorFlow object detection API, and verifying the trained YOLOv3 model to obtain a violation identification model; S4: identifying the logistics scene using the violation identification model, determining whether there are logistics violations, and obtaining the identification result.

[0005] Chinese patent document CN113076899A discloses a method for detecting foreign objects on high-voltage transmission lines based on a target tracking algorithm. The method includes: placement of a camera device on the transmission tower and data transmission; collection and labeling of foreign objects on the high-voltage transmission line; training and optimization of a YOLOv5 model; using Hough line detection to determine the approximate range of the high-voltage transmission line; detecting the video stream, optimizing the detection results using an algorithm, and selecting the target to be tracked; using the DeepSort target tracking algorithm and Kalman filtering to predict the target's position; using the Hungarian correlation algorithm to calculate the correlation between the predicted and detected targets and performing matching; determining whether the target will reach the high-voltage transmission line area based on the existing target trajectory; and determining whether the target is an abnormal target affecting the high-voltage transmission line based on its final position.

[0006] In summary, when applying horizontal object detection algorithms like YOLO to detect violations at conveyor belt stations, the large size of the conveyor belt causes the horizontal bounding boxes to fail to fit the belt well. This not only introduces a lot of background information into the model training but also includes workers behind the conveyor belt within the bounding box, failing to effectively separate the conveyor belt from the people. This results in numerous misidentifications of workers stepping on the conveyor belt. Furthermore, the complexity of factory environments makes it easy for targets to be obscured for extended periods, leading to mismatches in trajectory detection. Summary of the Invention

[0007] This invention aims to overcome at least one of the shortcomings of the prior art and provides a method for detecting violations by workers on conveyor belts based on an improved S2ANet and trajectory tracking. It addresses the problem of misidentification of worker violations caused by excessively large conveyor belt targets or overlap between the conveyor belt and the worker's horizontal target bounding box in current industrial applications. This method avoids trajectory mismatch issues caused by long-term target occlusion and significantly reduces false alarms with minimal cost increase.

[0008] The detailed technical solution of this invention is as follows:

[0009] A method for detecting violations by personnel at conveyor belt stations based on an improved S2ANet and trajectory tracking, including...

[0010] First, the S2ANet-CBAM algorithm is used to detect rotating targets on the conveyor belt. This invention uses rotating target detection to not only generate a rotating bounding box that fits the conveyor belt, but also to obtain information such as the direction and angle of the conveyor belt in each frame. Combined with trajectory tracking, it is easier to obtain the positional relationship between the worker and the conveyor belt and to more accurately identify violations.

[0011] Using YOLOv5 to detect worker targets effectively ensures the accuracy of target detection;

[0012] Added calculation and judgment of the overlap area of ​​the target box (the overlap between the target box of the worker and the target box of the conveyor belt) to effectively avoid the situation of misidentification caused by the worker passing by the horizontal conveyor belt;

[0013] Secondly, the Kalman filter is used to predict the target to improve tracking accuracy; the Hungarian correlation algorithm is used to match and track the target trajectory to avoid trajectory mismatch problems when the target is occluded for a long time during the tracking process.

[0014] To avoid errors caused by camera shake and conveyor belt vibration, this paper proposes a trajectory correction method to effectively solve the shake problem;

[0015] Finally, by analyzing the relationship between the conveyor belt angle and the worker's trajectory angle, violations can be identified, effectively solving the problem of false identification and reducing the false alarm rate at a low cost.

[0016] This invention discloses a method for detecting violations by personnel at conveyor belt stations based on an improved S2ANet and trajectory tracking. The specific steps are as follows:

[0017] S1: Obtain images segmented from video frames, add an improved CBAM attention module to the S2ANet (S2ANet-CBAM) rotating target detection algorithm to read the images for conveyor belt target detection, and simultaneously feed the images into the YOLOv5 algorithm to detect worker targets, further ensuring detection accuracy;

[0018] The S2ANet-CBAM model was trained using an experimental dataset to obtain the optimal model. The experiment used a factory dataset containing 10,000 images across two categories: people and conveyor belts. The training, validation, and test sets were divided in an 8:1:1 ratio. The live video stream was acquired, and then segmented into multiple images at 1 second per frame. Specifically... Figure 3 As shown, it is fed into the S2ANet-CBAM model. Figure 3 Images captured during false alarm scenarios.

[0019] S2: Outputs the position information of two targets, the conveyor belt and the worker, including the center coordinates, width and height, and angle. The worker does not output the angle.

[0020] Output detection results as follows Figure 4 As shown, it includes the target bounding box and angle information, where the angle is represented as follows:

[0021] A coordinate system is established with the center point of the conveyor belt as the origin, the right direction as the positive horizontal axis and the top direction as the positive vertical axis. The angle between the conveyor belt and the right direction coordinate axis is the angle information of the conveyor belt output by the model, which is in the range of [-90, 90].

[0022] S3: Determine whether the conveyor belt is horizontal based on the absolute value of the conveyor belt angle. Set an angle threshold. If the absolute value of the angle is less than the angle threshold, it is horizontal; if the absolute value of the angle is greater than the angle threshold, it is not horizontal.

[0023] If the work is horizontal, calculate the overlapping area between the lower half of the worker's target frame and the conveyor belt's target frame to avoid misidentification caused by the worker passing by the horizontal conveyor belt.

[0024] If the conveyor belt is not horizontal, it is determined that the conveyor belt in the application scenario is inclined. This will prevent workers from passing by a horizontal conveyor belt and causing misidentification. There is no need to calculate the overlapping area. The data is directly fed into the Kalman filter and Hungarian correlation algorithm for trajectory tracking.

[0025] S4: Use a Kalman filter to predict the target's position information in the next frame image, and use the Hungarian correlation algorithm to match and track the target trajectory. Then, correct and draw the worker's trajectory.

[0026] S5: Determine whether there is a violation by comparing the worker's trajectory angle with the conveyor belt angle to see if they meet the conditions: Obtain the angle of the worker's trajectory relative to the positive direction of the horizontal axis, i.e., the worker's trajectory angle and the conveyor belt angle, by using the position information of the worker and the conveyor belt and the established coordinate system. Set a violation threshold, calculate the difference between the two angles, and determine whether there is a violation if the absolute value of the difference is less than the violation threshold, or if it is higher than the violation threshold.

[0027] Track the trajectory between the center point of the conveyor belt and the center point of the worker, and plot it as follows: Figure 5 As shown, the trajectory changes are minimal, generally forming a straight line. While the exact straight line may not be fully represented due to camera shake or other factors, this confirms that the conveyor belt did not move or change during this period. Furthermore, the shape, direction, and angle of the conveyor belt are as follows... Figure 4 As shown in the diagram; when workers stand on the conveyor belt, they will regularly approach and move away from the center of the conveyor belt at an angle. The workers will regularly move away from and approach the conveyor belt, but they are generally in a straight line and do not appear to be in the same direction as the shape of the conveyor belt. This indicates that the workers are moving goods on one side of the conveyor belt.

[0028] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0029] (1) The present invention provides a method for detecting violations of the behavior of people standing on a conveyor belt based on the improved S2ANet and trajectory tracking. Based on the characteristics of the S2ANet algorithm that can generate target boxes that fit the conveyor belt and output angle information, the method realizes the detection of people standing on the conveyor belt. It solves the problem of misidentification of the violation of the behavior of workers stepping on the conveyor belt caused by the large size of the conveyor belt target and the overlap of the horizontal target box of the conveyor belt and the person in the current industry. Furthermore, the CBAM attention module is improved to further improve the recognition accuracy.

[0030] (2) The present invention provides a method for detecting violations of personnel at conveyor belt stations based on improved S2ANet and trajectory tracking. It uses Kalman filter to predict the target and improves the tracking accuracy; it uses Hungarian correlation algorithm to match and track the target trajectory to avoid the problem of trajectory mismatch after the target is blocked for a long time during the tracking process; and it proposes a trajectory and angle correction method to effectively reduce the error caused by camera shake.

[0031] (3) The present invention provides a method for detecting violations of workers at conveyor belt stations based on improved S2ANet and trajectory tracking. It judges violations by analyzing the relationship between the shape of the conveyor belt and the trajectory of the workers. It effectively solves the problem of false identification and reduces the false alarm rate at a low cost.

[0032] (4) The present invention provides a method for detecting violations of personnel at conveyor belt stations based on improved S2ANet and trajectory tracking. By increasing the calculation and judgment of the overlapping area of ​​the target box, it effectively avoids the situation of misidentification caused by workers passing by the horizontal conveyor belt.

[0033] (5) The present invention provides a method for detecting violations of rules by people at conveyor belt stations based on improved S2ANet and trajectory tracking, and proposes a trajectory correction method to effectively reduce the error caused by camera shake. Attached Figure Description

[0034] Figure 1 This is a schematic diagram of the overall process of the present invention.

[0035] Figure 2 This is a schematic diagram of the improved CBAM attention module network structure in Embodiment 1 of the present invention.

[0036] Figure 3 The input image is the video cropped in Embodiment 1 of the present invention.

[0037] Figure 4 This is a diagram showing the S2ANet-CBAM detection results in Embodiment 1 of the present invention.

[0038] Figure 5 This is a diagram illustrating the target trajectory and angle in Embodiment 1 of the present invention.

[0039] Figure 6 This is a diagram showing the corrected trajectory and angle of the non-violation target in Embodiment 1 of the present invention.

[0040] Figure 7 This is a diagram showing the corrected trajectory and angle of the illegal target in Embodiment 1 of the present invention. Detailed Implementation

[0041] The present disclosure will be further described below with reference to the accompanying drawings and embodiments.

[0042] It should be noted that the following detailed descriptions are exemplary and intended to provide further illustration of this disclosure. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.

[0043] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the exemplary embodiments according to this disclosure. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms “comprising” and / or “including” are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0044] Where there is no conflict, the embodiments and features described herein can be combined with each other.

[0045] Example 1

[0046] S1. Acquire video and segment it into images frame by frame. Input the images into the S2ANet-CBAM algorithm to perform conveyor belt target detection and obtain conveyor belt data information.

[0047] S11. Acquire video captured by factory terminal equipment, set video duration, such as 30 minutes, set segmentation frame rate, such as 1 second / frame, and segment the video to obtain images at 1 second / frame.

[0048] S12. The S2ANet-CBAM algorithm of the present invention incorporates an improved CBAM attention module. The improved CBAM attention module includes channel attention and spatial attention. The two modules are combined in the order of channel attention first and spatial attention second. Based on channel attention, a pooling allocation weight is added when performing feature pooling operation.

[0049] The S2ANet algorithm consists of three parts: a backbone network, a Neck network, and a Head network. Based on this, this invention adds an improved CBAM attention module to the backbone network to obtain relevant information between feature channels and in space, focusing attention on important parts of the features and further improving the accuracy of rotating target detection. The improved CBAM attention module is located between two ResNet blocks, specifically after the last convolutional layer of the previous ResNet block, without changing the ResNet network structure, and can still use pre-trained parameters. When there are three or more ResNet blocks, there are also corresponding multiple improved CBAM attention modules.

[0050] Preferably, rotating the target detection box and performing conveyor belt target detection yields an image that identifies the target object. The bounding box is the smallest bounding rectangle of the target, closely fitting the target object. It retains the positional information of the target bounding box in the image, such as the center point coordinates, width, height, and angle, as well as the label name. The width, height, and angle can reflect the physical size and orientation of the target object. Moreover, it contains fewer background pixels than the horizontal target box, resulting in less background noise when classifying it, and can effectively separate dense targets.

[0051] Backbone Network: The backbone network of S2ANet is a ResNet residual network, mainly composed of residual units. It addresses the vanishing or exploding gradients during backpropagation caused by increasing the number of convolutional layers. The residual units incorporate skip connections, directly adding the input and output to supplement the feature information lost during convolution. The skip connections perform an add operation.

[0052] The improved CBAM attention module is a lightweight attention module that mainly consists of channel attention and spatial attention. It can perform attention acquisition operations in both spatial and channel dimensions. In the network structure, the two modules are combined in a sequential order (channel attention first, spatial attention second). Figure 2 As shown, the squares and F, F', and F'' all represent feature maps, Mc represents channel attention features, and Ms represents spatial attention features.

[0053] (1) Channel Attention: The input feature map F (H×W×C) is processed by global max pooling and global average pooling based on width and height, respectively, to obtain two 1×1×C feature maps. Then, they are fed into a two-layer neural network (MLP). The first layer has C / r neurons (r is the reduction rate) and ReLU activation function, while the second layer has C neurons. This two-layer neural network is shared. Then, the features output by the MLP are summed at the pixel level and then activated by sigmoid (σ) to generate the channel attention feature, Mc. Finally, Mc and the input feature map F are multiplied element-wise to generate the input features required by the spatial attention module.

[0054] (1)

[0055] In the formula, F is the input feature map, Mc(F) is the output feature map, MLP is a two-layer neural network, AvgPool is average pooling, and MaxPool is max pooling.

[0056] Building upon channel attention, to reduce errors and minimize information loss during feature pooling, a pooling weight allocation is added. This ensures the output pooled features more closely resemble the original features, avoiding excessive information loss. The pooling weight allocation is calculated as follows:

[0057] (2)

[0058] (3)

[0059] In the formula, w1 is the weight of average pooling and w2 is the weight of max pooling.

[0060] Therefore, the final channel attention features are as follows:

[0061] (4)

[0062] In the formula, Mc'(F) represents the final channel attention feature.

[0063] (2) Spatial attention: The feature map output by the channel attention module The input feature map is used as the input feature map for this module. First, a channel-based global max pooling and global average pooling are performed to obtain two H×W×1 feature maps. Then, these two feature maps are concatenated based on their channels. Next, a 7×7 convolution operation is performed to reduce the dimensionality to one channel, i.e., H×W×1. Then, a sigmoid (σ) activation function is used to generate a spatial attention feature, Ms. Finally, this feature is multiplied with the input feature of this module to obtain the final generated feature.

[0064] (5)

[0065] In the formula, AvgPool() is average pooling, and MaxPool() is max pooling. These are feature maps generated after average pooling. It is the feature map generated after max pooling. f 7x7 For a 7×7 convolution operation, F' is the spatial attention input feature map, which is also the feature map output by the channel attention module; Ms(F') is the spatial attention output feature map.

[0066] Neck Network: It adopts the FPN structure, which enhances the feature fusion capability of the network. The FPN structure uses upsampling to improve the testing capability for small targets.

[0067] The Head network consists of a Feature Alignment Module (FAM) and an Orientation Detection Module (ODM). FAM generates high-quality anchors through an Anchor Refinement Network (ARN) and adaptively aligns features based on the corresponding anchor boxes using Align Conv. Unlike other methods that densely sample anchors, this method uses only one squared anchor at each location in the feature map, which the ARN refines into high-quality rotated anchors. Align Conv then adaptively aligns the features based on the shape, size, and orientation of its corresponding anchor. In the ODM, orientation information is first encoded using an Active Rotation Filter (ARF) to generate orientation-sensitive features, which are then pooled to extract orientation-invariant features. Finally, these features are fed into a regression sub-network and a classification sub-network to produce the final prediction.

[0068] S13. Input the image into the S2ANet-CBAM algorithm to perform conveyor belt target detection and obtain conveyor belt data information; the conveyor belt data information includes: center coordinates, width and height, angle and other position information and label name.

[0069] S2. Input the image into the YOLOv5 algorithm to perform worker target detection and obtain worker data information;

[0070] The image is input into the YOLOv5 algorithm for worker target detection to obtain worker data information, which includes: center coordinates, width and height position information, and label name.

[0071] The output of object detection can produce an image that identifies the target object. The bounding box is the smallest bounding rectangle of the target, which fits the target object tightly. It retains the positional information of the target bounding box in the image, such as the center point coordinates, width, height, and angle, as well as the label name. The width, height, and angle can reflect the physical size and orientation of the target object. Moreover, it contains fewer background pixels than the horizontal target box, resulting in less background noise when classifying it, and can effectively separate dense targets.

[0072] S3. Determine whether the conveyor belt is in a horizontal state by comparing the conveyor belt data information and the worker data information. If not, proceed to the next step. If it is, calculate whether the overlapping area of ​​the worker target detection box and the conveyor belt is less than the overlap threshold. If it is less, proceed to the next step. If it is not greater, end.

[0073] Based on the conveyor belt data, a coordinate system is established with the center point of the conveyor belt as the origin, the right direction as the positive horizontal axis and the top direction as the positive vertical axis. The angle between the conveyor belt and the right direction coordinate axis is the angle information of the conveyor belt output by the model, and the range is [-90, 90].

[0074] Whether the conveyor belt is horizontal is determined by the absolute value of the conveyor belt angle. An angle threshold is set. If the absolute value of the angle is less than the angle threshold, it is horizontal; if the absolute value of the angle is greater than the angle threshold, it is not horizontal.

[0075] If the direction is horizontal, calculate the overlapping area between the lower half of the worker's target frame and the conveyor belt target frame to prevent the worker from passing by the horizontal conveyor belt and causing misidentification. First, a default overlap threshold is set, the overlapping area between the lower half of the worker's target frame and the conveyor belt is calculated, and it is determined whether the overlapping area is less than the overlap threshold. If so, proceed to step S4. If it is greater than the overlap threshold, it is determined that the worker is standing on the conveyor belt and an alarm is set to prevent the worker from passing by and walking along the direction of the horizontal conveyor belt and causing misidentification.

[0076] Preferably, the angle threshold is set to 3 degrees. When the absolute value of the conveyor belt's conveying angle is 3 degrees or less, it is determined that the conveyor belt is horizontally conveying, and the overlapping area needs to be determined. Preferably, the overlapping threshold is set to 1 square meter. When the overlapping area is 1 square meter or less, step S4 is entered.

[0077] If the conveyor belt is not horizontal, it indicates that the conveyor belt in the application scenario is inclined, and there will be no situation where workers pass by a horizontal conveyor belt and cause misidentification. Preferably, the angle threshold is set to 3 degrees. When the absolute value of the conveyor belt's conveying angle is 4 degrees or more, it is determined that the conveyor belt is inclined and proceeds directly to S4.

[0078] S4. Use a Kalman filter to predict the position information of the target (worker and conveyor belt) in the next frame image, and use the Hungarian correlation algorithm to match and track the trajectory of the target (worker and conveyor belt). Finally, correct the trajectory and draw the worker's trajectory.

[0079] S41. Initialize the Kalman filter and predict the position information of the conveyor belt and the worker in the next frame image; the specific principle of the Kalman filter is as follows:

[0080] The target position is predicted using a Kalman filter, which has good noise immunity. Let the previous state of the target be xk⁻¹ = (Lk⁻¹, vk⁻¹), where L and v represent position and velocity respectively, and k⁻¹ represents the previous frame image. Considering external control variables and external noise interference, the state prediction equation can be obtained from kinematics and related mathematical formulas as follows.

[0081] (9)

[0082] In the formula, The optimal estimate for the current frame; Fk is the optimal estimate of the previous frame; FTk is its transpose; Bk is the external control matrix; uk is the external control quantity; Lk is the current frame position; Lk-1 is the previous frame position; Qk is the external noise covariance matrix. As can be seen from equation (1), the optimal estimate of the k-1 frame is obtained by adding the optimal estimate of the k-1 frame to the external control quantity, and the position of the k-1 frame is obtained by adding the position of the k-1 frame to the external interference.

[0083] To ensure real-time tracking and accuracy, some parameters in the model need to be updated. Let the detection output of S2ANet-CBAM be Zk, and its covariance matrix be Rk. Combined with previous prediction results, obtain the predicted Gaussian distribution (μ0, ∑0) = (Hk ,HkLkHTk), where Hk is the covariance matrix of the predicted Gaussian distribution; obtain the detected Gaussian distribution (μ1, ∑1)=(Zk, Rk). The Kalman filter state update equation is as follows:

[0084] (10)

[0085] In the formula, K is the Kalman gain, and x^' (whether it is...) ') is the current optimal estimate.

[0086] S42. Set the target in the next frame image as the predicted target; when the predicted next frame image becomes the current detection frame, generate the actual detection target, use motion association cost to calculate the IOU (Intersection over Union) value between the predicted target and the actual detection target to construct the trajectory association probability matrix, and integrate the final trajectory association probability into the Hungarian association algorithm to complete trajectory association matching and tracking.

[0087] The principles behind motion association cost and trajectory association probability are as follows:

[0088] Based on the location coordinates of the detected target, the intersection-union ratio (IOU) between the bounding boxes of the predicted target and the detected target is obtained, and the motion association cost DM(i,j) is calculated accordingly.

[0089] During detection, the area of ​​the target bounding box of the reference trajectory in each frame is calculated and denoted as SAi, i∈[1,m]; the area of ​​the target bounding box of the actual trajectory in each frame is calculated and denoted as SBi; then the area of ​​the shaded region where SAi and SBi intersect is calculated; finally, the intersection-union ratio (IOUi) of the two bounding box areas is calculated.

[0090] (11)

[0091] The trajectory association probability is generally a weighted sum of appearance cost and motion cost. This paper only involves motion cost, therefore the trajectory association probability D(i,j) is as follows:

[0092] (12)

[0093] The Hungarian association algorithm is based on the idea of ​​sufficiency proof in Hall's theorem. It is the most common algorithm for partial graph matching. The core of the algorithm is to find augmenting paths. It is an algorithm that uses augmenting paths to find the maximum matching of a bipartite graph.

[0094] (1) Finding an alternating path: A path formed by starting from an unmatched point and passing through unmatched edges, matched edges, and unmatched edges in sequence is called an alternating path.

[0095] (2) Finding an augmenting path: Starting from an unmatched point, take an alternating path. If the path passes through another unmatched point (not counting the starting point), then the alternating path is called an augmenting path.

[0096] Specifically, the Hungarian correlation algorithm is a recursive process that tries to find a one-to-one match between the target in the previous frame and the target in the current frame. This algorithm places high demands on the motion model, requiring a confidence level higher than 0.6 (confidence level is...). Figure 4 To obtain more accurate results, the numbers on the target bounding box (within the range [0,1]) are fed into the Hungarian association algorithm for matching.

[0097] S43. If the Hungarian correlation algorithm produces unmatched detection results, it is determined that the target may have been missed due to S2ANet-CBAM, and the number of missed frames in history is recorded. Preferably, if the cumulative number of missed frames is greater than 70, the target is considered to have disappeared, and a new Kalman filter is initialized; otherwise, trajectory correlation matching and tracking continue.

[0098] S44. Determine if all images have been processed. If not, repeat steps S13-S44, updating the relevant parameters of the Kalman filter, iterating multiple times until all images have been processed. If all images have been processed, continue with trajectory and angle correction.

[0099] S45. Preferably, trajectory correction takes a worker as an example, and the same applies to a conveyor belt. First, obtain the center point coordinates of the worker in each frame; then, take three adjacent center points as a batch, and only the third point of the previous batch and the first point of the next batch are the same point in the previous batch. Calculate the angle between the two straight lines formed by the first two points and the last two points. If it is lower than the threshold, it can be replaced by the straight line formed by the first and last points, and then process the next batch. This process is repeated until a trajectory without jitter is obtained; if it is higher than the threshold, this operation is not performed, and the center point of the next batch is judged.

[0100] Angle correction mainly targets the conveyor belt. First, the angle of the conveyor belt in each frame is obtained. θ i Then find the median. mid() and average value average Finally, the corrected angle is obtained using the following formula. θ :

[0101] (6)

[0102] (7)

[0103] (8).

[0104] In the formula, The weights are the median. The weight of the average value, mid () is the median function. average () is the average function. θ i For the angle of the conveyor belt in each frame, θ This is the corrected angle.

[0105] S5. Determine whether there is a violation by comparing the angle of the worker's trajectory with the angle of the conveyor belt to see if they meet the conditions. Obtain the angle of the worker's trajectory relative to the positive direction of the horizontal axis and the angle of the conveyor belt respectively. Calculate the difference between the absolute values ​​of the two angles. If the difference is less than the threshold, there is a violation; if it is higher than the threshold, there is no violation.

[0106] The existence of violations is determined by comparing the angle of the worker's trajectory with the angle of the conveyor belt to see if they meet the violation conditions. The angle θ1 of the worker's trajectory relative to the horizontal axis and the angle θ2 of the conveyor belt relative to the horizontal axis are obtained respectively. The absolute value θ of the difference between the two angles is calculated. If the difference is less than a threshold, a violation exists; if it is greater than the threshold, no violation exists.

[0107] Worker trajectories Figure 5 , 6 As shown in Figure 7, a coordinate system is established with the right and up directions as positive directions. The angle between the worker's trajectory and the right coordinate axis is the angle information of the worker's trajectory output by the model, and the range is also [-90, 90].

[0108] Track the trajectory between the center point of the conveyor belt and the center point of the worker, and plot it as follows: Figure 5 As shown, the trajectory changes are minimal, generally forming a straight line. While it may not perfectly straight due to camera tremors, this confirms that the conveyor belt did not move or change during this period. Furthermore, the conveyor belt angle is as follows... Figure 4 As shown in the diagram; when workers stand on the conveyor belt, they will regularly approach and move away from the center of the conveyor belt at an angle. The workers will regularly move away from and approach the conveyor belt, but they are generally in a straight line and do not appear to be in the same direction as the shape of the conveyor belt. This indicates that the workers are moving goods on one side of the conveyor belt.

[0109] Calculate the absolute difference between the worker's trajectory angle and the conveyor belt angle. For workers 1, 2, and 3, the angle is approximately 0 degrees, while the conveyor belt angle is 22 degrees. The threshold is 3 degrees. The absolute difference is 22, which is greater than 3. Therefore, it can be determined that the worker is not standing on the conveyor belt and is not considered a violation. The detection ends and no alarm is needed.

[0110] Worker trajectories Figure 6 In this case, worker 1's trajectory angle is 24.3 degrees, while the conveyor belt angle is still 22 degrees. The absolute difference is 2.3, which is less than the threshold of 3. Therefore, it is determined that the worker is standing on the conveyor belt, and an alarm needs to be triggered.

[0111] Since the conveyor belt does not move for a considerable period of time, its trajectory can be considered as a straight line. However, the shape, direction, and angle of the conveyor belt can be obtained based on the position information. When a worker stands on the conveyor belt, the trajectory will rise or fall at a regular angle. By comparing the angle of the worker's trajectory with the angle of the conveyor belt, it can be determined whether there is any violation. If there is, the result is output and an alarm is triggered; otherwise, the process ends.

[0112] Obviously, the above embodiments of the present invention are merely examples for clearly illustrating the technical solution of the present invention, and are not intended to limit the specific implementation of the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the claims of the present invention should be included within the protection scope of the claims of the present invention.

Claims

1. A conveyor belt station human violation behavior detection method based on improved S2A Net and trajectory tracking, characterized in that, include: S1. Acquire video and segment it into images frame by frame. Input the images into the S2ANet-CBAM algorithm to perform conveyor belt target detection and obtain conveyor belt data information. The S2ANet-CBAM algorithm is as follows: The S2ANet algorithm consists of three parts: a backbone network, a Neck network, and a Head network. On this basis, an improved CBAM attention module is added to the backbone network to obtain relevant information between feature channels and in space. The improved CBAM attention module is located between two ResNet blocks, specifically after the last convolutional layer of the previous ResNet block. The improved CBAM attention module includes channel attention and spatial attention, which are combined together with channel attention first and spatial attention second. Based on channel attention, a pooling weight is added when performing feature pooling operations. S2. Input the image into the YOLOv5 algorithm to perform worker target detection and obtain worker data information; S3. Determine whether the conveyor belt is in a horizontal state by comparing the conveyor belt data information and the worker data information. If not, proceed to the next step. If yes, calculate whether the overlapping area of ​​the worker target detection box and the conveyor belt is less than the preset overlap threshold. If it is less, proceed to the next step. If it is greater, end. S4. Use a Kalman filter to predict the position information of the worker and the conveyor belt in the next frame image, and use the Hungarian correlation algorithm to match and track the trajectory of the worker and the conveyor belt. Finally, correct the trajectory and draw the worker trajectory and the conveyor belt trajectory. S5. By analyzing the worker's trajectory and the conveyor belt trajectory, the worker's trajectory angle and the conveyor belt angle are obtained, and it is determined whether there is any violation.

2. The method of claim 1, wherein the method is based on improved S2ANet and trajectory tracking for conveyor belt station human violation detection. The process of acquiring video and segmenting it into images frame by frame, and then inputting the images into the S2ANet-CBAM algorithm for conveyor belt target detection to obtain conveyor belt data information specifically includes: Acquire videos captured by factory terminal equipment, set video duration and segmentation frame rate, and segment the video to obtain images based on the segmentation frame rate; The image is input into the S2ANet-CBAM algorithm, the target detection box is rotated, and the conveyor belt target detection is performed to obtain the conveyor belt data information.

3. The method of claim 1, wherein the method is based on improved S2ANet and trajectory tracking for conveyor belt station human violation detection. The step of determining whether the conveyor belt is in a horizontal state by combining conveyor belt data information and worker data information specifically includes: Based on the conveyor belt data, a coordinate system is established with the center point of the conveyor belt as the origin, the right direction as the positive horizontal axis and the top direction as the positive vertical axis. The angle between the conveyor belt and the right direction coordinate axis is the angle information of the conveyor belt output by the model. Set an angle threshold. Determine if the conveyor belt is level by comparing the absolute value of the conveyor belt's angle with the angle threshold. If the conveyor belt is not level, end the detection. If the conveyor belt is level, calculate the overlapping area between the lower half of the worker's target frame and the conveyor belt's target frame. Set an overlap threshold. Determine if the worker is walking along the horizontal conveyor belt by comparing the overlapping area with the overlap threshold.

4. The method of claim 1, wherein the method is based on improved S2ANet and trajectory tracking for conveyor belt station human violation detection. S4 specifically includes: Initialize the Kalman filter and predict the position information of the two targets, the conveyor belt and the worker, in the next frame image; The target in the next frame of the image is set as the predicted target. When the predicted next frame becomes the current detection frame, the actual detection target is generated. The IOU (Intersection over Union) value between the predicted target and the actual detection target is calculated using motion association cost to construct the trajectory association probability matrix. The final trajectory association probability is integrated into the Hungarian association algorithm to complete trajectory association matching and tracking.

5. The method of claim 4, wherein the method is based on improved S2ANet and trajectory tracking for conveyor belt station human violation detection. The trajectory association matching includes: When the Hungarian association algorithm performs a matching and there is no matching detection result, it is determined that the target was not detected due to the S2ANet-CBAM missed detection, and the number of historical missed detection frames is recorded. A preset upper limit for the number of missed detection frames is set. When the number of missed detection frames exceeds the upper limit, the target is considered to have disappeared.

6. The method of claim 1, wherein the method is based on improved S2ANet and trajectory tracking for conveyor belt station human violation detection. The trajectory and angle corrections specifically include: Obtain the center point coordinates of the target in each frame; then, take three adjacent center points as a batch, and only the third point of the previous batch and the first point of the next batch are the same point in the previous batch. Calculate the angle between the two straight lines formed by the first two points and the last two points, set a threshold, and if it is lower than the threshold, it can be replaced by the straight line formed by the first and last points, and then process the next batch. This process is repeated until a jitter-free trajectory is obtained; if it is higher than the threshold, it is kept as is, and the center point of the next batch is judged. For conveyor belts, angle correction is first performed by obtaining the angle of the conveyor belt for each frame, then calculating the median and average values, and finally obtaining the corrected angle using the following formula: ； wherein is the weight of the median, is the weight of the mean, mid () is the median function, average () is the mean function, θ i is the angle of the transport belt for each frame, θ is the corrected angle.

Images