Method and system for identifying violations of tobacco station personnel based on deep learning

Abstract

The present invention provides a method and system for identifying violations by tobacco station personnel based on deep learning. First, a video data set of tobacco leaf manipulation behavior within a preset time period of target tobacco station personnel is obtained, which includes multiple continuous video clip sequences. Then, spatiotemporal feature extraction is performed on the tobacco leaf manipulation behavior video data set to obtain skeletal key point motion trajectory features and operating instrument interaction features. Based on the dynamic matching of skeletal key point motion trajectory features and compliance action template features, a local action deviation score is generated. The operating instrument interaction features are input into a pre-trained violation feature classification network to generate an associated violation action feature set. Finally, violation behavior is identified on the video clip sequence based on the violation action feature set, and a violation identification result containing violation type labels and time positioning information is generated. The result is sent to the tobacco station supervision terminal to achieve accurate identification and positioning of tobacco station personnel violations.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a method and system for identifying violations by tobacco station personnel based on deep learning. Background Art

[0002] In tobacco industry tobacco station production operations, standardized operations are crucial for ensuring tobacco leaf processing quality, maintaining production safety, and improving production efficiency. However, traditional methods for identifying violations by tobacco station personnel rely primarily on manual inspections and on-site supervision. This approach not only consumes significant manpower and material resources but is also limited by factors such as human attention, experience, and fatigue, making it difficult to achieve real-time, comprehensive, and accurate monitoring of personnel operations. Manual inspections are often limited to specific time periods or specific operational links, which can easily lead to missed inspections and misjudgments, making it impossible to promptly detect and correct subtle violations that could have serious consequences.

[0003] With the advancement of video surveillance technology, some tobacco stations have begun to implement video surveillance systems to record personnel operations. However, most existing video surveillance systems offer only simple recording and playback capabilities, lacking the ability to deeply analyze and process video data. These systems typically only provide visual images of operator behavior and are unable to automatically identify and determine whether an operation violates regulations. Manual frame-by-frame review and analysis of large amounts of video data is still required, resulting in extremely low efficiency. Furthermore, the lack of in-depth exploration and quantitative analysis of operator behavior characteristics makes it difficult to accurately distinguish the subtle differences between normal and illegal operations, resulting in low accuracy and reliability in violation identification. Summary of the Invention

[0004] In view of the above-mentioned problems, in combination with the first aspect of the present invention, an embodiment of the present invention provides a method for identifying violations by personnel at a tobacco station based on deep learning, the method comprising:

[0005] Acquire a tobacco leaf handling behavior video data set of a target tobacco station staff within a preset time period, wherein the tobacco leaf handling behavior video data set comprises a plurality of continuous video clip sequences;

[0006] Performing spatiotemporal feature extraction processing on the tobacco leaf manipulation behavior video data set to obtain skeletal key point motion trajectory features and manipulation instrument interaction features of each video clip sequence;

[0007] Dynamically matching the motion trajectory features of the skeleton key points with the features of the preset compliant action template to generate a local action deviation score for the video clip sequence;

[0008] Inputting the operation instrument interaction features into a pre-trained violation feature classification network to generate a set of violation action features associated with the local action deviation score;

[0009] The video clip sequence is processed for violation behavior identification based on the violation action feature set, a violation identification result including a violation type label and time positioning information is generated, and the violation identification result is sent to the tobacco station supervision terminal.

[0010] On the other hand, an embodiment of the present invention also provides a tobacco station personnel violation identification system based on deep learning, including a processor and a machine-readable storage medium, the machine-readable storage medium is connected to the processor, the machine-readable storage medium is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the machine-readable storage medium to implement the above method.

[0011] Based on the above aspects, the embodiment of the present application realizes the accurate and efficient violation identification of tobacco leaf operation behavior of tobacco station personnel, and significantly improves the intelligent level and safety assurance capability of tobacco station production management. First, a continuous video data set of the target tobacco station personnel within a preset time period is obtained. By extracting the spatiotemporal features of the video data, not only the motion trajectory features of the key points of the skeleton of the personnel are accurately captured, but also the interaction features with the operating equipment are deeply excavated, and the operating behavior patterns of the personnel are comprehensively portrayed from multiple dimensions, effectively overcoming the recognition limitations caused by a single feature. Based on the dynamic matching processing of the motion trajectory features of the key points of the skeleton and the compliant action template, the degree of deviation of the local action can be quantified in real time. Furthermore, the interaction features of the operating equipment are input into the pre-trained violation feature classification network to generate a violation action feature set that is closely related to the local action deviation score, thereby realizing deep feature mining and intelligent classification of the violation behavior, and greatly improving the accuracy and reliability of violation identification. Finally, the video clip sequence was processed for violation identification based on the set of violation action features. The generated violation identification results not only included clear violation type labels, but also accurately located the time information of the violation, effectively solving the problem that the traditional violation identification method of tobacco station personnel relies on manual inspections, is inefficient and prone to missed detections. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 This is a schematic diagram of the execution flow of the method for identifying violations of tobacco station personnel based on deep learning provided by an embodiment of the present invention.

[0013] Figure 2 This is a schematic diagram of exemplary hardware and software components of a deep learning-based tobacco station personnel violation identification system provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0014] The present invention will be described in detail below with reference to the accompanying drawings. Figure 1This is a flow chart of a method for identifying violations by tobacco station personnel based on deep learning provided by an embodiment of the present invention. The method for identifying violations by tobacco station personnel based on deep learning is introduced in detail below.

[0015] Step S110 , obtaining a tobacco leaf manipulation behavior video data set of a target tobacco station staff within a preset time period, wherein the tobacco leaf manipulation behavior video data set includes a plurality of continuous video clip sequences.

[0016] In this example, to ensure the standardization and quality of tobacco leaf handling, it is necessary to monitor the operations of tobacco station personnel. For example, a preset time period is set as one day, from 8:00 AM to 5:00 PM. Within the tobacco station, there are multiple operation areas, each of which is equipped with a camera to capture the tobacco leaf handling activities of the tobacco station personnel. These cameras can continuously record at a fixed frame rate (for example, 30 frames per second), recording all operations of each tobacco station personnel during the day, thereby forming a video data set.

[0017] This video dataset contains multiple continuous video clip sequences. For example, the entire process of a tobacco station worker, from sorting tobacco leaves to packaging them, is recorded as one video clip sequence. Another video clip sequence records the process of another tobacco station worker, from preparing tobacco leaves for curing to placing them in the curing equipment. Each video clip sequence carries a start and end time, as well as information about tobacco leaf manipulation.

[0018] Step S120 , performing spatiotemporal feature extraction processing on the tobacco leaf manipulation behavior video data set to obtain skeleton key point motion trajectory features and manipulation instrument interaction features of each video clip sequence.

[0019] In this embodiment, human pose estimation can first be performed on each frame in a continuous video sequence. Specifically, taking a video sequence of tobacco sorting as an example, in a frame, each body part of a tobacco station worker (such as the head, shoulders, arms, hands, waist, legs, etc.) is considered a skeletal keypoint. For each keypoint, a skeletal keypoint sequence containing three-dimensional coordinates is generated. These three-dimensional coordinates represent each keypoint's position in space. For example, the coordinates of the head might be (x1, y1, z1), the coordinates of the hands might be (x2, y2, z2), and so on.

[0020] Next, the skeletal keypoint sequence is interpolated in the time dimension. Because actual filming may experience frame rate fluctuations or frame loss, interpolation can produce an interpolated skeletal keypoint sequence with evenly spaced time steps. Assuming the original keypoint sequence within a certain time period is uneven in time, interpolation ensures that a complete skeletal keypoint state record is recorded every 0.1 second (the set evenly spaced time step).

[0021] Next, a pre-trained spatiotemporal feature encoder is invoked to perform spatiotemporal convolutional encoding on the interpolated skeletal keypoint sequence to generate skeletal keypoint motion trajectory features. The interpolated skeletal keypoint sequence is divided into multiple time window segments, for example, each time window segment is 5 seconds. Two-stream feature extraction is performed within each time window segment. For the action of sorting tobacco leaves, the first-stream processing path performs one-dimensional convolutional encoding on the relative displacement between adjacent skeletal keypoints. For example, the displacement changes between adjacent keypoints in the hand and arm may exhibit specific patterns over time, and one-dimensional convolutional encoding can capture these local motion pattern characteristics. The second-stream processing path performs graph convolutional encoding on the spatial distribution of all skeletal keypoints within the time window. The graph convolutional encoding captures the spatial distribution changes of the overall body posture of tobacco station personnel when sorting tobacco leaves, such as the changes in the spatial relationship of skeletal keypoints caused by movements such as bending and reaching. Global posture evolution features are generated through graph convolutional encoding. The local motion pattern features are then concatenated with the global posture evolution features along the channel dimension to generate a fused motion feature for the time window segment. Finally, the fused motion features of each time window segment are processed by maximum pooling in the time dimension to obtain the motion trajectory features of the skeleton key points.

[0022] At the same time, instrument detection is performed on the continuous video clip sequence. In the tobacco leaf sorting scenario, tobacco station personnel may use sorting tools, such as specialized tweezers or a small rake. The category and spatial location of the handheld instrument in each frame are identified. For example, in a particular frame, a tobacco station employee is detected holding tweezers, and the coordinate range of the tweezers in the image is determined (x3, y3, x4, y4). Based on the category and spatial location of the handheld instrument, an instrument motion trajectory vector is constructed, such as the trajectory of tweezers moving from the left to the right side of the tobacco leaf pile. This instrument motion trajectory vector is input into the instrument interaction analysis network to generate the interaction features of the operating instrument.

[0023] Step S130 , performing dynamic matching processing based on the motion trajectory features of the skeleton key points and the features of the preset compliant action template to generate a local action deviation score of the video clip sequence.

[0024] Continuing with the tobacco leaf sorting process as an example, this embodiment pre-establishes a set of standard action template features corresponding to tobacco leaf sorting. This set of standard action template features contains template feature vectors for multiple standard action segments. For example, a standard tobacco leaf sorting action might involve reaching out to pick up a leaf, then turning it over in your hand to examine it, and then placing it into the appropriate container. This entire process is broken down into several standard action segments, each with a corresponding template feature vector.

[0025] The dynamic time warping distance (DTD) between the skeletal keypoint motion trajectory features and each template feature vector is calculated. Assuming the skeletal keypoint motion trajectory features represent the actual tobacco station worker sorting tobacco leaves, when compared with the standard template feature vector of reaching out to pick up tobacco leaves, a specific algorithm is used to calculate the dynamic time warping distance between the two. This DTD distance reflects the degree of temporal and spatial differences between the actual and standard movements.

[0026] The DTC distance is used to determine the matching similarity score between the motion trajectory features of the skeleton keypoints and the feature vectors of each template. If the DTC distance is small, it means that the actual movement is closer to the standard movement, and the matching similarity score will be high; conversely, if the DTC distance is large, the matching similarity score will be low.

[0027] Finally, the matching similarity score is denormalized to generate a local action deviation score. For example, the matching similarity score originally ranges from 0 to 1. After denormalization, when the matching similarity score is close to 1, the local action deviation score is close to 0, indicating that the action deviates slightly from the standard action. When the matching similarity score is close to 0, the local action deviation score is close to 1, indicating that the action deviates significantly from the standard action.

[0028] Step S140 : Inputting the operation instrument interaction features into a pre-trained violation feature classification network to generate a violation action feature set associated with the local action deviation score.

[0029] In this example, in the tobacco leaf sorting scenario, the previously obtained manipulation tool interaction features (such as the tweezers' trajectory and interaction with the tobacco leaves) and the local action deviation score can be concatenated across feature dimensions to generate a fused violation feature vector. Assuming the manipulation tool interaction feature is a multidimensional vector (a1, a2, a3, ...), and the local action deviation score is a single value d, the resulting fused violation feature vector is (a1, a2, a3, ..., d).

[0030] The multi-scale attention module in the violation feature classification network is invoked to perform multi-granular feature extraction on the fused violation feature vector, generating multi-level violation features. For example, one level might focus on subtle motion features when using tweezers, while another level might focus on the interaction between the tweezers and the tobacco leaf as a whole, as well as the impact of the motion deviation score on the overall probability of a violation.

[0031] Based on the multi-level violation features, a violation feature pyramid structure is constructed. Binary classification prediction is performed at each level of the violation feature pyramid structure to generate a hierarchical violation probability distribution. During tobacco sorting, the probability of improper tweezers use at one level might be predicted to be 0.3, while at another level, the probability of sorting errors due to deviations in the operating process might be predicted to be 0.2, and so on.

[0032] A weighted voting process is performed based on the hierarchical violation probability distribution of each level to generate a set of violation action features. For example, different weights are assigned based on the importance of different levels. The violation probability of each level is multiplied by the corresponding weight and then added together to obtain the final set of violation action features. This set of violation action features contains comprehensive assessment information for various possible violations.

[0033] Step S150 , performing violation identification processing on the video clip sequence according to the violation action feature set, generating a violation identification result including a violation type label and time location information, and sending the violation identification result to the tobacco station supervision terminal.

[0034] In this embodiment, a sliding window scanning process is performed on the feature set of the illegal action in the time dimension. Taking a video clip sequence of sorting tobacco leaves as an example, the sliding window size is set to 10 seconds and the step size is set to 5 seconds, thereby generating multiple candidate illegal time periods.

[0035] For each candidate violation time period, a subset of violation action features within the corresponding time range is extracted. For example, within a 10-second candidate violation time period, a subset of violation action features related to tweezers use and deviation in movement is extracted.

[0036] The feature subset of the illegal action is input into the violation classification decision tree model to generate a preliminary violation type prediction result. Assume that the decision tree model preliminarily judges the violation of tweezers use based on the input feature subset.

[0037] Then, based on the initial violation type prediction, a multi-evidence verification mechanism is triggered. When a device usage violation is predicted (e.g., a tweezers violation), the interaction characteristics of the device during the corresponding time period are verified to match the preset list of prohibited devices. In this example, the tweezers are checked to see if they fall within the permitted device range. If so, the violation may require reassessment; if not, the violation is confirmed. When a motion amplitude violation is predicted, the local motion deviation score during the corresponding time period is verified to see if it exceeds a dynamically adjusted deviation threshold. If the previously calculated local motion deviation score exceeds the dynamic deviation threshold calculated based on statistics from all operators in the tobacco station over a preset period, a motion amplitude violation is confirmed. When an operational process violation is predicted, the matching status of compliant motion templates across multiple consecutive video clips during the corresponding time period is verified to see if they violate the process logic constraints. For example, in tobacco leaf sorting, one grade of tobacco should be sorted first, followed by another. If this order is reversed, an operational process violation is confirmed.

[0038] Based on the verification results, the initial violation type prediction is modified to generate a final violation type label and time location information. For example, after verification, if a violation is determined to be the use of tweezers and the time period between 10:15:30 AM and 10:25:30 AM, the violation identification result, including the violation type label (tweezers use violation) and time location information (10:15:30 AM - 10:25:30 AM), is sent to the tobacco station supervisory terminal. Based on this violation identification result, tobacco station supervisors can take appropriate action regarding the illegal operation, such as providing training to the offending personnel or correcting the operating procedures.

[0039] Based on the above steps, the embodiment of the present application realizes the accurate and efficient violation identification of tobacco leaf operation behavior of tobacco station personnel, and significantly improves the intelligent level and safety assurance capability of tobacco station production management. First, a continuous video data set of the target tobacco station personnel within a preset time period is obtained. By extracting the spatiotemporal features of the video data, not only the motion trajectory features of the key points of the skeleton of the personnel are accurately captured, but also the interaction features with the operating equipment are deeply excavated, and the operating behavior patterns of the personnel are comprehensively portrayed from multiple dimensions, effectively overcoming the recognition limitations caused by a single feature. Based on the dynamic matching processing of the motion trajectory features of the key points of the skeleton and the compliant action template, the degree of deviation of the local action can be quantified in real time. Furthermore, the interaction features of the operating equipment are input into the pre-trained violation feature classification network to generate a violation action feature set that is closely related to the local action deviation score, thereby realizing deep feature mining and intelligent classification of the violation behavior, and greatly improving the accuracy and reliability of violation identification. Finally, the video clip sequence was processed for violation identification based on the set of violation action features. The generated violation identification results not only included clear violation type labels, but also accurately located the time information of the violation, effectively solving the problem that the traditional violation identification method of tobacco station personnel relies on manual inspections, is inefficient and prone to missed detections.

[0040] In a possible implementation, step S120 includes:

[0041] Step S121 , performing human posture estimation processing on each frame image in the continuous video clip sequence to generate a skeleton key point sequence containing three-dimensional coordinates.

[0042] In this embodiment, in a video clip of tobacco station workers sorting tobacco leaves, each frame contains the worker's body information. Skeletal key points can be determined using a pre-defined human pose estimation algorithm. For example, for the worker's arm, the shoulder joint might be identified as a skeletal key point, with its 3D coordinates perhaps being (x1, y1, z1). The elbow joint, another skeletal key point, might have coordinates of (x2, y2, z2). This process can be repeated to determine 3D coordinates for each key part of the human body, thereby forming a skeletal key point sequence containing 3D coordinates.

[0043] Step S122: performing time dimension interpolation processing on the skeleton key point sequence to obtain an interpolated skeleton key point sequence with equally spaced time steps.

[0044] In the embodiment of the present invention, the skeleton key point sequence of the present invention is recorded in the time step of the skeleton key point sequence.For example, in the original video shooting process, due to equipment or other factors, the record of the skeleton key point sequence in time may be uneven.Suppose that in a certain time period, there was a skeleton key point record per 0.2 second when starting, and the middle part may become a record per 0.3 second.The present embodiment makes the time step even by interpolation.For example, suppose that the present embodiment will obtain the interpolation skeleton key point sequence of equidistant time step per 0.1 second.The present embodiment determines the time interval variation in the original sequence earlier, and then according to the positional relationship and the time interval of adjacent key points, calculates the skeleton key point coordinates of the middle missing or needing to adjust the time point by linear interpolation or other suitable interpolation algorithm. For example, between two known time points t1 and t2 and the corresponding skeletal key point coordinates p1 and p2, to obtain the skeletal key point coordinate p3 of the intermediate time point t3, this embodiment can be calculated according to the time ratio. Assuming that the time difference from t1 to t2 is Δt, and the time difference from t1 to t3 is Δt', the coordinate calculation method of p3 is that for each coordinate component (such as the x coordinate), p3_x=p1_x+(p2_x-p1_x)×(Δt' / Δt). Each coordinate component is processed through such calculation to obtain an interpolated skeletal key point sequence with equally spaced time steps.

[0045] In step S123, a pre-trained spatiotemporal feature encoder is called to perform spatiotemporal convolution encoding on the interpolated skeleton key point sequence to generate the skeleton key point motion trajectory features, and simultaneously perform instrument detection on the continuous video clip sequence to identify the category and spatial position information of the handheld instrument in each frame image.

[0046] Step S124 : constructing an instrument motion trajectory vector based on the category and spatial position information of the handheld instrument, and inputting the instrument motion trajectory vector into an instrument interaction analysis network to generate the instrument operation interaction feature.

[0047] In a possible implementation, step S123 includes:

[0048] Step S1231: Divide the interpolated skeleton key point sequence into multiple time window segments.

[0049] For example, assume that the interpolated skeleton key point sequence corresponding to a video of sorting tobacco leaves within a day is divided into segments of 10 seconds each.

[0050] Step S1232: Perform a dual-stream feature extraction operation in each time window segment:

[0051] A. The first-stream processing path performs one-dimensional convolution encoding on the relative displacement between adjacent skeletal key points to generate local motion pattern features.

[0052] For example, in the process of tobacco leaf sorting, taking the arm as an example, the shoulder and elbow joints are adjacent skeletal key points. Within each time window, the relative displacement between these two joints changes over time. Assuming that within a 10-second time window, this embodiment records the coordinates of these two joints at 100 time points, calculates the relative displacements between adjacent time points, such as the relative displacement from the first time point to the second time point being (x2-x1, y2-y1, z2-z1), and then performs one-dimensional convolution encoding on these relative displacement sequences. This encoding process can be understood as a weighted summation of a series of numbers. Using a predefined convolution kernel (e.g., a convolution kernel of length 5 with weights w1, w2, w3, w4, and w5), starting from the beginning of the sequence, the convolution kernel is multiplied by five consecutive data points in the relative displacement sequence, and then the sum is performed to obtain a new numerical sequence. This new numerical sequence represents the local motion pattern characteristics.

[0053] B. The second-stream processing path performs graph convolution encoding on the spatial distribution of all skeletal key points in the time window to generate global posture evolution features.

[0054] For example, within a 10-second time window, the spatial distribution of all skeletal key points (including key points of the head, shoulders, arms, waist, legs, etc.) changes with the action of sorting tobacco leaves. This embodiment constructs a graph structure, with each skeletal key point as a node in the graph, and adjacent key points are connected by edges. For graph convolutional coding, this embodiment encodes based on the relationship between each node and its neighboring nodes and the coordinate information of the node itself. For example, for a certain skeletal key point, this embodiment considers its distance relationship with several adjacent key points, and then performs a weighted sum calculation on each node according to the predefined graph convolution rules, and updates the node information in the entire graph structure. After multiple such iterative calculations, a global posture evolution feature that can represent the changes in the spatial distribution of all skeletal key points within the entire time window is obtained.

[0055] Step S1233 , performing channel dimension splicing on the local motion pattern feature and the global posture evolution feature to generate a fused motion feature of the time window segment.

[0056] Assuming that the local motion pattern feature is a vector with a dimension of m, and the global posture evolution feature is a vector with a dimension of n, this embodiment concatenates these two vectors in sequence to form a fused motion feature vector with a dimension of m+n.

[0057] Step S1234 , performing time dimension maximum pooling processing on the fused motion features of each time window segment to generate the motion trajectory features of the skeleton key points.

[0058] For example, after obtaining fused motion features to each 10-second time window segment, the present embodiment performs maximum pooling processing in the time dimension. For example, if there are multiple 10-second time window segments, each time window segment has a corresponding fused motion feature vector. For the elements in the same position in each fused motion feature vector, the maximum value thereof can be selected as the element in the final skeleton key point motion trajectory feature vector. Such as the first element of the fused motion feature vector of the first time window segment is a1, and the first element of the fused motion feature vector of the second time window segment is a2. If a1>a2, then the first element of the final skeleton key point motion trajectory feature vector is a1. In this way, each element is processed to obtain the skeleton key point motion trajectory feature.

[0059] At the same time, during this process, instrument detection processing is synchronously performed on a sequence of continuous video clips. In the scenario of tobacco leaf sorting, tobacco station personnel will hold sorting tools, such as tweezers. Therefore, each frame of the image can be detected, the handheld instrument can be identified as a tweezers category, and its spatial position information in the image can be determined. For example, through an image recognition algorithm, the coordinates of the upper left corner of the tweezers in the image are determined to be (x3, y3), and the coordinates of the lower right corner are determined to be (x4, y4), which clarifies the spatial position of the tweezers in the image. The instrument motion trajectory vector is constructed based on the category of the handheld instrument (tweezers) and the spatial position information. Over time, the position of the tweezers in the image will change. This embodiment constructs a vector sequence based on the position information of the tweezers at each time point. This vector sequence represents the instrument motion trajectory vector. The instrument motion trajectory vector is then input into the instrument interaction analysis network to generate the interaction features of the operating instrument. The device interaction analysis network generates device interaction features through internal calculation and processing based on factors such as the position change information in the device motion trajectory vector and its relationship with other objects (such as tobacco leaves). The device interaction features can reflect the operation of the tweezers during the tobacco leaf sorting process, such as the frequency of contact with the tobacco leaves, the relationship between the movement direction and the tobacco leaf stacking direction, etc.

[0060] In a possible implementation, step S130 includes:

[0061] Step S131 : obtaining a set of compliant action template features corresponding to the current tobacco operation process, wherein the set of compliant action template features includes template feature vectors of a plurality of standard action segments.

[0062] In the tobacco leaf sorting process, standard sorting actions are broken down into multiple standard action segments, and corresponding template feature vectors are created. For example, a complete sorting action can be divided into several standard action segments: reaching out to pick up the tobacco leaf, turning the leaf over in the hand, and placing the leaf into the appropriate container. For the action segment of reaching out to pick up the tobacco leaf, its template feature vector includes the ideal motion trajectory features of key skeletal points such as the hand and arm, as well as the relationship features between these key points. These feature vectors are derived from the analysis and extraction of a large number of correct sorting actions and serve as the standard basis for judging whether the sorting action is compliant.

[0063] Step S132, calculating the dynamic time warping distance between the motion trajectory feature of the skeleton key point and each of the template feature vectors.

[0064] In a video clip of tobacco leaf sorting, the actual skeletal keypoint motion trajectory features of tobacco station personnel are compared with the template feature vectors of the standard action clips. Taking the action clip of reaching for tobacco leaves as an example, assume that the skeletal keypoint motion trajectory features are represented by the time-varying coordinate sequence of the skeletal keypoints of the actual sorter's arm and hand, while the template feature vector is the coordinate sequence corresponding to the standard reaching for tobacco leaf action. When calculating the dynamic time warping distance, the alignment of the time series needs to be considered. Therefore, starting from the starting point of the time series, different time alignment methods are tried to find the minimum distance between the skeletal keypoint motion trajectory features and the template feature vector. For example, assume that the first time point in the skeletal keypoint motion trajectory features is aligned with the first time point in the template feature vector. Then, at each time point, the sum of the squares of the differences between the skeletal keypoint coordinates and the corresponding coordinates in the template feature vector are calculated to obtain a distance value. Then, align the first time point in the skeletal keypoint motion trajectory features with the second time point in the template feature vector, and repeat the above calculation process to obtain another distance value. Similarly, we traverse all possible time alignments and find the minimum distance value. This minimum distance value is the dynamic time warping distance. The specific calculation process is as follows: assuming that the hand coordinates of the skeleton key point motion trajectory feature at a certain time point t1 are (x1, y1, z1), and the hand coordinates of the template feature vector at the corresponding time point (assuming it is t1') are (x1', y1', z1'). Then the distance at this time point is calculated as (x1-x1')²+(y1-y1')²+(z1-z1')². The distance calculation results of all time points are summed to obtain the distance value under one time alignment method. Then, the distance values under different time alignment methods are compared and the minimum value is selected as the dynamic time warping distance.

[0065] Step S133: determining a matching similarity score between the motion trajectory feature of the skeleton key point and each of the template feature vectors according to the dynamic time warping distance.

[0066] Generally speaking, the smaller the dynamic time warping distance, the closer the actual skeletal keypoint motion trajectory is to the template feature vector, and the higher the matching similarity score. For example, a functional relationship can be set such that when the dynamic time warping distance is 0, the matching similarity score is set to 1, indicating a perfect match; when the dynamic time warping distance reaches a certain maximum value (this maximum value is determined based on a large number of experiments and experience, for example, in the tobacco sorting scenario, when the dynamic time warping distance reaches 100), the matching similarity score is set to 0, indicating a complete mismatch. Within this range, the matching similarity score is inversely proportional to the dynamic time warping distance. Assuming the dynamic time warping distance is d and the maximum distance value is D, the matching similarity score s is calculated as s = 1-d / D. Through the above calculation, the matching similarity score between the skeletal keypoint motion trajectory features and each template feature vector can be obtained.

[0067] Step S134 : performing reverse normalization processing on the matching similarity score to generate the local action deviation score, wherein the local action deviation score is negatively correlated with the matching similarity score.

[0068] Assuming that the matching similarity score ranges from 0 to 1, reverse normalization is required. For example, the original matching similarity score is 0.8, which means that the action is relatively close to the standard action. The local action deviation score after reverse normalization is calculated as 1-0.8=0.2, which means that the degree of deviation of the action from the standard action is small. If the matching similarity score is 0.2, the local action deviation score after reverse normalization is 1-0.2=0.8, which means that the degree of deviation of the action from the standard action is large. In this way, the local action deviation score is obtained, which can intuitively reflect the degree of deviation of the actions of tobacco station personnel from the standard actions during the tobacco leaf sorting process.

[0069] In a possible implementation, step S140 includes:

[0070] Step S141 : performing feature dimension splicing on the local action deviation score and the operation device interaction feature to generate a fused violation feature vector.

[0071] In the tobacco leaf sorting scenario, it is assumed that the interaction feature of the operating instrument is a multidimensional vector containing information such as the movement trajectory of the tweezers during the sorting process and the frequency of contact with the tobacco leaves, for example (a1, a2, a3, ...), and the local action deviation score is a value d. They are spliced together to obtain the fused violation feature vector (a1, a2, a3, ..., d), which integrates information on both the interaction situation of the operating instrument and the degree to which the action deviates from the standard action.

[0072] Step S142: calling the multi-scale attention module in the violation feature classification network to perform multi-granularity feature extraction processing on the fused violation feature vector to generate multi-level violation features.

[0073] The multi-scale attention module can focus on information in the fused violation feature vector from different scales and perspectives. For example, at a finer-grained level, it might focus on the angle and force of contact between the tweezers and the tobacco leaf at a given moment (this information can be inferred from some elements of the operating instrument interaction features and the impact of the local action deviation score on the operation); at a coarser-grained level, it might focus on the difference between the overall motion trajectory of the tweezers during the entire sorting process and the standard trajectory, as well as the impact of the action deviation on the overall sorting effect. Through this multi-granular feature extraction process, multi-level violation features are generated, which can comprehensively reflect possible violations.

[0074] Step S143 : constructing a violation feature pyramid structure based on the multi-level violation features, and performing binary classification prediction processing on each level of the violation feature pyramid structure to generate a level-by-level violation probability distribution.

[0075] In the violation feature pyramid structure, each level has its own specific violation feature representation. For example, at the lowest level, the focus may be on whether the most basic operational elements, such as whether the tweezers are correctly handled, are in violation. At higher levels, the focus may be on whether there are any combinations of violations across multiple operational steps in the entire sorting process. At each level, a binary prediction process is performed to predict whether a violation has occurred and obtain a probability value. Suppose that at a certain level, the probability of a violation calculated based on the violation features at that level is 0.3, indicating that there is a 30% probability of a violation at that level. By performing this prediction process at each level, a hierarchical violation probability distribution is obtained.

[0076] Step S144 , performing weighted voting processing according to the hierarchical violation probability distribution of each level to generate the violation action feature set.

[0077] In this embodiment, different weights can be set according to the importance of each level. For example, the lower level focuses on basic operations, and its weight may be lower, set as w1; the higher level focuses on the overall process, and its weight may be higher, set as w2. Assuming that the violation probability of a lower level is p1, and the violation probability of a higher level is p2, the violation probability after weighted voting is calculated as p=w1×p1+w2×p2. By performing weighted summation processing on the violation probabilities of each level in this way, a comprehensive violation probability value is obtained. This violation probability value and the related violation feature information constitute a violation action feature set, which can comprehensively and accurately reflect the possible violations in the tobacco leaf sorting process.

[0078] In one possible implementation, the training method of the violation feature classification network includes:

[0079] Step S210 , collecting a historical tobacco leaf operation violation video dataset, and performing violation type labeling processing on the historical tobacco leaf operation violation video dataset to generate a training sample set with multi-level violation labels.

[0080] In this embodiment, video data of tobacco leaf sorting violations can be obtained from the historical records of the tobacco station. These video data cover various types of violations. Among them, for each video, detailed violation type labeling is required to construct multi-level violation labels. For example, in the tobacco leaf sorting process, the lowest level violation type may be the incorrect use of tweezers, such as improper picking method; the middle-level violation type may be an error in a certain link in the sorting process, such as failure to sort in the order of tobacco leaf grade; the highest level violation type may be that the overall operation seriously violates regulations, affecting the quality of tobacco leaves. For each illegal operation in the video, accurately mark its violation type at each level, thereby generating a training sample set with multi-level violation labels.

[0081] Step S220 , performing spatiotemporal feature extraction processing on each sample in the training sample set to generate sample skeleton key point motion trajectory features and sample operation instrument interaction features.

[0082] In a possible implementation, step S220 includes:

[0083] Step S221 : performing key frame sampling processing on the historical tobacco leaf operation violation video dataset to generate a sampling frame sequence with equal time intervals.

[0084] In these historical violation videos, due to the high frame rate and large data volume, key frame sampling is necessary to improve processing efficiency while preserving critical information. For example, if the original video frame rate is 30 frames per second, this embodiment sets the sampling interval to one frame every three seconds. This selects a sequence of sampled frames with equal time intervals from the entire video.

[0085] Step S222: performing data enhancement processing on the sampling frame sequence to obtain the enhanced sampling frame sequence, wherein the data enhancement processing includes at least one of the following operations: temporal random cropping, spatial random flipping, and skeleton key point jitter injection.

[0086] Taking temporal random cropping as an example, a time period can be randomly selected in the time dimension, and the frames within this time period are retained, while the remaining frames are discarded. Assuming that the original sampling frame sequence has 100 frames, the time period from frame 30 to frame 60 can be randomly selected, and these 30 frames can be retained. Spatial random flipping is a random flipping operation performed on the spatial dimension of the image, such as flipping around the vertical axis of the image. Skeletal key point jitter injection is to add a small random perturbation to the coordinates of the skeletal key points. For example, if the original coordinates of a skeletal key point are (x, y, z), a random number between -0.1 and 0.1 can be added to the x coordinate, and the same applies to the y and z coordinates. In this way, an enhanced sampling frame sequence is obtained.

[0087] Step S223 , calling a pre-trained lightweight posture estimation model, performing real-time skeleton key point detection processing on the enhanced sampling frame sequence, and generating an enhanced skeleton key point sequence.

[0088] In a possible implementation, step S223 includes:

[0089] Step S2231: input the enhanced single-frame image into the feature extraction backbone network of the lightweight pose estimation model to generate a multi-scale image feature map.

[0090] In the tobacco sorting scenario, an enhanced single-frame image captures the posture of tobacco station personnel performing sorting operations, as well as information about the surrounding environment. The lightweight pose estimation model's feature extraction backbone processes this image frame, a process that can be understood as a gradual abstraction and feature extraction process. For example, the feature extraction backbone network may include multiple convolutional and pooling layers. Convolutional layers perform convolution operations by sliding a specific convolution kernel across the image to extract local features. For example, a 3x3 convolution kernel moves one pixel at a time, starting from the top left corner of the image. The kernel then performs weighted summation on the 3x3 pixel region it covers to generate a new pixel value. After multiple such convolutional layers, image features are gradually extracted. Simultaneously, pooling layers downsample the image. For example, a max pooling layer selects the maximum value within a small region (e.g., a 2x2 pixel region) as the representative value for that region, thereby reducing the image size while preserving key features. This combination of convolutional and pooling layers ultimately generates a multi-scale image feature map. This means that in different layers, due to the different convolution kernel sizes, strides, and pooling methods, image feature maps of different scales will be obtained. The above image feature maps can reflect the feature information of the image at different resolutions.

[0091] Step S2232: Perform heat map regression processing on the multi-scale image feature map to generate a heat map distribution of each skeletal key point.

[0092] For images of tobacco station personnel in tobacco leaf sorting scenarios, each skeletal key point (such as head, shoulders, arms, hands, etc.) has a corresponding feature representation on the multi-scale image feature map. The heat map regression process aims to find the possible position distribution of each skeletal key point in the image. Specifically, the model generates a heat map for each skeletal key point based on the pre-learned knowledge and the feature information in the current multi-scale image feature map. The heat map is a two-dimensional image in which the value of each pixel represents the probability of the point being a skeletal key point. For example, for the heat map of the skeletal key points of the hand, the values of the pixels near the actual location of the hand will be higher, while the values of the pixels far away from the hand will be lower, thus forming a peak-like numerical distribution centered on the hand. This numerical distribution is the heat map distribution, which represents the probability distribution of the possible positions of the skeletal key points of the hand in the image.

[0093] Step S2233: Perform non-maximum suppression processing on the heat map distribution to determine the two-dimensional coordinate position of each skeleton key point.

[0094] After obtaining the heatmap distribution of each skeletal key point, since there may be multiple local maxima in the heatmap, these local maxima may interfere with the accurate determination of the position of the skeletal key point in this embodiment. Non-maximum suppression is used to eliminate these interferences. For example, for the heatmap of the hand skeletal key points, the point with the largest value can be found from all the pixels in the heatmap. This point is initially considered to be the position of the hand skeletal key point. However, there may be some points with relatively large values around this point, which may be misjudgments due to noise or other factors. The non-maximum suppression process sets a threshold or comparison rule. When the value of the surrounding points is less than the threshold or the ratio to the maximum point is less than a certain set value, these surrounding points are excluded, and finally a unique point is determined as the two-dimensional coordinate position of the hand skeletal key point. The same method is used for other skeletal key points (such as the shoulder, arm, etc.). By performing non-maximum suppression on their respective heatmap distributions, the two-dimensional coordinate position of each skeletal key point in the image is accurately determined.

[0095] Step S2234 , combining the optical flow information between adjacent frames of the sampling frame sequence, performing time domain smoothing processing on the two-dimensional coordinate position to generate a three-dimensional skeleton key point coordinate sequence.

[0096] In a tobacco leaf sorting video, the sampled frame sequence consists of multiple consecutive frames. The optical flow information between adjacent frames reflects the motion of objects in the image (here, the body parts of tobacco station personnel). For example, from one frame to the next, the tobacco station personnel's hand may move from one position to another. The optical flow information can describe information such as the direction and speed of this movement. Therefore, the previously determined 2D coordinate positions of skeletal key points can be temporally smoothed based on the optical flow information. Assume that the 2D coordinate positions of the hand skeletal key points in a certain frame have been determined as (x, y). In the next frame, the optical flow information indicates that the hand has moved rightward and downward. The distance of movement can be estimated based on the speed information in the optical flow information. Assume that the hand has moved by Δx horizontally and Δy vertically. Furthermore, this embodiment can also obtain depth information Δz based on some prior knowledge or depth estimation methods (for example, estimating depth based on the distance between the tobacco station personnel and the camera and the perspective relationship in the image). The 3D coordinate positions of the hand skeletal key points in the next frame can then be calculated as (x + Δx, y + Δy, z + Δz). By processing each skeletal key point in this way between adjacent frames in combination with optical flow information, the two-dimensional coordinate position can be converted into a three-dimensional skeletal key point coordinate sequence. This three-dimensional skeletal key point coordinate sequence can more accurately describe the changes in the body posture of tobacco station personnel during the tobacco leaf sorting process, and provide more comprehensive and accurate skeletal key point motion information for subsequent analysis (such as determining whether the action is compliant, etc.).

[0097] Step S224, input the enhanced skeletal key point sequence into the training copy of the spatiotemporal feature encoder to generate enhanced skeletal key point motion trajectory features, and simultaneously perform instrument detection processing on the enhanced sampling frame sequence to generate enhanced instrument motion trajectory vectors, and input the enhanced instrument motion trajectory vectors into the training copy of the instrument interaction analysis network to generate enhanced operation instrument interaction features.

[0098] In this embodiment, in the training copy of the spatiotemporal feature encoder, the method of processing the enhanced skeleton key point sequence is similar to the spatiotemporal feature encoder in the actual operation. By performing time window division, dual-stream feature extraction and other operations on the enhanced skeleton key point sequence, the enhanced skeleton key point motion trajectory feature is generated. At the same time, the enhanced sampling frame sequence is subjected to instrument detection processing to identify the handheld instrument (such as tweezers) in the image, determine its category and spatial position information in each frame, and construct an enhanced instrument motion trajectory vector. For example, in a sampling frame sequence containing 10 frames, the position coordinates of the tweezers in each frame are determined to obtain the motion trajectory vector of the tweezers. Then, the enhanced instrument motion trajectory vector is input into the training copy of the instrument interaction analysis network. After calculation and analysis within the network, the enhanced operation instrument interaction feature is generated. The enhanced operation instrument interaction feature reflects the operational interaction of the tweezers in the process of sorting tobacco leaves.

[0099] Step S230 , generating a sample local motion deviation score based on the matching result of the sample skeleton key point motion trajectory feature and the compliant motion template feature.

[0100] In this embodiment, the compliance action template features corresponding to tobacco leaf sorting can be obtained, and the compliance action template features include information such as the motion trajectory of the skeleton key points under the correct sorting action. The dynamic time warping distance between the sample skeleton key point motion trajectory features and the compliance action template features is calculated, and the calculation process is similar to the calculation in the actual operation before. For example, the motion trajectory of the arm skeleton key points when the tobacco station staff in the sample sorts tobacco leaves is compared with the difference in the motion trajectory of the arm skeleton key points in the compliance action template, and the minimum distance is found by trying different time alignment methods. The matching similarity score is determined based on the dynamic time warping distance. For example, according to the previously set calculation method, if the dynamic time warping distance is d and the maximum distance is D, the matching similarity score s=1-d / D. Finally, the matching similarity score is reverse normalized to obtain the sample local action deviation score. For example, if the matching similarity score is 0.6, the sample local action deviation score after reverse normalization is 1-0.6=0.4.

[0101] Step S240 : Concatenate the sample local action deviation score with the sample operation instrument interaction feature to generate a sample fusion violation feature vector.

[0102] Assuming that the sample operation instrument interaction feature is a multidimensional vector (a1, a2, a3, ...), and the sample local action deviation score is d, they are spliced together to obtain the sample fusion violation feature vector (a1, a2, a3, ..., d).

[0103] Step S250: construct an initial violation feature classification network including a multi-scale attention module, input the sample fusion violation feature vector into the initial violation feature classification network, and generate a predicted violation probability distribution.

[0104] The multi-scale attention module within the initial violation feature classification network focuses on information in the sample-fused violation feature vector at different scales. Within the network, through multi-layer neural network calculations, the probability of a violation is predicted at different levels based on the information in the sample-fused violation feature vector, generating a predicted violation probability distribution. For example, at one level, the probability of a tweezers violation is predicted to be 0.3, while at another level, the probability of a sorting process violation is predicted to be 0.2, and so on.

[0105] Step S260 , training the initial violation feature classification network according to the cross entropy loss between the predicted violation probability distribution and the multi-level violation labels until the loss converges, thereby obtaining the violation feature classification network.

[0106] In this embodiment, when calculating the cross-entropy loss, the predicted violation probability distribution of each sample and its corresponding multi-level violation label are calculated according to the cross-entropy calculation method. For example, for a sample, the predicted violation probability at a certain level is p, and the actual violation label is y (y is 0 or 1, indicating no violation or violation), then the cross-entropy loss at this level is calculated as -[y×log(p)+(1-y)×log(1-p)]. The cross-entropy loss of the entire sample is obtained by summing the cross-entropy losses of all levels. During the training process, by continuously adjusting the parameters in the initial violation feature classification network, the cross-entropy loss of the entire training sample set is gradually reduced until the loss converges, that is, the loss no longer decreases significantly. At this time, the network obtained is the trained violation feature classification network.

[0107] In a possible implementation, step S150 includes:

[0108] Step S151 : performing a time dimension sliding window scanning process on the illegal action feature set to generate a plurality of candidate illegal time periods.

[0109] In the video clip sequence of tobacco leaf sorting, the illegal action feature set contains various feature information related to possible violations in the entire sorting process. In order to determine the specific time period when the violation occurred, a time dimension sliding window scanning process is adopted. For example, the sliding window size is set to 10 seconds and the step length is 5 seconds. Starting from the start time of the video clip sequence, the first sliding window covers the time period of 0-10 seconds. The illegal action feature set is analyzed within this sliding window to obtain the relevant feature information within this time period. The sliding window then moves in steps of 5 seconds, and the next window covers the time period of 5-15 seconds, and so on, until the entire video clip sequence is covered. In this way, multiple candidate violation time periods are generated, each of which contains the illegal action feature information within this time period, providing a time range limitation for subsequent further analysis.

[0110] Step S152 : for each candidate violation time period, extract a subset of violation action features within the corresponding time range.

[0111] For each candidate violation time period, such as 10-20 seconds, a feature subset related to that time period is extracted from the set of illegal action features. This feature subset contains various feature information related to the tobacco station personnel's operations during that time period, such as the local action deviation score and the interaction characteristics of the operating equipment during that time period. These feature subsets can reflect the operation of the tobacco station personnel during that specific time period and provide targeted information for determining whether there has been any illegal behavior.

[0112] Step S153: input the illegal action feature subset into the violation classification decision tree model to generate a preliminary violation type prediction result.

[0113] In the tobacco leaf sorting scenario, a violation classification decision tree model is constructed based on a large amount of historical data and known violations. A subset of violation action features corresponding to each candidate violation time period is input into the violation classification decision tree model. For example, for the subset of violation action features for the 10-20 second time period, the model analyzes the violation based on the relationships between the individual features within the feature subset and the model's predefined decision rules. Assume that the model's decision rules determine the violation type based on the value range or combination of different features. If, within this feature subset, the instrument interaction features indicate that the tobacco station personnel used tweezers in a manner significantly different from normal usage during that time period, and the local action deviation score is also high, the model may preliminarily determine that an instrument usage violation occurred during that time period based on the relationships between these features and the predefined decision rules. The same analysis is performed for other candidate violation time periods to generate preliminary violation type predictions.

[0114] Step S154: triggering a multi-evidence verification mechanism based on the preliminary violation type prediction result:

[0115] When a device usage violation is predicted, the matching degree between the interaction characteristics of the device during the corresponding time period and the preset list of prohibited devices is verified.

[0116] For example, in a tobacco leaf sorting scenario, if the initial prediction is that there is an instrument use violation within a certain time period (such as 15-25 seconds), it is assumed that the tobacco station staff used a special type of tweezers during this time period, and the preset list of banned instruments includes certain types of tweezers that do not comply with regulations. In this case, it is necessary to verify the matching degree between the interaction characteristics of the instrument operation during this time period and the list of banned instruments. The interaction characteristics of the instrument operation include information such as how the tweezers are used during this time period and the contact between them and the tobacco leaves. For example, check whether the shape and size of the tweezers are similar to certain types on the list of banned instruments. If similar features exist and these features meet the criteria for determining banned instruments, then it is further confirmed that there was an instrument use violation within this time period; if there are no similar features or the criteria for determining banned instruments are not met, then it is necessary to re-evaluate this preliminary prediction result of the violation type.

[0117] When the prediction is that the movement amplitude is in violation, check whether the local movement deviation score in the corresponding time period exceeds the dynamically adjusted deviation threshold.

[0118] For example, suppose a preliminary prediction indicates that a motion amplitude violation occurs between 20 and 30 seconds. First, the local motion deviation score for this time period must be determined. This local motion deviation score, obtained through previous calculations, reflects the degree to which the operator's movements during this time period deviate from the standard. Next, a dynamically adjusted deviation threshold is determined. To determine this deviation threshold, the distribution of local motion deviation scores for all operators in the current station over a preset period (e.g., one week) must be statistically analyzed. Assume that the local motion deviation score data for all operators during this week has been collected, forming a score distribution. The mean and standard deviation of this score distribution are calculated. For example, the mean is calculated by adding all local motion deviation scores and then dividing by the number of scores. To calculate the standard deviation, the square of the difference between each score and the mean is calculated. These squared values are summed and divided by the number of scores, and the square root is taken to obtain the standard deviation. A dynamic deviation threshold function is constructed based on the mean and standard deviation. For example, the dynamic deviation threshold can be set to the mean minus k times the standard deviation (k is a coefficient determined based on actual conditions and experience, such as k=1.5), so that the dynamic deviation threshold decreases as the standard deviation increases. If the local movement deviation score within the 20-30 second timeframe exceeds the dynamically adjusted deviation threshold, then a movement amplitude violation is confirmed within that timeframe; if it does not, the initial violation type prediction results need to be re-evaluated.

[0119] When an operation process violation is predicted, the compliance action template matching status of multiple consecutive video clip sequences in the corresponding time period is checked to see whether it violates the process logic constraints.

[0120] For example, suppose a preliminary prediction indicates an operational process violation in the 30-40 second timeframe. During tobacco leaf sorting, there is a standard operational process that is broken down into multiple consecutive actions, each with a corresponding compliance action template. During the 30-40 second timeframe, the operations of tobacco station personnel in multiple consecutive video clip sequences are checked to see if they comply with this operational process. For example, the standard operational process involves first picking up tobacco leaves from the stack, then sorting them, and finally placing the sorted leaves into the appropriate containers. If, in the video clip sequences within this timeframe, tobacco station personnel first place unsorted tobacco leaves into the container before sorting them, this violates the process logic constraints, confirming that an operational process violation occurred within this timeframe. If the operations comply with the process logic constraints, the preliminary violation type prediction results need to be re-evaluated.

[0121] Step S155 , correcting the preliminary violation type prediction result according to the verification result to generate a final violation type label and time location information.

[0122] In this embodiment, each initially predicted violation time period is modified based on the aforementioned verification results. For example, for the 15-25 second time period, if verification confirms an equipment use violation, the final violation type label is "equipment use violation," and the time location information is "15-25 seconds." All candidate violation time periods are processed in this manner, ultimately generating a violation identification result containing accurate violation type labels and time location information. This violation identification result accurately identifies the violations occurring during the tobacco sorting process and the time range within which they occurred.

[0123] The method for determining the dynamically adjusted deviation threshold includes:

[0124] Step S310: Count the distribution of local action deviation scores of all operators in the current tobacco station within a preset period.

[0125] In this embodiment, during the tobacco leaf sorting process, the tobacco station has multiple operators involved. In order to determine a reasonable dynamic deviation threshold, it is necessary to collect statistics on the local action deviation scores of all operators within a preset period (for example, one month). When each operator sorts tobacco leaves, his or her operating action is compared with the standard action template to obtain a local action deviation score. Assuming that there are 50 operators in the tobacco station, each sorting operation of each operator is monitored every day during this month and their local action deviation score is obtained. All these scores are collected to form a data set, which reflects the score distribution of the local action deviation of all operators in the current tobacco station within the preset period. The score distribution may be diverse. The scores of most operators may be concentrated in a certain interval, while the scores of a few operators may deviate from this interval. By collating and analyzing the entire data set, a full picture of the score distribution can be obtained.

[0126] Step S320 , calculating the mean and standard deviation of the local action deviation score distribution.

[0127] For the calculation of the mean, add up all the collected local action deviation scores and then divide them by the total number of scores. For example, add up all the local action deviation scores obtained by these 50 operators for all sorting operations in a month. Assume that the sum is S. Since there are a total of N scores (N is the total number of operations performed by the 50 operators in a month), the mean μ is equal to S divided by N. In the process of calculating the standard deviation, first, for each local action deviation score x, calculate its difference from the mean μ (x-μ), and then square the difference to get (x-μ)². Add up all (x-μ)², assume that the sum is T, and then divide it by the total number of scores N. Take the square root of the final value, and the result is the standard deviation σ. For example, if there is a score , the difference between it and the mean μ is , the square of the difference Participate in the calculation of the total T, and through this calculation of all scores, we finally get the standard deviation σ.

[0128] Step S330: constructing a dynamic deviation threshold function based on the mean and the standard deviation, so that the dynamic deviation threshold decreases as the standard deviation increases.

[0129] In this embodiment, the purpose of constructing the dynamic deviation threshold function is to dynamically adjust the deviation threshold according to changes in the score distribution. A common construction method is to set the dynamic deviation threshold to the mean minus k times the standard deviation, that is, threshold = μ-kσ, where k is a coefficient determined based on experience. When the standard deviation σ increases, the threshold will decrease because the subtracted value kσ increases. This is because an increase in the standard deviation means that the discreteness of the score increases, and there may be more abnormal situations. In order to more accurately identify movement amplitude violations, the threshold needs to be lowered to adapt to this change. For example, if k is set to 1.5, when the mean μ is 0.3 and the standard deviation σ is 0.1, the dynamic deviation threshold is 0.3-1.5×0.1=0.15; when the standard deviation increases to 0.2, the dynamic deviation threshold becomes 0.3-1.5×0.2=0.

[0130] Step S340 , monitoring the change trend of the local action deviation score distribution in real time, and recalculating the dynamic deviation threshold when a shift in the distribution form is detected.

[0131] During the ongoing tobacco sorting process, the distribution of local movement deviation scores may shift due to factors such as training, accumulated experience, or changes in the working environment. For example, newly recruited operators with lower proficiency may shift the overall score distribution away from standard movements. Alternatively, after a period of training, as operators become more standardized, the score distribution may shift closer to standard movements. Regularly (e.g., weekly) recalculate the local movement deviation score distribution for all operators and analyze whether there are significant changes in its shape compared to the previous distribution. If scores that were previously concentrated in a certain interval begin to spread to other intervals, or if the overall score value changes significantly, the distribution is considered to have shifted. Once such a shift is detected, the dynamic deviation threshold is recalculated according to the previous steps to ensure that it consistently adapts to the actual operating conditions of the tobacco station operators, thereby more accurately determining movement amplitude violations.

[0132] Figure 2A schematic diagram illustrates exemplary hardware and software components of a deep learning-based system 100 for identifying violations by tobacco station personnel, as provided in some embodiments of the present application, that can implement the concepts of the present application. For example, a processor 120 can be used in the deep learning-based system 100 for identifying violations by tobacco station personnel, and can be used to perform the functions described in the present application.

[0133] The deep learning-based tobacco station personnel violation identification system 100 can be a general-purpose server or a special-purpose server, both of which can be used to implement the deep learning-based tobacco station personnel violation identification method of this application. Although only one server is shown in this application, for convenience, the functions described in this application can be implemented in a distributed manner on multiple similar platforms to balance the processing load.

[0134] For example, the deep learning-based tobacco station personnel violation identification system 100 may include a network port 110 connected to a network, one or more processors 120 for executing program instructions, a communication bus 130, and storage media 140 in different forms, such as a disk, ROM, or RAM, or any combination thereof. Exemplarily, the deep learning-based tobacco station personnel violation identification system 100 may also include program instructions stored in ROM, RAM, or other types of non-transitory storage media, or any combination thereof. The method of the present application can be implemented according to these program instructions. The deep learning-based tobacco station personnel violation identification system 100 also includes an input / output (I / O) interface 150 between the computer and other input and output devices.

[0135] For ease of explanation, only one processor is described in the deep learning-based tobacco station personnel violation identification system 100. However, it should be noted that the deep learning-based tobacco station personnel violation identification system 100 in this application may also include multiple processors, so the steps performed by one processor described in this application may also be performed jointly or individually by multiple processors. For example, if the processor of the deep learning-based tobacco station personnel violation identification system 100 executes step A and step B, it should be understood that step A and step B may also be performed jointly by two different processors or individually in one processor. For example, the first processor executes step A, the second processor executes step B, or the first processor and the second processor execute steps A and B together.

[0136] In addition, an embodiment of the present invention also provides a readable storage medium, in which computer-executable instructions are preset. When the processor executes the computer-executable instructions, the above-mentioned method for identifying violations of tobacco station personnel based on deep learning is implemented.

[0137] It should be noted that in order to simplify the description of the present invention and thus help understand one or more embodiments of the invention, in the foregoing description of the embodiments of the present invention, multiple features are sometimes combined into one embodiment, figure or description thereof.

Claims

1. A method for identifying violations of tobacco station personnel based on deep learning, characterized in that: The method comprises: Acquire a tobacco leaf handling behavior video data set of a target tobacco station staff within a preset time period, wherein the tobacco leaf handling behavior video data set comprises a plurality of continuous video clip sequences; Performing spatiotemporal feature extraction processing on the tobacco leaf manipulation behavior video data set to obtain skeletal key point motion trajectory features and manipulation instrument interaction features of each video clip sequence; Dynamically matching the motion trajectory features of the skeleton key points with the features of the preset compliant action template to generate a local action deviation score for the video clip sequence; Inputting the operation instrument interaction features into a pre-trained violation feature classification network to generate a set of violation action features associated with the local action deviation score; Performing illegal behavior identification processing on the video clip sequence according to the illegal action feature set, generating an illegal behavior identification result including an illegal behavior type label and time location information, and sending the illegal behavior identification result to the tobacco station supervision terminal; The dynamic matching process based on the motion trajectory features of the skeleton key points and the preset compliance action template features to generate the local action deviation score of the video clip sequence includes: Acquire a compliance action template feature set corresponding to a current tobacco operation process, wherein the compliance action template feature set includes template feature vectors of multiple standard action segments; Calculating the dynamic time warping distance between the motion trajectory feature of the skeleton key point and each of the template feature vectors; Determine the matching similarity score between the motion trajectory feature of the skeleton key point and each of the template feature vectors according to the dynamic time warping distance; performing a reverse normalization process on the matching similarity score to generate the local action deviation score, wherein the local action deviation score is negatively correlated with the matching similarity score; The step of inputting the operation instrument interaction features into a pre-trained violation feature classification network to generate a set of violation action features associated with the local action deviation score includes: Performing feature dimension splicing on the local action deviation score and the operation instrument interaction feature to generate a fused violation feature vector; Calling the multi-scale attention module in the violation feature classification network to perform multi-granularity feature extraction processing on the fused violation feature vector to generate multi-level violation features; Constructing a violation feature pyramid structure based on the multi-level violation features, and performing binary classification prediction processing on each level of the violation feature pyramid structure to generate a hierarchical violation probability distribution; A weighted voting process is performed according to the hierarchical violation probability distribution of each level to generate the violation action feature set.

2. The method for identifying violations of tobacco station personnel based on deep learning according to claim 1 is characterized in that: The spatiotemporal feature extraction processing is performed on the tobacco leaf manipulation behavior video data set to obtain the skeleton key point motion trajectory features and manipulation instrument interaction features of each video clip sequence, including: Performing human posture estimation processing on each frame image in the continuous video clip sequence to generate a skeleton key point sequence containing three-dimensional coordinates; Performing time dimension interpolation processing on the skeleton key point sequence to obtain an interpolated skeleton key point sequence with equal time steps; Calling a pre-trained spatiotemporal feature encoder to perform spatiotemporal convolution encoding on the interpolated skeleton key point sequence to generate the skeleton key point motion trajectory features, and simultaneously performing instrument detection processing on the continuous video clip sequence to identify the category and spatial position information of the handheld instrument in each frame of the image; An instrument motion trajectory vector is constructed based on the category and spatial position information of the handheld instrument, and the instrument motion trajectory vector is input into an instrument interaction analysis network to generate the instrument operation interaction feature.

3. The method for identifying violations of tobacco station personnel based on deep learning according to claim 2 is characterized in that: The calling of a pre-trained spatiotemporal feature encoder to perform spatiotemporal convolution encoding processing on the interpolated skeleton key point sequence to generate the skeleton key point motion trajectory feature includes: Dividing the interpolated skeleton key point sequence into a plurality of time window segments; Perform two-stream feature extraction operations within each time window segment: Among them, the first-stream processing path performs one-dimensional convolution encoding on the relative displacement between adjacent skeleton key points to generate local motion pattern features; The second-stream processing path performs graph convolution encoding on the spatial distribution of all skeletal key points within the time window to generate global posture evolution features; Perform channel dimension splicing on the local motion pattern feature and the global posture evolution feature to generate a fused motion feature of the time window segment; The fused motion features of each time window segment are subjected to time dimension maximum pooling processing to generate the motion trajectory features of the skeleton key points.

4. The method for identifying violations of tobacco station personnel based on deep learning according to claim 1 is characterized in that: The training method of the violation feature classification network includes: Collecting a historical tobacco leaf operation violation video dataset, and labeling the violation type of the historical tobacco leaf operation violation video dataset to generate a training sample set with multi-level violation labels; Performing spatiotemporal feature extraction processing on each sample in the training sample set to generate sample skeleton key point motion trajectory features and sample operation instrument interaction features; Generating a sample local motion deviation score based on the matching result of the sample skeleton key point motion trajectory feature and the compliance motion template feature; splicing the sample local action deviation score with the sample operation instrument interaction feature to generate a sample fusion violation feature vector; Constructing an initial violation feature classification network including a multi-scale attention module, inputting the sample fusion violation feature vector into the initial violation feature classification network to generate a predicted violation probability distribution; The initial violation feature classification network is trained according to the cross entropy loss between the predicted violation probability distribution and the multi-level violation labels until the loss converges to obtain the violation feature classification network.

5. The method for identifying violations of tobacco station personnel based on deep learning according to claim 4 is characterized in that: The performing spatiotemporal feature extraction processing on each sample in the training sample set includes: Performing key frame sampling processing on the historical tobacco leaf operation violation video dataset to generate a sampling frame sequence with equal time intervals; Performing data enhancement processing on the sampling frame sequence to obtain the enhanced sampling frame sequence, wherein the data enhancement processing includes at least one of the following operations: temporal random cropping, spatial random flipping, and skeleton key point jitter injection; Calling a pre-trained lightweight posture estimation model to perform real-time skeleton key point detection processing on the enhanced sampling frame sequence to generate an enhanced skeleton key point sequence; The enhanced skeletal key point sequence is input into the training copy of the spatiotemporal feature encoder to generate enhanced skeletal key point motion trajectory features, and the enhanced sampling frame sequence is simultaneously subjected to instrument detection processing to generate enhanced instrument motion trajectory vectors, and the enhanced instrument motion trajectory vectors are input into the training copy of the instrument interaction analysis network to generate enhanced operation instrument interaction features.

6. The method for identifying violations of tobacco station personnel based on deep learning according to claim 5 is characterized in that: The method of calling a pre-trained lightweight posture estimation model and performing real-time skeleton key point detection processing on the enhanced sampling frame sequence to generate an enhanced skeleton key point sequence includes: Inputting the enhanced single-frame image into the feature extraction backbone network of the lightweight pose estimation model to generate a multi-scale image feature map; Performing heat map regression processing on the multi-scale image feature map to generate a heat map distribution of each skeletal key point; Performing non-maximum suppression processing on the heat map distribution to determine the two-dimensional coordinate position of each skeletal key point; Combined with the optical flow information between adjacent frames of the sampling frame sequence, the two-dimensional coordinate position is subjected to time domain smoothing processing to generate a three-dimensional skeleton key point coordinate sequence.

7. The method for identifying violations of tobacco station personnel based on deep learning according to claim 1 is characterized in that: The step of performing illegal behavior identification processing on the video clip sequence according to the illegal action feature set to generate an illegal behavior identification result including an illegal behavior type label and time location information includes: Performing a time dimension sliding window scanning process on the illegal action feature set to generate multiple candidate illegal time periods; For each candidate violation time period, extract the subset of violation action features within the corresponding time range; Inputting the subset of violation action features into a violation classification decision tree model to generate a preliminary violation type prediction result; Based on the preliminary violation type prediction result, the multi-evidence verification mechanism is triggered: When a device use violation is predicted, the matching degree between the interaction characteristics of the device during the corresponding time period and the preset list of prohibited devices is verified; When the prediction is that the movement amplitude is in violation, check whether the local movement deviation score in the corresponding time period exceeds the dynamically adjusted deviation threshold; When an operation process violation is predicted, the matching status of the compliance action templates of multiple consecutive video clip sequences within the corresponding time period is checked to see if they violate the process logic constraints; The preliminary violation type prediction result is modified according to the verification result to generate the final violation type label and time location information; The method for determining the dynamically adjusted deviation threshold includes: Count the distribution of local action deviation scores of all operators in the current smoke station within the preset period; Calculating the mean and standard deviation of the local action deviation score distribution; constructing a dynamic deviation threshold function based on the mean and the standard deviation, so that the dynamic deviation threshold decreases as the standard deviation increases; The changing trend of the local action deviation score distribution is monitored in real time, and the dynamic deviation threshold is recalculated when a shift in the distribution morphology is detected.

8. A deep learning-based system for identifying violations of tobacco station personnel, characterized by: The deep learning-based tobacco station personnel violation identification system includes a processor and a memory, the memory is connected to the processor, the memory is used to store programs, instructions or codes, and the processor is used to execute the programs, instructions or codes in the memory to implement the deep learning-based tobacco station personnel violation identification method described in any one of claims 1 to 7 above.

Images