Sewing action recognition method, system and terminal based on hand skeleton key points

Abstract

This invention discloses a sewing action recognition method, system, and terminal based on key points of the hand skeleton, belonging to the field of digital technology in the sewing industry. It includes: acquiring images of the sewing machine's working area; identifying twenty-one key points of the hand skeleton based on the MediaPipeHands model; extracting position, posture, and temporal features; identifying four types of actions—material handling, sewing, material preparation, and material feeding—according to feature parameters and preset rules, combined with a priority mechanism; determining the start and end frames of the actions and calculating the actual time consumption, time deviation ratio, and efficiency improvement potential. This invention also provides a corresponding system, terminal, and storage medium. This invention relies solely on visual data, requiring no hardware modification to the sewing machine, and solves the problems of high deployment costs, low recognition accuracy, and inability to quantify action efficiency in existing technologies. It has advantages such as strong compatibility, accurate recognition, and precise efficiency quantification, and is suitable for production management and employee skills training in sewing enterprises.

Description

Technical Field

[0001] In the process of digital transformation of the sewing industry, accurate identification and efficiency quantification of sewing actions are core requirements for improving production management and optimizing employee operational skills. However, existing technical solutions have significant shortcomings. This invention aims to address the following key technical issues: Background Technology

[0002] The sewing industry is a core link in the textile, apparel, and home textile sectors, directly determining a company's production capacity and market competitiveness. Under the trend of digital transformation in the industry, production efficiency monitoring, employee skills assessment, and process optimization have become core needs for sewing companies. These companies urgently need to accurately identify key actions in the sewing process, quantify the time consumption and optimization potential of individual actions, and thus specifically improve employee operational skills and eliminate inefficient processes. Currently, sewing factories generally use traditional piece-rate accounting as the core method for sewing efficiency monitoring, with some companies supplementing it with manual recording and video playback analysis. The core logic of piece-rate accounting is to count the number of sewing products or stitches completed by the sewing worker per unit of time to calculate wages and assess overall efficiency. Manual recording relies on managers conducting regular inspections and manually recording production progress. Video playback analysis requires extracting video clips afterward for manual observation and judgment of action standardization.

[0003] However, with the increasing demand for industrial upgrading, the limitations of traditional methods have become increasingly apparent: on the one hand, piece-rate accounting can only reflect the final output results and cannot penetrate to the level of individual actions such as "taking materials, sewing materials, sorting materials, and putting materials in", making it difficult to locate the root cause of inefficient actions; on the other hand, manual recording and video analysis have problems such as low efficiency, strong subjectivity, and obvious lag, making it impossible to provide real-time feedback on operational anomalies and difficult to objectively quantify the standardization and optimization potential of employees' actions.

[0004] To address the aforementioned issues, the industry has begun exploring automated and precise sewing motion recognition technologies, shifting from simple manual statistics to technology-driven monitoring methods. Among these, Jack Technology Co., Ltd.'s Chinese invention patent application, "Sewing Motion Recognition Method, System, Medium, and Terminal," with publication number CN120412106A, discloses a representative solution using visual technology to assist in sewing motion recognition. Through multimodal fusion of visual and electronic control data, it achieves automated sewing motion recognition for the first time, providing an important reference for the industry's technological development. However, this solution still has shortcomings: First, it relies on the sewing machine's built-in sensors and electronic control data, resulting in high deployment costs and poor compatibility. It requires sewing machines with sensor adaptability and necessitates hardware modifications to older equipment, further increasing deployment costs. Moreover, the electronic control data format is limited by the equipment model, making it incompatible with different brands and specifications of sewing machines, thus lacking versatility. Second, its hand feature extraction is coarse, leading to limited action recognition accuracy. It only uses the center point of the hand's bounding box as the hand position, failing to capture fine skeletal key points such as fingertips and knuckles, and cannot distinguish between different hand postures such as grasping, extending, and stabilizing, posing a risk of action misjudgment. Third, it fails to address frame loss issues, resulting in insufficient data reliability. Frame loss may occur during image acquisition or transmission, but the solution lacks a data completion mechanism, leading to distorted judgment of action start and end times, affecting the accuracy of action recognition and time consumption statistics.

[0005] The high-fidelity hand tracking solution, MediaPipeHands, employs a multi-stage cascaded machine learning pipeline architecture. This solution primarily includes the following core technical features: First, a two-stage model collaboration mechanism. This solution typically consists of two cascaded neural network models: First, a palm detector (PalmDetector) is used, usually based on a single-shot multiple-box (SSD) architecture, to perform full-image detection on the relatively rigid area of ​​the palm, handling complex hand poses such as clenched fists and occlusions, outputting the region of interest (ROI) of the hand; subsequently, a hand landmark model (HandLandmarkModel) is used to refine the processing within the aforementioned ROI, predicting the coordinates of twenty-one 3D skeletal keypoints of the hand through regression analysis. Second, a dynamic switching strategy between detection and tracking. To reduce computational load and achieve real-time operation on mobile devices, this solution adopts a "detection-tracking" logic. Full-image palm detection is triggered only in the first frame of the video sequence or when tracking is lost and cannot be recovered. In consecutive video frames, the system uses the predicted keypoint positions from the previous frame to infer the hand cropping region for the current frame, directly inputting it into the landmark model for regression, thus avoiding time-consuming full-image detection frame by frame. Third, it possesses 2.5D spatial inference capabilities. In addition to conventional two-dimensional image coordinates, this type of model can learn the geometric constraints of the hand to infer the relative depth information (Z-axis coordinates) of keypoints relative to the wrist, thereby constructing the 3D spatial pose of the hand to adapt to complex interaction requirements. Summary of the Invention

[0006] One of the objectives of this invention is to avoid the reliance on built-in sensors in sewing machines in the prior art, and to propose a solution based on single visual data and key point detection technology of hand bones, thereby solving at least one of the problems in the prior art, such as high deployment cost, low recognition accuracy, inability to quantify efficiency, and weak anti-interference ability.

[0007] Therefore, some embodiments of this application provide a sewing action recognition method based on key points of the hand skeleton. This method relies solely on visual data and possesses refined action recognition and efficiency analysis capabilities. According to some embodiments of this application, the method includes the following steps: acquiring an image of the sewing machine's working area during a sewing operation as the sole data source; the image covers the sewing machine table area, the fabric placement area, and the sewing machine operator's hand operation area; processing the image based on a key point detection model to identify the three-dimensional coordinates of twenty-one key points of the hand skeleton, including the fingertips, proximal phalanges, middle phalanges, distal phalanges, and wrist key points of the thumb, index finger, middle finger, ring finger, and little finger; extracting feature parameters of the twenty-one key points, including: the height coordinates of each key point relative to the sewing machine table, the bending angle of the finger joints, the movement trajectory of the key points, the displacement of key points in adjacent frames, and the temporal change rate; and obtaining recognition results for four types of sewing actions—picking up material, sewing material, arranging material, and placing material—based on the feature parameters and preset action judgment rules.

[0008] In some embodiments, the keypoint detection model employs the MediaPipeHands model.

[0009] In some embodiments, the preset action determination rules include: the rule for determining the material-grabbing action is: within a first preset duration, the height of the wrist key point is continuously higher than a preset material-grabbing height, where the material-grabbing height is a threshold of 70mm~100mm above the sewing machine table height, and the joint bending angle of at least three fingers is in the grasping range of 110°~170°, while the hand key point movement trajectory is towards the fabric placement area; wherein, the first preset duration covers more than 95% of the material-grabbing actions; the rule for determining the material-releasing action is: within a second preset duration, the height of the wrist key point is continuously lower than a preset material-releasing height, where the material-releasing height is 5mm~25mm above the sewing machine table height, and the joint bending angle of all fingers is in the extended range of 0°~40°, while the hand key point movement trajectory is towards the fabric placement area; wherein, the first preset duration covers more than 95% of the material-grabbing actions; The movement trajectory of the point stops at the fabric placement area; the rule for determining the sewing action is: within the third preset time period, the height of the wrist key point is within the preset operation height range, the height range is based on the height of the table and adjusted by a third threshold, the third threshold is from a decrease of -10mm to an increase of 20mm, the fluctuation range of the finger joint bending angle is less than or equal to 6°~15°, and the displacement of the hand key point in adjacent frames is less than or equal to 2mm / frame~5mm / frame; the rule for determining the whole-piece action is: after the sewing action is recognized, the height of the wrist key point is within the preset operation height range, the movement trajectory of the hand key point is a reciprocating straight line or arc, and the bending angle of the joints of at least two fingers changes periodically within the adjustment range of 20°-130°, while the movement range of the key point covers the fabric outline area.

[0010] In some embodiments, the preset action determination rule adopts the following parameters: the first preset duration is 2.5 seconds, the material picking height threshold increment is 85mm, and the grasping range is 120°~160°; the second preset duration is 1.8 seconds, the material releasing height threshold increment is 15mm, and the stretching range is 0°~30°; the third preset duration is 2.2 seconds, the operation height range is the sewing machine table height -5mm to +15mm, the joint angle fluctuation range is ≤10°, the adjacent frame displacement is ≤3mm / frame; and the adjustment range is 30°~120°.

[0011] In some embodiments, an action priority determination mechanism is also included, wherein when multiple action determination conditions are met simultaneously, the recognition results are output in the order of priority: sewing action > assembling action > picking action > placing action.

[0012] In some embodiments, the coordinate data of the skeletal key points are further filtered, and a Kalman filter algorithm is used to eliminate key point jitter caused by image noise and improve the stability of feature parameters.

[0013] In some embodiments, the finger joint bending angle is calculated by the vector angle between adjacent skeletal key points, and the temporal change rate is calculated by fitting the change in key point coordinates of 10 consecutive frames of images.

[0014] In some embodiments, a frame loss detection and completion step is also included. When the displacement of the same hand key point in two consecutive frames changes abruptly to a value greater than or equal to the frame loss displacement threshold Dloss and there is no corresponding pose change, it is determined to be a frame loss. The key point coordinates of the lost frame are completed by linear interpolation, and the displacement after completion is verified to be less than or equal to the verification threshold Dcheck to ensure that the calculation error of the action time is ≤5%.

[0015] Some embodiments of this application also provide a sewing action recognition system based on key points of the hand skeleton. The system includes: an image acquisition module for acquiring images of the sewing machine's working area, the images covering the sewing machine table area, the fabric placement area, and the operator's hand operation area; using an industrial camera with a frame rate ≥30fps and an image resolution ≥1920×1080; a key point recognition module for performing any of the above recognition steps to identify twenty-one key points of the hand skeleton and outputting the three-dimensional coordinates of the key points; a feature extraction module for extracting the feature parameters, including height coordinates, joint bending angles, and motion trajectories; and an action recognition module for outputting sewing action recognition results according to the preset action judgment rules and priority mechanism.

[0016] Some embodiments of this application also provide a terminal, the terminal including a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory, so that the terminal performs the sewing action recognition method described in any of the preceding claims.

[0017] Compared with existing sewing action recognition methods or systems, the core advantages of this invention are as follows: First, it eliminates hardware dependence, significantly improving compatibility and deployment flexibility. This invention does not rely on built-in sensors and electronic control data in sewing machines; it achieves motion recognition solely through visual data. No modifications to the sewing machine are required, making it compatible with different brands, models, and ages of sewing machines, greatly enhancing its versatility.

[0018] Secondly, the motion recognition accuracy is higher and the discrimination is stronger. This invention uses the detection of 21 key skeletal points of the hand to capture the refined posture features of the hand, replacing the coarse positioning method of "center point of hand bounding box" in the existing technology. It can accurately distinguish the differences in postures such as grasping, stretching, and stabilization, effectively reducing the risk of motion misjudgment.

[0019] This invention adds efficiency quantification capabilities to support refined management. Based on motion recognition, it builds an efficiency analysis system that automatically calculates the actual time spent on each action, the proportion of time deviation, and the potential for efficiency improvement. This overcomes the shortcomings of existing technologies that can only identify action types but cannot quantify optimization space, providing precise data support for employee skills training and production process optimization.

[0020] It boasts stronger anti-interference capabilities and higher data reliability. This invention adds a frame loss completion mechanism, which uses a linear interpolation algorithm to complete lost frame data during image acquisition / transmission, ensuring that the calculation error of action time is ≤5%. Compared with existing technologies that do not handle frame loss, the accuracy and stability of data are significantly improved. Attached Figure Description

[0021] Figure 1 This is a schematic diagram of the overall structure of a sewing motion recognition system based on key points of the hand skeleton according to an embodiment of this application.

[0022] Figure 2 This is a flowchart of a sewing action recognition method based on key points of the hand skeleton according to an embodiment of this application.

[0023] Figure 3 This is a flowchart of hand skeletal key point recognition and feature extraction according to an embodiment of this application.

[0024] Figure 4 This is a flowchart of the action recognition and determination process according to an embodiment of this application.

[0025] Figure 5This is a flowchart of action boundary determination and efficiency analysis according to an embodiment of this application.

[0026] Figure 6 This is a flowchart of parameter calibration and threshold setting according to an embodiment of the present application.

[0027] Figure 7 This is a flowchart of frame loss detection and completion according to an embodiment of this application. Detailed Implementation

[0028] This invention is based on single visual data and key point detection technology of hand bones to achieve accurate recognition of four types of actions in the sewing process: "picking up material, sewing material, arranging material, and putting material down". It also quantifies the time consumption and optimization space of the actions through an efficiency analysis module, without relying on the sewing machine's built-in sensors and electronic control data.

[0029] like Figure 1 As shown, the overall architecture of the sewing action recognition system based on key points of hand bones according to the embodiments of this application includes five core modules: image acquisition module 101, hand bone key point recognition module 102, feature extraction module 103, action recognition module 104, and efficiency analysis module 105. Each module works together to ensure the reliability of recognition accuracy and efficiency quantification. The following is a detailed description of each module and the overall workflow.

[0030] The technical architecture of this invention adopts a layered design, forming a closed loop from data acquisition to result output, specifically including: The data input layer includes an image acquisition module 101, which acquires images of the sewing machine's working area based on an industrial camera, serving as the sole data source. The core processing layer includes a hand skeleton key point recognition module 102, a feature extraction module 103, and an action recognition module 104, which realizes the conversion from image data to action type. The functional extension layer includes an efficiency analysis module 105, which performs time calculation, frame loss completion, and optimized spatial quantization based on the action recognition results; The data output layer includes a result output module 107 for outputting action recognition results (material picking / sewing / assembling / feeding) and efficiency quantification indicators (actual time consumption, time deviation ratio, efficiency improvement potential), supporting local display or integration with the production management system.

[0031] Preferably, the core function of the parameter calibration module 106 is to initialize and dynamically calibrate key threshold parameters of the system, including height threshold, joint angle range, displacement threshold, and frame loss judgment threshold, to ensure recognition accuracy under different sewing machine models and different operating habits. This module adopts a two-stage approach: first, it establishes the mapping relationship between pixel coordinates and physical coordinates through a calibration board, i.e., pixel physical calibration; second, it collects standard operation videos, i.e., collects standard operation samples, statistically analyzes the key features of each action stage, determines the threshold using the mean, standard deviation, or quantiles, and supports adaptive fine-tuning based on actual production data.

[0032] As mentioned above, the core advantages of this architecture lie in its single data source, refined feature extraction, and full-process automation, which eliminates the dependence on sewing machine hardware. At the same time, it captures hand posture details through skeletal key points, solving the problem of insufficient recognition accuracy in traditional vision solutions.

[0033] like Figure 2 As shown, the steps of the sewing action recognition method based on key points of the hand skeleton according to an embodiment of this application mainly include: Overall, the sewing action recognition method of the present invention specifically includes the following steps: S101: The sewing image acquisition uses an industrial camera to acquire real-time images of the sewing machine's working area at a frame rate of no less than 30fps. The images cover the sewing machine table, the fabric placement area, and the area where the sewing machine operator's hands are operating, and each frame is assigned a consecutive frame number.

[0034] S102: Identify key points of the hand. The MediaPipeHands model is called to process the acquired image, identify the three-dimensional coordinates (X, Y, Z) of twenty-one key points of the hand bones, and eliminate lens distortion and shaking through coordinate calibration and Kalman filtering.

[0035] S103: Extract position and pose features. Based on the coordinates of the identified key points, extract wrist height (Wz), planar position (Wx, Wy), bending angle of each finger joint, and hand shape features (finger-tip distance, degree of opening, etc.).

[0036] S104: The sliding window method (window size is 10 frames) is used to extract the temporal change features. The hand movement trajectory, the displacement of adjacent frames (ΔX, ΔY, ΔZ) and the temporal change rate are extracted to form a temporal feature vector.

[0037] S105: The action determination module inputs position features, posture features and timing features into the action recognition module. Based on the preset four types of action determination rules (picking up material, sewing material, arranging material, and placing material) and combined with the action priority mechanism (sewing material > arranging material > picking up material > placing material), it outputs the action category of the current frame.

[0038] S106: Determine the start and end frames of the action. Based on the action recognition results, determine the start frame number S and end frame number E of each action as the basis for subsequent time calculation.

[0039] S107: Calculate the efficiency index based on the start and end frames of the action and the preset reference time, calculate the actual time T, the time deviation ratio P, and the efficiency improvement potential Q; at the same time, detect whether there is frame loss, and if so, complete it by linear interpolation.

[0040] S108: Output recognition results. Display the action recognition results (action type, start and end frames, actual time consumption, time deviation ratio, and efficiency improvement potential) through the result output module or upload them to the production management system.

[0041] The detailed technical implementation of each core module will be introduced below.

[0042] First, the technical implementation of the image acquisition module 101.

[0043] The core function of the image acquisition module 101 is to acquire clear and stable images of the sewing machine's working area, providing high-quality data support for subsequent key point recognition. Specific implementation schemes can be adopted as follows: Hardware Selection: An industrial-grade high-definition camera is used, featuring high frame rate, high definition, low latency, and strong anti-interference capabilities to ensure clear capture of hand details and fabric contours. Preferably, the camera frame rate is ≥30fps and the image resolution is ≥1920×1080 to meet the requirements for real-time high-precision hand key point detection.

[0044] Installation method: The camera is installed directly above the sewing machine, with the lens pointing vertically downwards. The installation height is 60cm above the sewing machine table, ensuring that the shooting range completely covers the sewing machine table area, the fabric placement area, and the area where the sewing worker's hands operate (horizontal coverage ≥ 80cm, vertical coverage ≥ 60cm); the camera is fixed to an adjustable bracket, which supports fine-tuning of the height and angle according to the sewing machine model to ensure consistent shooting effects under different equipment scenarios.

[0045] Second, the technical implementation of the hand skeletal key point recognition module 102. The core function of the hand skeletal key point recognition module 102 is to accurately identify the hand from acquired images, such as input video frames, and then identify the three-dimensional coordinates of twenty-one key skeletal points of the hand from the hand region, providing basic data for feature extraction. The specific implementation scheme is as follows: Model selection: The MediaPipe Hands model was selected as the core detection model. This model supports real-time detection of 21 skeletal key points of one or both hands, and features lightweight, high precision and low latency, making it suitable for real-time processing needs in industrial scenarios.

[0046] Key point definition: Clearly define the location of 21 skeletal key points: including five fingers, namely thumb, index finger, middle finger, ring finger and little finger, each finger contains four key points: fingertip, proximal phalanx, middle phalanx and distal phalanx, plus 1 wrist key point, totaling 21 skeletal key points covering the core posture representation parts such as fingertip, phalanx and wrist.

[0047] Coordinate Output and Calibration: The model outputs the 3D coordinates (X, Y, Z) of twenty-one key points, where the X-axis is the horizontal coordinate, the Y-axis is the horizontal coordinate, and the Z-axis is the vertical coordinate. The 3D coordinates are based on the sewing machine table plane, with the Z-coordinate of the table plane set to 0 and upward as the positive direction. The output coordinate data is calibrated by establishing the mapping relationship between image pixel coordinates and physical space coordinates through a preset calibration plate (size 50cm×50cm, checkerboard spacing 5cm), eliminating coordinate errors caused by lens distortion, and ensuring the accuracy of the physical meaning of the key point coordinates.

[0048] Third, the technical implementation of the feature extraction module 103. The core function of the feature extraction module 103 is to extract parameters that can characterize hand posture and movement features from the three-dimensional coordinates of key points, providing a basis for action recognition. The specific implementation scheme is as follows: The feature classification and extraction logic extracts three core feature parameters: position features, posture features, and temporal change features. These three types of feature parameters complement each other and comprehensively characterize the static and dynamic attributes of hand movements.

[0049] Among them, the extraction of location features includes the extraction of height features and planar location features.

[0050] The height feature extraction process includes calculating the Z-axis coordinate of each key point, with a focus on the Z-axis coordinate of the wrist key point (denoted as Wz), which serves as the core indicator for judging hand height; at the same time, the average Z-axis coordinate of all key points of the hand (denoted as Hz) is calculated to help judge the overall height status of the wrist.

[0051] Extracting planar position features involves calculating the X and Y coordinates (Wx, Wy) of key points on the wrist, and combining them with the X and Y coordinate ranges of the fabric outline to determine the positional relationship of the hand relative to the fabric, such as whether it is located within the fabric area or whether it is moving towards the fabric area.

[0052] The extraction of posture features includes extracting joint flexion angles and hand shape features: The extraction of joint bending angles, i.e., joint angles, includes calculating the finger joint bending angle by using the vector angle between adjacent skeletal key points. Taking the index finger as an example, the key points of the distal phalanx (P1), middle phalanx (P2), and proximal phalanx (P3) are selected to construct vectors P2P1 (P1-P2) and P2P3 (P3-P2). The angle between the two vectors is calculated using the vector dot product formula, which is the bending angle of the middle phalanx joint of the index finger. Similarly, the bending angles of each joint of the thumb, middle finger, ring finger, and little finger are calculated (3 joint angles for each finger, for a total of 15 joint angles). The range of joint bending angles is 0° (fully extended) to 180° (fully bent), used to distinguish different hand postures such as grasping, extending, and adjusting.

[0053] The extraction of hand morphological features includes: calculating the distance between key points of the fingers and key points of the wrist (such as the distance from the fingertip to the wrist) to determine the degree of hand extension; calculating the distance between key points of adjacent fingertips to determine the degree of finger spread, and assisting in distinguishing between picking up materials (finger grasping, small distance between adjacent fingertips) and putting down materials (finger extension, large distance between adjacent fingertips) actions.

[0054] Among them, extracting temporal change features includes extracting motion trajectory, displacement, and temporal change rate: The extraction of motion trajectory includes: using the sliding window method (window size is 10 frames), recording the changes in the X, Y, and Z axis coordinates of key points of the wrist in 10 consecutive frames of images, fitting the hand motion trajectory, and determining the direction of motion (such as towards the fabric placement area, away from the fabric area, reciprocating motion, etc.).

[0055] The extraction of displacement includes: calculating the coordinate difference (ΔX, ΔY, ΔZ) of the same key point in two adjacent frames to obtain the instantaneous displacement of the key point; calculating the average displacement within the sliding window to determine the speed of hand movements (e.g., small displacement when sewing and large displacement when picking up materials).

[0056] The extraction of temporal change rate includes: calculating the change in joint bending angle in 10 consecutive frames of images and fitting the angle change curve.

[0057] like Figure 3 As shown, the hand skeleton key point recognition and feature extraction process of one embodiment of the present invention is implemented using the MediaPipeHands model, and specifically includes the following steps: S201: Input video frames. The industrial camera captures continuous video frame images of the sewing machine's working area in real time at a frame rate of no less than 30fps, and inputs these video frames into the hand bone key point recognition module for processing.

[0058] S202: Detect the hand region. Call the PalmDetector in the MediaPipeHands model. Based on the Single Multiple Frame Detection (SSD) architecture, perform full-image detection on the input video frame, locate the area where the hand is located, and output the region of interest (ROI) to deal with complex hand poses such as clenched fists and occlusion.

[0059] S203: Extract 21 key points. Within the detected hand ROI area, the coordinates of 21 three-dimensional skeletal key points of the hand are predicted by regression analysis using the HandLandmarkModel, including the fingertips, proximal phalanges, middle phalanges, distal phalanges, and wrist key points of the thumb, index finger, middle finger, ring finger, and little finger.

[0060] S204: Coordinate Calibration. The 3D coordinates (X, Y, Z) output by the model are calibrated. A mapping relationship between image pixel coordinates and physical space coordinates is established by using a preset calibration plate (50cm×50cm, checkerboard spacing 5cm), eliminating coordinate errors caused by lens distortion and ensuring the accuracy of the physical meaning of key point coordinates.

[0061] S205: Calculate wrist height. Extract the Z-axis coordinates (denoted as Wz) of the wrist key points, and simultaneously calculate the average Z-axis coordinate (denoted as Hz) of all key points of the hand to determine the height of the hand relative to the sewing machine table.

[0062] S206: Calculate joint angles. Calculate the bending angle of each finger joint using the vector angle between adjacent skeletal key points. Taking the index finger as an example, select the key points of the distal phalanx (P1), middle phalanx (P2), and proximal phalanx (P3) to construct vectors P2P1 and P2P3. Calculate the angle between the two vectors using the dot product formula, which is the bending angle of the middle phalanx joint of the index finger. Similarly, calculate the bending angles of each joint of the other fingers (3 joints per finger, 15 joint angles in total). The range of joint bending angles is 0° (fully extended) to 180° (fully bent).

[0063] S207: Calculate the displacement trajectory. Using the sliding window method (window size of 10 frames), record the changes in the X, Y, and Z axis coordinates of the wrist key points in 10 consecutive frames, calculate the displacement (ΔX, ΔY, ΔZ) of key points in adjacent frames, fit the hand movement trajectory, and determine the direction and amplitude of the movement.

[0064] S208: Output feature vector. Integrate position features (wrist height Wz, planar position Wx / Wy), posture features (joint bending angle, hand shape features), and temporal change features (motion trajectory, displacement, temporal change rate) into a feature vector, and output it to the action recognition module 104 for subsequent sewing action determination.

[0065] The technical implementation of the action recognition module 104 is described below with reference to the accompanying drawings.

[0066] The core function of the action recognition module 104 is to accurately identify four types of actions, namely "picking up materials, sewing materials, arranging materials, and putting materials", based on the feature parameters extracted by the feature extraction module 103 and combined with the preset action judgment rules and priority mechanism, and output the action category, action start and end frames and key feature quantities for subsequent efficiency analysis.

[0067] This module employs a priority decision-making mechanism to accurately identify and resolve conflicts among four types of actions, such as... Figure 4 As shown, the specific process is as follows: First, after receiving the feature parameters, the S301 action recognition module performs condition judgments in a preset priority order. The priority order is: sewing action > material handling action > material picking action > material placement action. When multiple action judgment conditions are met simultaneously, the system outputs the action category with the highest priority.

[0068] The specific judgment process is as follows: S302, the module first determines whether the sewing action conditions are met. If the sewing height condition is met, such as the wrist Z-axis coordinate being between -5mm and +15mm from the table height, the posture stability condition, such as the joint bending angle fluctuation range being ≤10°, and the displacement condition, such as the wrist displacement in adjacent frames being ≤3mm / frame, then the sewing action is directly output, S303. If not, it continues to determine whether the material handling action conditions are met, S304. The material handling action must simultaneously meet the prerequisite conditions, such as having entered the sewing station and not detecting a material picking action, the height condition, the trajectory reciprocating condition, and the posture periodic change condition, such as the angle of at least two finger joints periodically changing within the range of 30° to 120°. If these conditions are met, the material handling action is output, S305.

[0069] If the conditions are still not met, continue to check if the material picking action conditions are met (S306). The material picking action must simultaneously meet the following conditions: height (e.g., the wrist Z-axis coordinate is continuously higher than the table height + 85mm), posture (e.g., at least 3 finger joints are bent at an angle between 120° and 160°), and trajectory (e.g., the movement direction is towards the fabric placement area). If these conditions are met, output the material picking action (S307). If the conditions are still not met, check if the material placing action conditions are met (S308). The material placing action must simultaneously meet the following conditions: height (e.g., the wrist Z-axis coordinate is continuously lower than the table height + 15mm), posture (e.g., all finger joints are bent at an angle between 0° and 30°), and position (e.g., the wrist coordinate is within the fabric outline). If these conditions are met, output the material placing action (S309). If all the above conditions are not met, output an invalid or pending state (S310), and continue processing the next frame.

[0070] The inherent relationship between the input / output data of the action recognition module 104: The input data of the motion recognition module 104 includes at least: the Nth frame image IN, the set of coordinates of twenty-one hand skeleton key points KN output by the key point detection module, the position features, posture features, and temporal feature vector FN output by the feature extraction module, and optional feature parameters such as cloth contour information CN. The position features preferably include the Z-axis coordinate Wz, X / Y plane position, and displacement in adjacent frames of the wrist key points; the posture features preferably include the bending angle of each finger joint; and the temporal features preferably include the trajectory direction, angle fluctuation range, and duration within the sliding window.

[0071] The output data of the action recognition module 104 includes at least: the action label AN of the current frame, the action start frame S, the action end frame E, the action confidence result RN, and intermediate quantities required for the efficiency analysis module to calculate the action duration, time deviation ratio, and efficiency improvement potential. In other words, the input image does not directly output the action category, but first forms a judgment basis through key points and feature parameters, and then maps the action rules to the output action result. Preferably, the mapping relationship can be expressed as: AN=g(FN,FN-1,…,FN-m,R), where R is the preset action rule set and m is the temporal window length.

[0072] The action recognition module 104 is designed with the following action determination rules: based on the feature differences of the four types of actions, refined determination rules are formulated. All rules are based on a frame rate of 15fps and calibrated physical coordinates to ensure the operability and accuracy of the determination logic.

[0073] Specifically: The rules for determining the material handling action in the action recognition module 104 include: Material handling height conditions: The Z-axis coordinate Wz of the wrist key point is continuously greater than or equal to the sewing machine table height H0 plus the material handling height threshold DeltaH1; preferably, the default value of DeltaH1 is 85mm, and the reasonable calibration range is 70mm-100mm; the default value of the gripping interval Theta1 is 120°-160°, and the reasonable adjustment range is 110°-170°.

[0074] Material handling posture conditions: the joints of at least 3 fingers are bent at an angle within the grasping range Theta1; Material handling trajectory conditions: The fitted direction of the hand movement trajectory is towards the fabric placement area; Material handling trigger logic: When the above conditions are met simultaneously and the duration is greater than or equal to the first preset duration, it is determined to be a material handling action.

[0075] The rules for determining the feeding action in the action recognition module 104 include: Fabric feed height conditions: The Z-axis coordinate Wz of the wrist key point is continuously less than or equal to the sewing machine table height H0 plus the fabric feed height threshold DeltaH2, and the Z-axis coordinate of the fabric outline centroid is less than or equal to H0 plus DeltaH2; preferably, the default value of DeltaH2 is 15mm, and the reasonable range is 5mm-25mm; the default value of the stretching interval Theta2 is 0°-30°, and the reasonable range is 0°-40°.

[0076] Material feeding posture conditions: The bending angle of all finger joints is in the extended range Theta2; Material placement conditions: The X and Y coordinates of the wrist key point must be within the X and Y coordinate range of the fabric outline; Material release trigger logic: When the above conditions are met simultaneously and the duration is greater than or equal to the second preset duration, it is determined to be a material release action.

[0077] The motion recognition module 104 includes the following rules for determining the movement of the sewing fabric: Sewing height conditions: The Z-axis coordinate Wz of the wrist key point is within the preset operating height range H0+DeltaHs; preferably, the default value of the operating height range is from table height -5mm to table height +15mm, and the reasonable range is from table height -10mm to table height +20mm; the default value of AngleStable is 10°, and the reasonable range is 6°-15°; the default value of Dmove is 3mm / frame, and the reasonable range is 2mm / frame-5mm / frame.

[0078] Sewing posture conditions: The range of fluctuation in the joint flexion angle of all fingers is less than or equal to AngleStable; Fabric displacement condition: The displacement of the wrist key point in adjacent frames is less than or equal to Dmove; Sewing trigger logic: When the above conditions are met simultaneously and the highest priority is achieved, the sewing action is directly determined.

[0079] The motion recognition module 104 includes the following rules for determining the motion of the entire material: Prerequisites for material preparation: The material has entered the sewing station and no material handling action has been detected. Material handling height condition: The Z-axis coordinate Wz of the wrist key point is within the preset operating height range; The conditions for the whole material trajectory are: the hand movement trajectory is a reciprocating straight line or arc, and the movement range covers the outline area of ​​the fabric; The posture conditions of the whole material: the bending angle of the joints of at least two fingers changes periodically within the adjustment range Theta3; preferably, the default value of the adjustment range Theta3 is 30°-120°, and the reasonable range is 20°-130°; the default value of the periodic change period is within 1 second, and the reasonable range is 0.5 seconds-1.5 seconds.

[0080] Whole material triggering logic: If the above conditions are met simultaneously, and the current sewing action stability condition is not met, it is determined to be a whole material action.

[0081] The action priority mechanism for the above four actions is as follows: To resolve recognition conflicts caused by the superposition of multiple action features, the action priority order is set as follows: sewing action > assembling action > picking up action > placing action. When the judgment conditions of multiple actions are met simultaneously, the action with the highest priority is selected as the recognition result.

[0082] Calibration basis and reasonable range of key parameters: The height threshold, angle threshold, and displacement threshold are preferably determined through a two-stage method of "equipment calibration + sample statistics". For example... Figure 6 As shown, the parameter calibration and threshold setting process may include: First, taking the upper surface of the sewing machine table as the zero height reference H0, S501, and completing the conversion from pixel coordinates to physical coordinates through a calibration board or a known size block, S502; Second, collecting several standard operation videos at the same workstation, S503; Manually labeling the four types of actions "material handling, sewing, material preparation, and material placement" S504; Statistically analyzing the wrist height S505, joint angle S506, and inter-frame displacement distribution of each action in the stable phase, and preferably using the mean, standard deviation, quantile, or a combination thereof as the threshold setting basis S508, and writing the rule parameters, S509.

[0083] For example, the material handling height threshold can be taken between the lower quartile and the median value of the wrist lifting height distribution during the material handling stage of a skilled worker; the grasping angle range can be determined based on the concentrated distribution range of the bending angles of at least three fingers during the grasping stage; and the sewing material displacement threshold can be determined based on the 95th percentile value of the displacement of adjacent frames of the wrist key points during the continuous sewing stage. For different machine models, different workstations, different fabric thicknesses, or different operating habits, adaptive adjustments can be made within the aforementioned reasonable range.

[0084] An example of a typical input-output mapping for a sewing frame: Taking the Nth frame as an example, the system first extracts 21 skeletal key points of the hand from the IN image, obtaining the wrist key point height Wz=H0+8mm in the current frame; further, it calculates the bending angles of the five finger joints, obtaining average bending angles of approximately 145°, 149°, 151°, 147°, and 148° within the current window, with angle fluctuations within consecutive windows all less than 6°; then, it calculates the wrist displacement in adjacent frames, obtaining an average displacement of approximately 1.5mm / frame; simultaneously, it detects that the fabric outline covers the needle working area. Since this frame meets the height, posture stability, and displacement conditions for "sewing action," the system outputs the action state of the Nth frame as "sewing."

[0085] Furthermore, if the start frame of the sewing action segment is S=1180 and the end frame is E=1228, then the actual action time T=(1228-1180) / 15=3.2 seconds. If the reference time of the sewing action T0=3.0 seconds, then the time deviation ratio P=(3.2-3.0) / 3.0×100%=6.67%, and the efficiency improvement potential Q=(3.0-3.2) / 3.2×100%=-6.25%. This clearly shows that after keypoint and feature parameter extraction, the input image not only drives the action category output but also further drives the output of efficiency indicators such as time consumption, deviation ratio, and efficiency potential.

[0086] Action boundary determination: Clarify the start and end criteria for various actions to provide frame boundary basis for subsequent efficiency analysis.

[0087] Material handling action: The start frame is the image frame of the first contact between the key points of the hand bones and the fabric, and the end frame is the image frame of the fabric being positioned to the starting position of the sewing machine. Material feeding action: The start frame is an image frame in which the key points of the hand bones contact the sewing completed fabric, and the end frame is an image frame in which the fabric is removed from the hand and placed in the designated area; Sewing action: The start frame is an image frame where the hand posture is stable and the fabric covers the needle position, and the end frame is an image frame where the sewing stitch stops extending; Material handling action: The start frame is the image frame where the hand key points enter the fabric outline area and the posture begins to adjust, and the end frame is the image frame where the hand posture returns to stability.

[0088] Fifth, the technical implementation of the efficiency analysis module 105 The core function of the efficiency analysis module 105 is to calculate the actual action time T and handle frame loss anomalies based on the action recognition results, and to quantify the action's time deviation and efficiency improvement potential. For example... Figure 5 As shown, its specific process may include steps S401, inputting the action recognition result, S402, determining the start frame S, S403, determining the end frame E, S404, calculating the actual time T, S405, reading the reference time T0, S406, calculating the deviation ratio P, S407, calculating the improvement potential Q, and S408, outputting the efficiency result.

[0089] Input / output relationship of efficiency analysis module 105: The input to the efficiency analysis module 105 is the action recognition result, which includes at least the action label output by the action recognition module, the start frame S, the end frame E, the frame rate F, the reference time T0, the inter-frame displacement information, and the key point change information; the output includes at least the actual action time T, the time deviation ratio P, the efficiency improvement potential Q, and the action efficiency evaluation result. Preferably, the relationship can be expressed as: {T,P,Q}=h(S,E,F,T0).

[0090] Actual action time calculation: Based on a fixed frame rate of 15fps, the actual time is calculated using the start frame number S and the end frame number E of the action. The formula is: T=(ES) / F, where F is preferably 15fps. When the system uses other frame rates, the same formula can be used for conversion.

[0091] Frame sequence number synchronization: During image acquisition, a unique consecutive frame sequence number is assigned to each image frame to ensure that the difference between the sequence number of the start frame and the end frame of the action accurately reflects the number of frames of the action; the frame sequence number is transmitted and stored synchronously with the image data to avoid sequence number confusion.

[0092] like Figure 7 As shown, the frame loss detection and completion process includes: Frame loss determination: S601 Input two consecutive frames, i.e., key points of the same hand in adjacent frames; S602 Calculate the displacement abrupt change; S603 Calculate the pose change; then determine whether the frame loss condition is met. If the displacement abrupt change of the same hand key point in two consecutive frames is greater than or equal to the frame loss displacement threshold Dloss, and there is no corresponding pose change, it is determined to be a frame loss during image acquisition or transmission. Otherwise, it is marked as a normal frame S605. Preferably, the default value of Dloss is 10mm, and the reasonable range is 8mm-15mm; the default value of the pose change threshold is 5°, and the reasonable range is 3°-8°. The Dloss can be determined by acquiring normal continuous sewing operation video, statistically analyzing the upper limit of the natural displacement of adjacent frames, and then adding a safety margin.

[0093] Completion Algorithm: S606 preferably uses linear interpolation to complete the key point coordinates of lost frames. Assuming the lost frame is located between frame n and frame (n+1), the key point coordinates of frame n are (Xn, Yn, Zn), and the coordinates of frame (n+1) are (Xn+1, Yn+1, Zn+1). The coordinates of the lost frame can be obtained by interpolation.

[0094] Post-completion verification: S607, calculate the displacement between the lost frame and adjacent frames after completion; S608, determine whether the verification condition is met, for example, the verification condition is to ensure that the completed displacement is less than or equal to the verification threshold Dcheck. Preferably, the default value of Dcheck is 5mm, and the reasonable range is 3mm-6mm; if it is not met, readjust the interpolation parameters or mark it as an abnormal segment; S601, if it is met, output the completion result; S609.

[0095] Efficiency quantification index calculation: Baseline Time Setting: Baseline times for four types of actions are preset, preferably determined based on statistical analysis of the standard action times of skilled workers. Preferably, at least 10 complete process samples can be collected, and the median or average value (excluding extreme values) of the actual time consumed for each type of action can be calculated as the initial baseline time for that type of action; adjustments can also be made based on the company's production line, fabric type, and process complexity. Preferred baseline times can be set as follows: material handling 1.5 seconds, material placement 1.0 second, sewing 3.0 seconds, and material preparation 2.0 seconds. A reasonable adjustment range is preferably 20%-30% above or below the baseline value.

[0096] Time Deviation Ratio: Reflects the degree of deviation between the actual action time and the standard time. The formula is: P=(T-T0) / T0×100%; a positive value of P indicates that the actual time is higher than the standard, and a negative value of P indicates that the actual time is lower than the standard.

[0097] Efficiency improvement potential: Quantifies the optimization space of a single action, the formula is: Q=(T0-T) / T×100%; A positive value of Q indicates that there is a possibility of efficiency improvement, and the larger the value, the greater the optimization potential; A negative value of Q indicates that the current action is already better than the standard or close to the limit, and there is no need for key optimization.

[0098] Typical efficiency calculation example: If a certain "sewing" motion segment has a start frame of S=1500, an end frame of E=1548, and a frame rate of F=15fps, then the actual time taken is T=(1548-1500) / 15=3.2 seconds. If the current workstation's calibrated sewing reference time T0=3.0 seconds, then P=(3.2-3.0) / 3.0×100%=6.67%, Q=(3.0-3.2) / 3.2×100%=-6.25%. If another segment takes only 2.7 seconds, then P=(2.7-3.0) / 3.0×100%=-10%, Q=(3.0-2.7) / 2.7×100%=11.11%. This example demonstrates a clear computational correspondence between efficiency indicators, motion recognition results, start and end frames, and reference time.

[0099] Output format: Efficiency analysis results are output in structured data format, including at least action type, start frame number, end frame number, actual time consumption, time deviation ratio, efficiency improvement potential, and anomaly markers; if necessary, corresponding key feature values ​​and threshold hit status can also be output to facilitate subsequent quality inspection traceability and process optimization.

[0100] Technical implementation of terminals and computer-readable storage media. Technical implementation of the terminal: The terminal of the present invention is used to run a computer program that implements the above-mentioned methods and system functions, so as to complete functions such as data processing after image acquisition, key point recognition of hand bones, sewing action recognition, action timing analysis and efficiency evaluation.

[0101] The terminal may include a processor, a memory, and an input / output interface connected to the processor. The processor is used to execute program instructions stored in the memory, which stores programs, model parameters, recognition rules, and related business data. The input / output interface is used to realize data interaction with an image acquisition device, a display device, a host computer system, or a production management system.

[0102] The terminal can be deployed locally, in the cloud, or in a combination of both. Regardless of the deployment method, as long as it can achieve the key point recognition, action classification, duration calculation, efficiency analysis, and result output described in this invention, it falls within the protection scope of this invention.

[0103] Technical implementation of a computer-readable storage medium: The present invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to implement the steps of the aforementioned sewing action recognition method based on key points of hand bones.

[0104] The computer-readable storage medium can be any non-transitory storage medium capable of storing program code and allowing it to be read by a terminal, server, or other computing device. The program can be stored as an executable file, script file, model file, configuration file, or a combination thereof.

[0105] When the computer program is loaded and run, it enables the relevant devices to perform functions such as image data processing, skeletal key point extraction, motion recognition, result statistics, and efficiency analysis. The method of installing, deploying, calling, updating, or transmitting the program is not intended to limit the scope of protection of this invention.

[0106] Complete workflow of the technical solution: The technical solution of the present invention realizes sewing action recognition and efficiency analysis through the following steps, which are fully automated and require no manual intervention.

[0107] System initialization: After the terminal starts, it automatically loads the computer program, the trained key point detection model, and configuration parameters, including camera parameters, reference time, motion judgment rules, etc., and completes camera self-test and parameter calibration, such as automatic exposure calibration and white balance calibration. The system then enters the ready state.

[0108] Image acquisition: The camera acquires images of the sewing machine's working area in real time at a frame rate of 15fps. The image data is transmitted to the terminal via USB 3.0 or Ethernet interface. The terminal assigns a consecutive frame number to each image and temporarily stores it in RAM.

[0109] Key point recognition: The terminal calls the MediaPipeHands model to process each frame of image, identify the three-dimensional coordinates of twenty-one key points of the hand bones, eliminate lens distortion through coordinate calibration, and then optimize the coordinate data through Kalman filtering to output stable key point coordinates.

[0110] Feature extraction: Based on the coordinates of key points, the terminal calculates position features (wrist height, planar position), posture features (joint bending angle, hand shape), and temporal change features (motion trajectory, displacement, and temporal change rate) to form a feature parameter set.

[0111] Action recognition: The terminal inputs the feature parameter set into the action recognition module, and judges whether the conditions for picking up, sewing, arranging and placing materials are met according to the preset judgment rules. The terminal outputs the final action recognition result in combination with the priority mechanism, and records the start frame number and end frame number of the action.

[0112] Efficiency Analysis: The terminal calls the efficiency analysis module to calculate the actual time consumption based on the start and end frame numbers of the action; detects whether there are any frame losses, and if so, completes the data using linear interpolation; and calculates the time deviation ratio and efficiency improvement potential by combining the preset baseline time.

[0113] Results output and storage: The terminal displays the action recognition results and efficiency quantification indicators in real time (output to the monitor via HDMI interface when deployed locally, and output to the management platform via network when deployed in the cloud); at the same time, the recognition results, efficiency data, and corresponding image frame sequence numbers are stored to solid-state drives or cloud storage services for a storage period of ≥90 days, supporting subsequent queries and data analysis.

[0114] Cyclic operation: The system continuously repeats steps 2-7 to achieve real-time recognition and efficiency analysis of sewing actions; when the sewing machine stops working (no action recognition results for 30 consecutive seconds), it automatically enters low power consumption mode to reduce resource consumption.

[0115] Basis for determining the values ​​of key technical parameters.

[0116] Key technical parameters involved in this invention, such as preset duration, height threshold, and joint angle range, are all determined based on extensive experimental verification to ensure their rationality and practicality. The specific values ​​are based on the following: Preset duration: Based on statistics of sewing actions of 50 skilled workers, the average time for picking up the material is 1.2 to 1.8 seconds, so the first preset duration is set to 2.5 seconds (covering more than 95% of the material picking actions); the average time for putting the material down is 0.8 to 1.3 seconds, so the second preset duration is set to 1.8 seconds; the average duration of the stable posture of sewing is 1.5 to 2.0 seconds, so the third preset duration is set to 2.2 seconds.

[0117] Height thresholds: The standard height of the sewing machine table is 75cm. Considering hand operating habits, when picking up materials, the hand needs to be 80-90mm higher than the table. Therefore, the material picking height threshold is set to 85mm higher than the table height. When feeding materials, the hand needs to be close to the table. Therefore, the material feeding height threshold is set to 15mm higher than the table height. When sewing and sewing materials, the hand needs to operate near the table surface. Therefore, the operating height range is set to 5mm less than the table height and 15mm more than the table height.

[0118] Joint angle range: Joint angle data were collected experimentally for four postures: grasping, extending, adjusting, and stabilizing. The finger joint bending angle in the grasping posture is 110°~170°, so the grasping range is set to 120°~160°; the joint angle in the extending posture is 0°~40°, so the extending range is set to 0°~30°; the joint angle in the adjusting posture is 20°~130°, so the adjusting range is set to 30°~120°; and the joint angle fluctuation in the stabilizing posture is ≤15°, so the fluctuation range is set to ≤10°.

[0119] Frame loss determination threshold: By simulating different frame loss scenarios, when the displacement of key points in consecutive frames is ≥10mm, the calculation error is ≥10%. Therefore, the frame loss determination threshold is set to 10mm.

[0120] The solution of this invention solves the problems of existing multimodal data fusion recognition requiring hardware modification of sewing machines to adapt to sensors, resulting in high deployment costs, poor compatibility, inability to adapt to different models and age levels of sewing machines, and insufficient versatility; moreover, sensor data is easily affected by equipment operating conditions, limiting stability. The solution achieves sewing action recognition solely through visual data, reducing deployment threshold and cost, and improving the versatility of the technology.

[0121] The present invention solves the problem of the recognition accuracy deficiency of traditional single vision recognition schemes. Based on the refined detection of 21 key skeletal points of the hand (fingertips, knuckles, wrists), it captures the detailed features of hand posture and achieves accurate differentiation of four types of actions: "picking up materials, sewing materials, arranging materials, and placing materials". This solves the problems of coarse hand feature extraction and high action misjudgment rate of traditional single vision schemes.

[0122] The solution of this invention solves the problem of the lack of quantitative ability of motion efficiency in the prior art. It constructs a quantitative analysis system for motion efficiency, automatically identifies the start and end frames of motion, calculates the actual time consumption, defines the motion time deviation ratio and efficiency improvement potential indicators, and solves the defects of the prior art that cannot be refined to the motion level, lacks objective quantitative basis and optimization space assessment ability, and provides accurate data support for employee skills training and production process optimization.

[0123] The present invention solves the problem of time consumption calculation error caused by frame loss during image acquisition. It adds a frame loss completion mechanism, which uses an interpolation algorithm to complete the data for frame loss during image acquisition or transmission, so as to ensure the accuracy and reliability of action time calculation and efficiency analysis results, and improve the anti-interference ability of the technical solution.

Claims

1. A sewing action recognition method based on key points of hand bones, characterized in that, The method includes the following steps: The images of the sewing machine's working area during the sewing operation are collected as the sole data source, and the images cover the sewing machine table area, the fabric placement area, and the area where the sewing machine operator's hands operate. The image is processed based on a key point detection model to identify the three-dimensional coordinates of twenty-one skeletal key points of the hand. The skeletal key points include the fingertips, proximal phalanges, middle phalanges, distal phalanges, and wrist key points of the thumb, index finger, middle finger, ring finger, and little finger. Extract the feature parameters of the twenty-one skeletal key points. The feature parameters include: the height coordinate of each key point relative to the sewing machine table, the bending angle of the finger joint, the movement trajectory of the key point, the displacement of the key point in adjacent frames, and the temporal change rate. Based on the aforementioned feature parameters and preset action determination rules, the recognition results of four types of sewing actions—picking up material, sewing material, arranging material, and placing material—are obtained.

2. The sewing action recognition method according to claim 1, characterized in that, The keypoint detection model uses the MediaPipeHands model.

3. The sewing action recognition method according to claim 1, characterized in that, The preset action determination rules include: The rules for determining the material handling action are as follows: within a first preset time period, the height of the wrist key point is continuously higher than the preset material handling height, which is the material handling height threshold of 70mm~100mm higher than the height of the sewing machine table, and the bending angle of at least three finger joints is in the grasping range of 110°~170°, while the movement trajectory of the hand key point is towards the fabric placement area; wherein, the first preset time period covers more than 95% of the material handling actions; The rule for determining the feeding action is: within the second preset time period, the height of the wrist key point is continuously lower than the preset feeding height, the feeding height is the height of the sewing machine table increased by 5mm~25mm, and the bending angle of all finger joints is in the extended range of 0°~40°, while the movement trajectory of the hand key point stops in the fabric placement area. The rule for determining the sewing action is: within the third preset time period, the height of the wrist key point is within the preset operation height range. The height range is the third threshold adjusted based on the height of the table. The third threshold is from a decrease of -10mm to an increase of 20mm. The fluctuation range of the finger joint bending angle is less than or equal to 6°~15°, and the displacement of the hand key point between adjacent frames is less than or equal to 2mm / frame~5mm / frame. The rule for determining the sewing action is: after the sewing action is identified, the height of the wrist key point is within the preset operating height range, the movement trajectory of the hand key point is a reciprocating straight line or arc, and the bending angle of the joints of at least two fingers changes periodically within the adjustment range of 20°-130°, while the movement range of the key point covers the fabric outline area.

4. The sewing action recognition method according to claim 3, characterized in that, The preset action determination rules adopt the following parameters: the first preset duration is 2.5 seconds, the material picking height threshold increment is 85mm, and the grasping range is 120°~160°; the second preset duration is 1.8 seconds, the material releasing height threshold increment is 15mm, and the stretching range is 0°~30°; the third preset duration is 2.2 seconds, the operation height range is the sewing machine table height -5mm to +15mm, the joint angle fluctuation range is ≤10°, the displacement of adjacent frames is ≤3mm / frame, and the adjustment range is 30°~120°.

5. The sewing action recognition method according to claim 3, characterized in that, It also includes an action priority determination mechanism, which outputs the recognition results in the order of priority of sewing action > material gathering action > material picking action > material placement action when multiple action determination conditions are met at the same time.

6. The sewing action recognition method according to claim 1, characterized in that, It also includes filtering the coordinate data of the skeletal key points, using the Kalman filter algorithm to eliminate key point jitter caused by image noise, and improving the stability of feature parameters.

7. The sewing action recognition method according to claim 1, characterized in that, The finger joint bending angle is calculated by the vector angle between adjacent bone key points, and the temporal change rate is calculated by fitting the change in key point coordinates of 10 consecutive frames of images.

8. The sewing action recognition method according to claim 1, characterized in that, It also includes frame loss detection and completion steps. When the displacement of the same hand key point in two consecutive frames changes abruptly to a value greater than or equal to the frame loss displacement threshold Dloss and there is no corresponding pose change, it is determined to be a frame loss. The key point coordinates of the lost frame are completed by linear interpolation, and the displacement after completion is verified to be less than or equal to the verification threshold Dcheck, ensuring that the calculation error of the action time is ≤5%.

9. A sewing motion recognition system based on key points of the hand skeleton, characterized in that, The system includes: Image acquisition module: used to acquire images of the sewing machine's working area, covering the sewing machine table area, fabric placement area, and operator's hand operation area; using an industrial camera with a frame rate ≥30fps and an image resolution ≥1920×1080; Key point recognition module: used to perform the method described in any one of claims 1 to 8 to recognize twenty-one key points of the hand skeleton and output the three-dimensional coordinates of the key points; Feature extraction module: used to extract the feature parameters, including height coordinates, joint bending angle, and motion trajectory; and Action recognition module: used to output sewing action recognition results according to the preset action judgment rules and priority mechanism.

10. A terminal, characterized in that, The terminal includes a processor and a memory; the memory is used to store a computer program; the processor is used to execute the computer program stored in the memory to cause the terminal to perform the method according to any one of claims 1 to 8.

Images