A method for predicting production line operation intentions and scheduling computing power based on hand movement characteristics.

Abstract

This invention discloses a method for predicting operational intentions and scheduling computing power on production lines based on hand motion characteristics, belonging to the field of intelligent manufacturing technology. Its purpose is to solve the problems of numerous invalid alarms, wasted computing power, and high response latency in existing production line assembly inspection. This method first calibrates and divides workstation areas, and obtains a standard assembly process through self-learning modeling; then, it extracts hand motion characteristics using a YOLO model, and predicts the operator's hand and non-operator's hand based on kinematic functions; it dynamically schedules computing power, initiating high-precision compliance inspection only for operators, and periodically iterates and updates the standard process. This invention can accurately filter invalid targets, significantly reduce computing power consumption and detection latency, improve alarm accuracy, adapt to large-scale deployment of flexible production lines, be compatible with existing systems, and has low upgrade costs.

Description

Technical Field

[0001] This invention belongs to the field of intelligent manufacturing and industrial machine vision technology, specifically relating to a method for detecting the operation behavior of manual assembly stations on a production line, and more particularly to a method for predicting the operation intention of a production line based on hand movement characteristics and scheduling computing power. Background Technology

[0002] With the rapid advancement of industrial intelligence, many customized and flexible assembly processes in discrete manufacturing industries such as small home appliances and 3C products still rely on manual labor. The compliance of manual assembly processes directly determines the final product quality and production yield. Currently, error-proofing and detection technologies for manual assembly processes have become a research hotspot in the field of intelligent manufacturing.

[0003] The existing authorized patent (Patent No.: 202410634132.3, Patent Name: Self-learning and Detection Method for Manual Assembly Process of Products on Production Line) provides a self-learning and detection scheme for assembly process based on hand trajectory tracking. This scheme obtains hand trajectories by combining a MediaPipe (MediaPipe Hands) hand detector with the BYTE (ByteTrack) tracking method, achieves trajectory alignment through a Dynamic Time Warping (DTW) algorithm, and optimizes the standard assembly process through a Genetic Algorithm (GA), thereby realizing the detection of assembly operation violations. Compared with traditional human posture recognition schemes, this scheme significantly improves the accuracy and robustness of hand tracking, reduces computing power consumption, and is suitable for customized assembly scenarios on flexible production lines.

[0004] However, in the actual large-scale deployment of the above technical solutions, the following technical defects still need to be addressed:

[0005] 1. Frequent invalid alarms significantly reduce system reliability: Current solutions perform full-process compliance detection on all hand targets entering the camera's field of view. However, the hand movements of personnel frequently passing through the production line environment (such as inspection managers and material handlers) often do not conform to standard operating procedures in terms of area switching sequence and trajectory. This makes them highly susceptible to being misjudged by the system as violations and triggering alarms. A large number of invalid alarms can obscure real quality hazards, leading to distrust of the system among on-site personnel and even causing them to directly disable the alarm function, rendering the detection system ineffective.

[0006] 2. Meaningless waste of computing resources and high deployment costs: Existing solutions cannot distinguish between "operating hands" with assembly intentions and "passing hands" without such intentions. High-precision MediaPipe tracking, trajectory comparison, and compliance analysis models are initiated for all hand targets entering the screen. In production line scenarios involving concurrent processing of multiple high-definition video streams, the full computation of numerous invalid targets consumes enormous amounts of GPU / CPU computing power, increasing not only server hardware procurement and operating electricity costs but also causing the system to operate under high load for extended periods, leading to decreased stability.

[0007] 3. High detection response latency, failing to meet real-time intervention requirements: The enormous computational overhead of full-scale computation can lead to queue congestion during multi-channel video stream processing, significantly increasing the delay from the occurrence of an operation to the system's output of a compliance judgment result. For assembly scenarios requiring real-time alerts and immediate correction of erroneous operations, excessive latency prevents timely intervention in violations, ultimately resulting in defective products and negating the core purpose of detection.

[0008] In addition, other existing assembly behavior detection solutions either rely on dedicated vision inspection equipment with high computing power, which has poor adaptability to flexible production lines and high deployment costs; or they can only achieve post-assembly finished product inspection and cannot achieve real-time intervention during the assembly process. None of them can simultaneously solve the above three major technical problems. Therefore, in order to adapt to the actual needs of large-scale deployment of flexible production lines, the applicant has proposed the technical solution of this invention. Summary of the Invention

[0009] The purpose of this invention is to overcome the above-mentioned defects in the prior art and provide a method for predicting operation intentions and scheduling computing power based on hand movement characteristics. By constructing a two-stage hierarchical heterogeneous computing architecture, the method first achieves operation intention prediction through lightweight detection to distinguish between operators and non-operators. Then, based on the prediction results, the computing power is dynamically scheduled to initiate high-precision compliance detection only for operators, thereby fundamentally solving the problems of numerous invalid alarms, serious waste of computing power, and high detection latency in the prior art.

[0010] To achieve the above-mentioned objectives, the production line operation intention prediction and computing power scheduling method based on hand movement characteristics designed in this invention specifically includes the following steps:

[0011] S1 Work Area Calibration and Division: For the target assembly station, the station image captured by the camera is divided into areas using OpenCV edge detection and rectangle detection technology. The work area within the image is manually divided into parts area, conveying area and assembly area, and the spatial coordinate information of each area is saved.

[0012] S2 Standard Assembly Process Self-Learning and Modeling: Entering the learning and debugging phase, standard assembly action data of multiple skilled workers are collected. The trajectory sequence and area switching information of the workers' hands are obtained by using the MediaPipe (MediaPipe Hands) hand detector combined with the BYTE (ByteTrack) tracking method. The Dynamic Time Warping (DTW) algorithm is used to align the trajectory sequences of different lengths in time. Then, the trajectory parameters are optimized by the Genetic Algorithm (GA) to obtain a standard assembly process adapted to the target assembly process. The standard assembly process includes a standard area switching sequence, a standard trajectory template, and a compliance judgment threshold.

[0013] S3 Lightweight Real-time Hand Detection and Motion Feature Extraction: For video streams acquired in real time at the workstation, a lightweight YOLO model is used to detect hand targets in each frame, obtain the detection boxes and real-time coordinates of all hand targets in the frame, construct a time series function of hand position changing over time, and calculate the instantaneous velocity time series function and instantaneous acceleration time series function of the hand based on the time series function.

[0014] S4 Operation Intent Prediction and Dynamic Computing Power Scheduling: Based on the instantaneous acceleration time series function of the hand, the acceleration stationary point is identified, and it is determined whether the stationary point is a velocity minimum point. If the velocity minimum of the hand is equal to the preset zero velocity threshold, the hand is determined to be an operator with assembly operation intent, triggering a computing power scheduling command and starting a high-precision assembly behavior compliance detection link; if the hand does not have a velocity minimum point that meets the requirements, the hand is determined to be a non-operator hand, the high-precision detection link is not started, and only lightweight hand detection continues.

[0015] S5 Assembly Behavior Compliance Real-time Detection: For hand targets identified as operators, the MediaPipe hand detector combined with the BYTE tracking method is activated to track hand trajectories and area switching information in real time. The real-time collected area switching sequence is compared with the standard area switching sequence of the standard assembly process. If the sequence is inconsistent, a violation alarm is triggered. At the same time, a dynamic time warping algorithm is used to calculate the similarity score between the real-time trajectory and the standard trajectory template, and the operation compliance is determined based on the compliance judgment threshold.

[0016] S6 Standard Assembly Process Iterative Update: Collect assembly trajectory data of operators deemed compliant, and periodically iterate and optimize the standard assembly process using dynamic time warping algorithm and genetic algorithm, updating the standard area switching sequence, standard trajectory template and compliance judgment threshold.

[0017] Furthermore, in step S3, the time-series function of the hand position changing over time is:

[0018]

[0019] in , These are the horizontal and vertical pixel coordinates of the center point of the hand detection box at time 𝑡, respectively;

[0020] Based on the aforementioned hand position timing function, the instantaneous velocity timing function is obtained through differentiation, and its expression is as follows:

[0021]

[0022] Based on the instantaneous velocity time series function, the instantaneous acceleration time series function is obtained through second derivative operation, and its expression is:

[0023] .

[0024] Furthermore, in step S4, the criteria for identifying acceleration stagnation points are: ,in The time corresponding to the stationary point; the criterion for determining the velocity minimum point is: when hour ,when hour ,and instantaneous velocity at time 1 The modulus is less than or equal to the preset zero-speed threshold.

[0025] Furthermore, the zero-speed threshold is set according to the frame rate and pixel accuracy of the workstation video stream, and the value range is 0-5 pixels / frame.

[0026] Furthermore, in step S4, non-operating hands include the hands of personnel passing by the workstation and the hands of personnel without the intention to perform assembly operations. For non-operating hands, only the lightweight hand detection in step S3 is performed, and the high-precision compliance detection in step S5 is not performed.

[0027] Furthermore, in step S2, the fitness function of the genetic algorithm is constructed based on the completion time and path length of the assembly trajectory. The shorter the trajectory completion time and the shorter the path length, the higher the fitness score. After iteration, the trajectory and region switching order corresponding to the individual with the highest fitness score are selected as the standard assembly process.

[0028] Furthermore, in step S5, the violation alarm includes audible and visual alarms and workstation terminal pop-up alarms. After the alarm is triggered, the operator's self-inspection results need to be received. After confirming that the self-inspection is completed, the alarm is cleared and the operation log is recorded.

[0029] Furthermore, in step S5, the similarity score is calculated as follows: the minimum cumulative distance between the real-time trajectory and the standard trajectory template is calculated using a dynamic time warping algorithm, and the minimum cumulative distance is mapped to a similarity score of 0-100. When the similarity score is lower than the compliance judgment threshold, an operation deviation alarm is triggered.

[0030] Furthermore, in step S6, the iterative update cycle of the standard assembly process is configurable. During the update, only compliant assembly trajectory data is used, and the compliance judgment threshold in step S5 is updated synchronously after iterative optimization.

[0031] Furthermore, in step S3, the YOLO model adopts the YOLOv8n lightweight model, the frame rate of the input video stream is 15-30fps, and the inference time per frame does not exceed 10ms.

[0032] Compared with the prior art, the present invention has the following significant technical and beneficial effects:

[0033] 1. Eliminate invalid alarms at the source and significantly improve system reliability and usability: This invention predicts operation intentions by using hand kinematic characteristics, accurately distinguishing between operators with assembly intentions and passersby without such intentions. Compliance checks are only performed on operators, completely avoiding invalid alarms caused by passersby. Actual testing shows that the alarm accuracy rate can be increased to over 95%, solving the core pain point of invalid alarms drowning out real hidden dangers and the abandonment of the system in existing technologies.

[0034] 2. Significantly Reduced Computing Power Consumption and Lowered Large-Scale Deployment Costs: This invention constructs a two-stage hierarchical heterogeneous computing architecture. The first stage uses a lightweight YOLO model with a single-frame inference time of no more than 10ms for hand detection and intent prediction. Only when the hand is identified as an operator's hand is the second-stage high-precision MediaPipe tracking and compliance analysis model activated. Actual testing shows that in typical production line scenarios, the overall system computing power consumption can be reduced by 60%-80%. When processing multiple video streams concurrently, it can significantly reduce server hardware procurement costs and operating power costs, adapting to the large-scale deployment needs of flexible production lines.

[0035] 3. Significantly reduced detection latency to meet real-time intervention requirements: By filtering out a large number of meaningless calculations of invalid targets, the system processing queue congestion problem is completely solved. The response latency for compliance detection of valid operators can be reduced to less than 50ms, realizing real-time capture of operation actions and immediate alarm of violations. It can intervene at the first moment when erroneous operation occurs, avoid the generation of defective products, and truly realize closed-loop control of the entire assembly process.

[0036] 4. Compatible with existing technical solutions, low upgrade and transformation costs: The technical solution of this invention is fully compatible with the self-learning and detection process of the existing authorized patent (202410634132.3). Existing deployed systems only need to add a lightweight intent prediction and computing power scheduling module to complete the upgrade and transformation. There is no need to replace hardware equipment or remodel the process. It has strong compatibility and practicality. Attached Figure Description

[0037] Figure 1 This is the overall flowchart of the production line operation intention prediction and computing power scheduling method based on hand movement characteristics described in this invention;

[0038] Figure 2 This is a schematic diagram of the kinematic analysis and determination process for predicting operational intent in this invention. Detailed Implementation

[0039] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0040] like Figure 1 As shown, the production line operation intention prediction and computing power scheduling method based on hand movement characteristics described in this embodiment is applied to the manual assembly station of a flexible assembly production line for small household appliances, and specifically includes the following complete implementation steps:

[0041] I. Preliminary Preparation and Calibration Stage

[0042] S1 Work Area Marking and Division

[0043] For the target small appliance assembly station, a 2-megapixel, 25fps high-definition network camera is installed directly above the station. The camera's field of view covers the entire assembly area. The camera is connected to the station's edge computing terminal via a network cable. The edge computing terminal runs on the Windows 10 operating system and is equipped with an 8-core CPU and a GPU with 4GB of video memory to meet real-time processing requirements.

[0044] Static images of the workstation are captured by an edge computing terminal. The OpenCV Canny edge detection algorithm is used to identify the edge contours of the workbench, material box, and conveyor belt within the image. Then, a rectangle detection algorithm is used to extract the bounding rectangles of each area. These rectangles are manually marked by on-site process personnel, dividing the work area within the image into three core regions:

[0045] Parts area: The material box area where parts to be assembled are placed, storing the pixel coordinates of the top left and bottom right corners of each area;

[0046] Conveying area: The area where the production line conveyor belt is located, used to transport products to be assembled and finished products;

[0047] Assembly area: The core assembly operation area on the workbench, used to install and fix parts.

[0048] After calibration is completed, the spatial coordinate information of the three regions is stored in the system configuration file for use in subsequent steps.

[0049] S2 Standard Assembly Process Self-Learning and Modeling

[0050] After completing the regional calibration, the system enters the learning and debugging phase. For the small appliance assembly process at the target workstation, five skilled workers with more than three years of assembly experience are organized to conduct a simulated demonstration of standard assembly actions at the workstation. Each worker repeats the standard assembly process 20 times, and the system collects video stream data of all demonstration processes.

[0051] For each demonstration video, the system uses the MediaPipe hand detector combined with the BYTE multi-object tracking method to obtain the directional bounding box of the worker's hand in real time, extract the coordinates of the center point of the bounding box, and combine it with the area coordinates marked in step S1 to determine the area where the hand is located. At the same time, it records the complete trajectory sequence of the hand and the timestamp information of the area switching.

[0052] After all demonstration data collection was completed, the system adopted the Dynamic Time Warping (DTW) algorithm to stretch and compress the time axis of all collected trajectory sequences of different lengths, since the assembly demonstration time of different workers varied. This achieved time alignment of different trajectory sequences and eliminated the impact of time differences.

[0053] After completing trajectory alignment, the system uses a genetic algorithm to optimize the trajectory parameters. The specific steps are as follows:

[0054] 1. Initialize all aligned trajectory parameters into a population with a population size of 100 and an iteration count of 200.

[0055] 2. Define the fitness function. The formula for calculating the fitness function is: Fitness = 0.6*(1 / T) + 0.4*(1 / L), where T is the completion time of a single wheel assembly and L is the total path length of the hand trajectory. That is, the shorter the assembly completion time and the shorter the trajectory path, the higher the fitness score.

[0056] 3. Perform selection, crossover, and mutation operations, and iterate repeatedly to optimize. Finally, select the individual with the highest fitness score after the iteration as the optimal standard trajectory.

[0057] 4. Extract the order of regional information changes corresponding to the optimal standard trajectory as the standard regional switching order, and set compliance judgment thresholds, including regional order violation judgment rules and trajectory similarity qualification threshold (set to 75 points).

[0058] The final standard assembly process adapted to the target assembly process is obtained, including the standard area switching sequence, standard trajectory template and compliance judgment threshold, and stored in the system database.

[0059] II. Real-time Operation Phase

[0060] S3 Lightweight Real-time Hand Detection and Motion Feature Extraction

[0061] After the system enters real-time operation mode, the camera continuously captures real-time video streams from the workstation at a frame rate of 25fps. Each frame is input into the pre-trained YOLOv8n lightweight hand detection model, which has been trained in advance using 100,000 hand images from the production line environment. The average accuracy (AP) of hand detection reaches over 98%, and the inference time per frame does not exceed 8ms.

[0062] For each frame, the YOLO model outputs the bounding boxes and confidence scores for all hand targets within the frame, filters out detection results with confidence scores below 0.5, extracts the center pixel coordinates of the valid hand bounding boxes, and constructs a temporal function for each hand target to show how its position changes over time.

[0063] For each hand target, a time series function is constructed to describe the change in its position over time, defined as:

[0064]

[0065] in, The timestamp corresponding to the frame. for The horizontal pixel coordinates of the center point of the hand at any given time. for The vertical pixel coordinates of the center point of the hand at any given time.

[0066] For continuous time-series coordinate data, the system uses the central difference method to find the first derivative of the time-series function, thus obtaining the instantaneous velocity time-series function of the hand:

[0067] For continuous time-series coordinate data, the system uses the central difference method to obtain the first derivative of the position time-series function, thus obtaining the instantaneous velocity time-series function of the hand. Its engineering implementation expression is as follows:

[0068]

[0069] in The time interval between two adjacent frames; in this embodiment, a frame rate of 25fps corresponds to... It takes 0.04 seconds.

[0070] Taking the first derivative of the instantaneous velocity time series function, we obtain the instantaneous acceleration time series function of the hand:

[0071] Taking the first derivative of the instantaneous velocity time series function, we obtain the instantaneous acceleration time series function of the hand, whose engineering implementation expression is as follows:

[0072]

[0073] in , They are respectively The instantaneous speed of the hand in both horizontal and vertical directions at all times.

[0074] The system continuously updates the temporal data of the position, velocity, and acceleration of each hand target in the image, providing data support for subsequent intention prediction.

[0075] S4 Operation Intent Prediction and Dynamic Computing Power Scheduling

[0076] like Figure 2 As shown, for each hand target's acceleration time-series function, the system performs the following real-time intention prediction and computing power scheduling steps:

[0077] 1. Acceleration stationary point identification: Traverse the acceleration time series data and identify points that satisfy the following conditions. The moment This moment is the stationary point of acceleration;

[0078] 2. Minimum point determination: For the identified stationary points To determine whether it is a point of minimum velocity, the criterion is: when hour, (The hand makes a decelerating motion); when hour, (The hand makes an accelerated movement); conditions are met. This is the point where the velocity is at its minimum.

[0079] 3. Zero-speed determination: Calculate the speed minimum point. Corresponding instantaneous velocity Length of the module In this embodiment, the preset zero-speed threshold is 3 pixels / frame. If the resolution is 1 pixel / frame, it is determined that the hand has made a valid stop at that moment and is an operator's hand with the intention of assembly operation;

[0080] 4. Dynamic computing power scheduling:

[0081] If the target is identified as an operator: the system immediately triggers a computing power scheduling command, initiates a high-precision assembly behavior compliance detection link, allocates computing power resources to the target hand, and executes the compliance detection in step S5;

[0082] If no minimum speed point that meets the requirements is identified: the hand is determined to be a non-operating hand, including personnel passing by the workstation or personnel without assembly intention. The system does not activate the high-precision detection link, but only continues to track the hand through the lightweight YOLO model in step S3 until it leaves the camera's field of view.

[0083] In this embodiment, for non-operating hands, the system only performs YOLO detection with a single frame time of 8ms, without starting subsequent high-computation-consuming modules such as MediaPipe tracking and DTW trajectory comparison, thereby reducing invalid calculations at the source.

[0084] S5 Assembly Behavior Compliance Real-time Detection

[0085] For the hand target identified as the operator's hand in step S4, the system activates the MediaPipe hand detector combined with the BYTE tracking method to track the complete trajectory and area switching information of the hand in real time, and performs the following compliance detection steps:

[0086] 1. Area Sequence Compliance Detection: The system records the area switching sequence of the hands in real time and compares it with the standard area switching sequence in the standard assembly process. If the sequence is reversed or an area is omitted, the system immediately triggers a violation alarm, including an audible and visual alarm at the workstation and a pop-up alarm on the workstation operation terminal, prompting the operator to perform a self-check immediately. After the operator completes the self-check, they need to report the self-check results on the terminal. The system confirms the self-check and then cancels the alarm, and records the complete violation event and self-check results in the operation log.

[0087] 2. Trajectory Similarity Compliance Detection: For the operator's single-round complete assembly trajectory, the Dynamic Time Warping (DTW) algorithm is used to align the real-time trajectory with the standard trajectory template on the time axis, calculate the minimum cumulative distance between the two trajectories, and then linearly map the minimum cumulative distance to a similarity score of 0-100. The higher the score, the higher the fit between the trajectory and the standard process. If the similarity score is lower than the preset qualified threshold of 75 points, the system triggers an operation deviation alarm, prompting the operator to optimize the assembly action.

[0088] In this embodiment, the high-precision detection is performed only on targets identified as operators, ensuring that all computing resources are used for the effective control of assembly operations and avoiding meaningless waste of computing power.

[0089] S6 Standard Assembly Process Iterative Update

[0090] The system is set to iterate and update the standard assembly process once a week. During the update cycle, the assembly trajectory data of all operators who are judged to be compliant is collected, and the newly added compliant trajectory data is added to the modeling database. The DTW trajectory alignment and genetic algorithm optimization process in step S2 is repeated to iteratively optimize the standard assembly process, update the standard area switching order, standard trajectory template and compliance judgment threshold, so that the standard process continuously adapts to the actual operation of the production line and continuously improves the accuracy and adaptability of the detection.

[0091] In actual production line deployment testing, the technical solution of this embodiment achieves the following technical effects compared to existing technologies:

[0092] 1. The number of invalid alarms was reduced by 92%, and the alarm accuracy rate was improved to 96.8%, which completely solved the problem of false alarms caused by passers-by, and the on-site operators' acceptance of the system was greatly improved;

[0093] 2. The overall computing power consumption of the system was reduced by 76%, the average CPU load of the edge computing terminal decreased from 85% to 22%, and the average GPU load decreased from 78% to 18%, resulting in a significant improvement in operational stability;

[0094] 3. The compliance inspection response latency has been reduced from 280ms to 32ms, enabling real-time capture and immediate alarm of violations. This allows for intervention at the first sign of an error, resulting in a 31% reduction in assembly defect rate.

[0095] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for predicting production line operation intentions and scheduling computing power based on hand movement characteristics, characterized in that, Includes the following steps: S1 Work Area Calibration and Division: For the target assembly station, the station image captured by the camera is divided into areas using OpenCV edge detection and rectangle detection technology. The work area within the image is manually divided into parts area, conveying area and assembly area, and the spatial coordinate information of each area is saved. S2 Standard Assembly Process Self-Learning and Modeling: Entering the learning and debugging phase, standard assembly action data of multiple skilled workers are collected. The trajectory sequence and area switching information of the workers' hands are obtained by using the MediaPipe hand detector combined with the BYTE tracking method. The dynamic time warping algorithm is used to align the trajectory sequences of different lengths in time. Then, the trajectory parameters are optimized by the genetic algorithm to obtain a standard assembly process adapted to the target assembly process. The standard assembly process includes a standard area switching sequence, a standard trajectory template, and a compliance judgment threshold. S3 Lightweight Real-time Hand Detection and Motion Feature Extraction: For video streams acquired in real time at the workstation, the YOLO model is used to detect hand targets in each frame, obtain the detection boxes and real-time coordinates of all hand targets in the frame, construct a time series function of hand position change over time, and calculate the instantaneous velocity time series function and instantaneous acceleration time series function of the hand based on the time series function. S4 Operation Intent Prediction and Dynamic Computing Power Scheduling: Based on the instantaneous acceleration time series function of the hand, the acceleration stationary point is identified, and it is determined whether the stationary point is a velocity minimum point. If the velocity minimum of the hand is equal to the preset zero velocity threshold, the hand is determined to be an operator with assembly operation intent, triggering a computing power scheduling command and starting a high-precision assembly behavior compliance detection link; if the hand does not have a velocity minimum point that meets the requirements, the hand is determined to be a non-operator hand, the high-precision detection link is not started, and only lightweight hand detection continues. S5 Assembly Behavior Compliance Real-time Detection: For hand targets identified as operators, the MediaPipe hand detector combined with the BYTE tracking method is activated to track hand trajectories and area switching information in real time. The real-time collected area switching sequence is compared with the standard area switching sequence of the standard assembly process. If the sequence is inconsistent, a violation alarm is triggered. At the same time, a dynamic time warping algorithm is used to calculate the similarity score between the real-time trajectory and the standard trajectory template, and the operation compliance is determined based on the compliance judgment threshold. S6 Standard Assembly Process Iterative Update: Collect assembly trajectory data of operators deemed compliant, and periodically iterate and optimize the standard assembly process using dynamic time warping algorithm and genetic algorithm, updating the standard area switching sequence, standard trajectory template and compliance judgment threshold.

2. The method according to claim 1, characterized in that, In step S3, the time-series function of the hand position changing over time is defined as: in , These are the horizontal and vertical pixel coordinates of the center point of the hand detection box at time 𝑡, respectively; Based on the aforementioned hand position timing function, the instantaneous velocity timing function is obtained through differentiation, and its expression is as follows: Based on the instantaneous velocity time series function, the instantaneous acceleration time series function is obtained through second derivative operation, and its expression is: 。 3. The method according to claim 2, characterized in that, In step S4, the criteria for identifying acceleration stagnation points are: ,in The time corresponding to the stationary point; the criterion for determining the velocity minimum point is: when hour ,when hour ,and instantaneous velocity at time 1 The modulus is less than or equal to the preset zero-speed threshold.

4. The method according to claim 3, characterized in that, The zero-speed threshold is set according to the frame rate and pixel precision of the workstation video stream, and its value ranges from 0 to 5 pixels per frame.

5. The method according to claim 1, characterized in that, In step S4, non-operating hands include the hands of personnel passing by the workstation and the hands of personnel without the intention to perform assembly operations. For non-operating hands, only the lightweight hand detection in step S3 is performed, and the high-precision compliance detection in step S5 is not performed.

6. The method according to claim 1, characterized in that, In step S2, the fitness function of the genetic algorithm is constructed based on the completion time and path length of the assembly trajectory. The shorter the completion time and path length, the higher the fitness score. After iteration, the trajectory and region switching order corresponding to the individual with the highest fitness score are selected as the standard assembly process.

7. The method according to claim 1, characterized in that, In step S5, the violation alarms include audible and visual alarms and workstation terminal pop-up alarms. After the alarm is triggered, the operator's self-inspection results need to be received. After confirming that the self-inspection is completed, the alarm is cleared and the operation log is recorded.

8. The method according to claim 1, characterized in that, In step S5, the similarity score is calculated as follows: the minimum cumulative distance between the real-time trajectory and the standard trajectory template is calculated by using a dynamic time warping algorithm, and the minimum cumulative distance is mapped to a similarity score of 0-100. When the similarity score is lower than the compliance judgment threshold, an operation deviation alarm is triggered.

9. The method according to claim 1, characterized in that, In step S6, the iterative update cycle of the standard assembly process is configurable. During the update, only compliant assembly trajectory data is used, and the compliance judgment threshold in step S5 is updated synchronously after iterative optimization.

10. The method according to claim 1, characterized in that, In step S3, the YOLO model adopts the YOLOv8n lightweight model, the frame rate of the input video stream is 15-30fps, and the inference time per frame does not exceed 10ms.

Images