Self-learning and detection method for manual assembly process of products on production line

Abstract

The application discloses a self-learning and detection method, in particular to a self-learning and detection method of product manual assembly process on a production line. Firstly, the working area is divided by the edge detection and rectangle detection technology of OpenCV, the spatial information of the working area is saved, and the learning and debugging stage is started; in this stage, multiple skilled workers simulate the assembly action on the work station, the hand positioning frame with direction is acquired by the palm recognition technology in MediaPipe to position the palm position, the region and trajectory information of the palm are determined by comparing the region information, and the trajectory data is aligned to ensure the consistency in time; next, the genetic algorithm is used to find the optimal trajectory; when the workers actually assemble, MediaPipe tracks the palm position and judges the region information, and then compares it with the standard process. The application solves the technical problem of providing the self-learning and detection method of product manual assembly process on a production line, and has the advantages of faster speed, lower cost, easy popularization and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to a self-learning and detection method, in particular to a self-learning and detection method for product manual assembly process on a production line. BACKGROUND

[0002] With the continuous advancement of industrial intelligent digitalization process, the intelligent demand of manufacturing industry has not only been the intelligent of production line. Due to the flexibility, customization, specificity and cost control of small household appliances production, a large number of parts need to be operated by manual selection and installation, which cannot be replaced by machines. Therefore, the detection of missing parts in the production line personnel assembly behavior in the production process is also an important link in the intelligent process of manufacturing industry. Therefore, in the production line management, whether the operation process of the production line front-line staff is strictly standardized and whether the selected parts are correct are important links that affect the production quality. The existing technology usually relies on personnel who have been trained to operate, but manual operation will inevitably have problems such as non-standard process and selection of incorrect parts. Some parts have small differences, which are difficult to accurately distinguish in a short time by naked eye. This may lead to the selection of parts not meeting the needs, and further cause quality problems such as incorrect and missing parts, affecting production efficiency and yield.

[0003] At present, there are many methods for missing part and missing part detection in various industries. In the production of electric control board components, the quality requirements for electric control boards are particularly strict, and there are many components on the electric control board and the distribution is dense. Visual fatigue caused by long-time work of the operator, subjective error of naked eye, discontinuous defect detection, etc. cannot meet the production requirements. Through image processing technology, according to the characteristics of the electric control board, Harris operator is selected to extract feature points, and image gray value template matching method is used to realize the quality detection of missing parts and incorrect parts of the electric control board. This algorithm has a high accuracy for the quality detection of electric control boards.

[0004] In addition to directly detecting the quality of finished products, recognizing the behavior of workers is also an important way in assembly production. Beijing Haiyan Automation Technology Co., Ltd. has researched an error prevention system for household appliance assembly based on pose recognition. The system obtains the image of the assembly station and the operation video stream of the operator, estimates the human pose in the video stream by using AlphaPose algorithm, determines the 2D pixel coordinates of the left wrist and right wrist of the operator, and when the wrists are in the assembly position of the parts, it is determined that the operation is completed and meets the specified assembly process. This scheme realizes real-time monitoring on a high-computing-capability GPU, ensures that the correct parts are installed in the correct way to the correct position, and prevents non-standard operation or misoperation.

[0005] In the production process of automobile automatic production line, the part assembly has the characteristics of large scale, multi-variety and high automation, and a large number of visual online detection special machines are added in the automatic production line. Such as the AI machine vision flexible online detection special machine independently developed by Visionbit robot, the multi-view visual online detection scheme realizes efficient detection of workpieces through online detection by fixed installation of cameras based on full-view angle image acquisition of multiple cameras; the eye-on-hand online detection scheme is adopted, the camera module is carried by the industrial robot, combined with the angle adjustment of the displacement machine for moving and shooting detection, and the self-developed small sample and online learning technology realizes accurate detection of misassembly and missing parts.

[0006] At present, the technology of monitoring whether the operation steps of personnel are omitted is still in the development stage in the industrial market environment. The monitoring scene includes checking whether the workers miss components, miss screws or miss accessories. However, the behavior mistake-proofing technology for personnel operation steps still faces some technical challenges. For example, it is difficult to realize accurate detection by relying only on visual shooting, and there are difficulties in monitoring operation steps, and the adaptability of the scheme is poor.

[0007] Taking the visual detection technology as an example, when data is collected through visual detection, the target object is difficult to ensure the accuracy of detection due to different shapes and sizes, and the shielding of hand actions during assembly and other factors. Therefore, in the assembly production line of small household appliances, it is difficult to realize detection directly through image feature matching method. At the same time, the online detection algorithm based on posture recognition is limited by the model accuracy in the actual production environment and the detection of a large number of stations at the same time, which makes it difficult to ensure the behavior recognition accuracy and the high calculation cost.

[0008] In addition, although the AI machine vision flexible online detection special machine is suitable for high automation production and mass production assembly line, it is too high in cost for the flexibility, customization and cost control of small household appliance production line. When the production scheme and object change, it is difficult to debug and train.

[0009] Therefore, the applicant puts forward the present application. SUMMARY

[0010] The purpose of the present application is to provide a self-learning and detection method for product manual assembly process on production line, which can solve the problem of high cost of retraining and debugging the entire program model when the production scheme and object change to adapt to the change of different assembly processes in assembly behavior detection.

[0011] In order to achieve the above purpose, the self-learning and detection method for product manual assembly process on production line designed by the present application comprises:

[0012] When facing a new assembly process, first, the working area is divided by edge detection and rectangle detection technology of OpenCV, and the working area is divided into part area, conveying area and assembly area by artificial division; Then, save the spatial information of the working area, and start the learning and debugging stage;

[0013] At this stage, several experienced workers simulate assembly actions at the workstations, and by combining the MediaPipe palm detector with the BYTE method, a hand positioning box with direction is obtained to locate the palm position; By comparing the palm position and the working area space, the palm area is determined, and the trajectory information is recorded; After collecting the hand trajectory data of several workers, since the demonstration time of each worker may be different, the dynamic time warping algorithm is used to stretch or compress the trajectory data sequences of different lengths in time to find the best match between them, align them to ensure consistency in time;

[0014] Next, use genetic algorithm to search for trajectory parameters, first initialize the trajectory parameter set as a population, define the fitness function, which is used to evaluate the performance of each individual in the population, in the assembly scene, the fitness score is calculated based on the efficiency of the assembly trajectory and the length of the assembly path, the higher the efficiency of the trajectory and the shorter the path length, the higher the fitness score; Repeat iterations to find the individual with the highest fitness as the optimal trajectory, the sequence of region information changes of this trajectory represents the standard process of assembly behavior, which is used to guide the actual assembly operation, and the trajectory information is used as a reference for detecting operation deviation;

[0015] After getting the standard process, when the operator performs assembly, continue to track the hand trajectory by combining the MediaPipe palm detector with the BYTE method and judge the region information, then compare it with the standard process, when the worker's region information change sequence is inconsistent with the standard process, issue a warning to remind the worker to self-check, and the worker needs to feedback the self-check result after completing the self-check to ensure the integrity of the assembly process; In addition, compare the captured trajectory in real time with the standard process trajectory determined by the genetic algorithm, perform time series analysis on the real-time captured trajectory and the preset standard process trajectory through the dynamic time warping algorithm, to elastically align the trajectory points and measure their similarity, and calculate the minimum cumulative distance between the two trajectories to get their similarity, and convert the distance into a similarity score;

[0016] The palm tracking model based on MediaPipe palm recognition and BYTE method implementation process is as follows:

[0017] 1. Initialization: set an empty trajectory set T to store the tracking information of each target in the video;

[0018] 2. Object detection: For each frame f in the video sequence k , use the hand detector D et to predict the bounding box and score of the object D k ;

[0019] 3. Detection result classification: classify the detection results into high-confidence boxes D high and low-confidence boxes D low according to a score threshold τ;

[0020] 4. Trajectory prediction: use a Kalman filter algorithm to predict the new position of each existing trajectory T in the current frame;

[0021] 5. First data association: associate existing trajectories T with high-confidence boxes D high using a similarity measure ReID;

[0022] 6. Update remaining detections and trajectories: determine the remaining unmatched detection boxes D remain1 and unmatched trajectories T remain1 after the first association;

[0023] 7. Second data association: associate unmatched trajectories T remain1 with low-confidence boxes D low using another similarity measure IoU;

[0024] 8. Clean up unmatched trajectories: remove all unmatched trajectories T remain2 from the trajectory set T that are not matched in the second association;

[0025] 9. Initialize new trajectories: for each unmatched detection box D remain2 remaining after the second association, initialize a new trajectory for each unmatched detection box D remain2 and add it to the trajectory set T;

[0026] 10. Return results: return the updated trajectory set T, representing the tracking paths of all objects in the video.

[0027] For small household appliance production process, the assembly behavior has flexibility, assembly process complexity, customization and cost control characteristics, under the premise of ensuring recognition accuracy, a more rapid, low-cost, easy-to-promote production line product manual assembly process self-learning and detection method is provided. Compared with the prior art, the method has the following technical advantages:

[0028] 1. Use hand target tracking algorithm instead of pose recognition algorithm to realize wrist positioning in assembly behavior. In actual production process, due to the existence of a large number of occlusions in human body, the accuracy of existing pose recognition algorithm is difficult to guarantee, most of the recognized joint nodes are not used, mainly relying on wrist joint node, and the required computing power is higher. Use mediapipe based on ML palm detector, which can detect occluded and self-occluded hands, achieve higher average accuracy of 95.7%, and require less computing power.

[0029] 2. Combine MediaPipe palm detector with BYTE method to ensure tracking hand trajectory, use the similarity between detection frame and tracking trajectory, remove background from low-score detection results while retaining high-score detection results, and mine out the real object (even in difficult samples such as hand occlusion, blur, etc.), so as to reduce missed detection and improve the continuity of trajectory. BRIEF DESCRIPTION OF DRAWINGS

[0030] Figure 1 is the logic flow chart of the self-learning and detection method of the product manual assembly process on the production line;

[0031] Figure 2 is the flow chart of the palm tracking model based on MediaPipe palm recognition and BYTE method. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application.

[0033] As shown in Figure 1 , the self-learning and detection method of the product manual assembly process on the production line provided by the present application comprises:

[0034] When facing a new assembly process, first, the working area is divided by the edge detection and rectangular detection technology of OpenCV, and the working area is divided into a part area, a conveying area and an assembly area by manual. Then, save the spatial information of the working area, and start the learning and debugging stage.

[0035] In this stage, several experienced workers simulate assembly actions at their workstations. By combining MediaPipe's hand detector with the BYTE method, directional hand positioning boxes are obtained to locate the hand positions. The hand area is determined by comparing the hand position with the work area space, and trajectory information is recorded simultaneously. After collecting hand trajectory data from multiple workers, since the demonstration time for each worker may vary, a Dynamic Time Warping (DTW) algorithm is used to stretch or compress trajectory data sequences of different lengths in time to find the best match between them and align them to ensure temporal consistency.

[0036] Next, a Genetic Algorithm (GA) is used to optimize the trajectory parameters. First, the trajectory parameter set is initialized as a population, and a fitness function is defined to evaluate the performance of each individual in the population. In the assembly scenario, the fitness score can be calculated based on the efficiency of the assembly trajectory (e.g., the time required to complete the task) and the assembly path length. Higher trajectory efficiency and shorter path length should result in a higher fitness score. The process is iteratively repeated to find the individual with the highest fitness as the optimal trajectory. The order of changes in the region information of this trajectory represents the standard process of assembly behavior and can be used to guide actual assembly operations. The trajectory information serves as a reference for detecting operational deviations.

[0037] After obtaining the standard process, when operators perform assembly, the system continues to track their hand trajectories and determine area information by combining MediaPipe's hand detector with the BYTE method. This information is then compared to the standard process. If the worker's area information changes in a different order than the standard process (indicating misoperation or omission), the program issues a warning (signal light) to remind the worker to perform a self-inspection. After completing the self-inspection, the worker must report the results to ensure the integrity of the assembly process. Furthermore, the captured trajectories are compared in real-time with the standard process trajectories determined by a genetic algorithm. A dynamic time warping algorithm is used to first perform time series analysis on the real-time captured trajectories and the preset standard process trajectories to flexibly align trajectory points and measure their similarity. Then, the dynamic time warping algorithm calculates the minimum cumulative distance between the two trajectories to determine their similarity level, which can be converted into a similarity score. By statistically recording the timestamps of workers entering different areas, their dwell time in each area can be calculated, providing fine-grained recording of the production line's cycle time. This data can be used to analyze the production line's cycle time and efficiency, as well as predict potential bottleneck areas.

[0038] Among them, such as Figure 2 As shown, the implementation process of the hand tracking model based on MediaPipe hand recognition and the BYTE method is as follows:

[0039] 1. Initialization: Set an empty track set T for storing tracking information of each target in the video;

[0040] 2. Target detection: For each frame f in the video sequence k , use a hand detector D et to predict the detection box and score D k of the target;

[0041] 3. Detection result classification: Classify the detection results into high-confidence boxes D high and low-confidence boxes D low according to a score threshold τ;

[0042] 4. Track prediction: Use Kalman filter algorithm to predict the new position of each existing track T in the current frame;

[0043] 5. First data association: Use a similarity measure ReID to associate existing tracks T with high-confidence boxes D high ;

[0044] 6. Update remaining detections and tracks: Determine the remaining unmatched detection boxes D remain1 and unmatched tracks T remain1 after the first association;

[0045] 7. Second data association: Use another similarity measure IoU to associate unmatched tracks T remain1 with low-confidence boxes D low ;

[0046] 8. Clean up unmatched tracks: Remove all unmatched tracks T remain2 that are not matched in the second association from the track set T;

[0047] 9. Initialize new tracks: For each unmatched detection box D remain2 remaining after the second association, initialize a new track for each unmatched detection box D remain2 and add it to the track set T;

[0048] 10. Return results: Return the updated track set T, representing the tracking paths of all targets in the video.

[0049] In the assembly production of small household appliances, the worker's hand often holds the parts. In the training without the assembly hand model, these incomplete hands (holding parts) are often biased towards traditional multi-target tracking methods, which often only consider high-confidence detection results when data association, while ignoring those with lower confidence. The breakthrough of the BYTE method is that it makes full use of these low-confidence detection results to assist the tracking process according to their motion consistency and appearance information. This method significantly improves the continuity and robustness of tracking, especially in handling occlusions and dynamic scenes, effectively reducing ID switching and target loss.

Claims

1. A self-learning and detection method for manual product assembly processes on a production line, characterized by: The steps include the following: When faced with a new assembly process, the work area is first divided using OpenCV edge detection and rectangle detection techniques. The work area is then manually divided into a parts area, a transfer area, and an assembly area. Then, save the work area space information and begin the learning and debugging phase; At this stage, several experienced workers simulate assembly actions at their workstations. By combining MediaPipe's hand detector with the BYTE method, directional hand positioning boxes are obtained to locate the hand position. The hand area is determined by comparing the hand position with the work area space, and trajectory information is recorded. After collecting hand trajectory data from multiple workers, since the demonstration time of each worker may be different, a dynamic time warping algorithm is needed to stretch or compress trajectory data sequences of different lengths in time in order to find the best match between them and align them to ensure temporal consistency. Next, a genetic algorithm is used to optimize the trajectory parameters. First, the trajectory parameter set is initialized into a population, and a fitness function is defined to evaluate the performance of each individual in the population. In the assembly scenario, the fitness score is calculated based on the efficiency of the assembly trajectory and the length of the assembly path. The higher the efficiency of the trajectory and the shorter the path length, the higher the fitness score. The individual with the highest fitness is repeatedly iterated to find the optimal trajectory. The order of changes in the regional information of this trajectory represents the standard process of assembly behavior and is used to guide the actual assembly operation. The trajectory information is used as a reference for detecting operational deviations. After obtaining the standard process, when the operator is assembling, the hand trajectory is tracked and the area information is judged by combining the hand detector of MediaPipe with the BYTE method. Then it is compared with the standard process. If the order of the worker's area information changes is inconsistent with the standard process, a warning is issued to remind the worker to perform self-inspection. After the worker completes the self-inspection, he / she must report the self-inspection results to ensure the integrity of the assembly process. In addition, the captured trajectory is compared with the standard process trajectory determined by the genetic algorithm in real time. The time series analysis of the real-time captured trajectory and the preset standard process trajectory is performed by the dynamic time warping algorithm to flexibly align the trajectory points and measure their similarity. The minimum cumulative distance between the two trajectories is calculated to obtain their similarity and the distance is converted into a similarity score. The implementation process of the palm tracking model based on MediaPipe palm recognition and the BYTE method is as follows:

1. Initialization: Set an empty trajectory set T to store the tracking information of each target in the video; 2. Object Detection: For each frame f in the video sequence k Using a palm detector D et Predicted target detection box and score D k ; 3. Detection Result Classification: The detection results are classified into high-confidence boxes D based on the score threshold τ. high and low confidence box D low ; 4. Trajectory Prediction: The Kalman filter algorithm is used to predict the new position of each existing trajectory T in the current frame; 5. First data association: Using the similarity metric ReID, the existing trajectory T is compared with the high-confidence bounding box D. high To establish a connection; 6. Update remaining detections and trajectories: Determine the remaining unmatched bounding boxes D after the first association. remain1 and unmatched trajectory T remain1 ; 7. Second Data Association: Using another similarity metric, IoU, to link unmatched trajectories T remain1 With low confidence box D low To establish a connection; 8. Clean up unmatched trajectories: Delete all unmatched trajectories T from the trajectory set T that were not matched in the second association. remain2 ; 9. Initialize the new trajectory: For the remaining unmatched detection boxes D after the second association... remain2 For each unmatched detection box D remain2 Initialize a new trajectory and add it to the trajectory set T; 10. Return result: Returns the updated trajectory set T, representing the tracking paths of all targets in the video.

Images