A material picking specification detection system, method and storage medium

Abstract

A material picking specification detection system, method and storage medium, comprising: an image acquisition module, which acquires initial image frames and real-time acquisition of images to be detected; a system preset module, which sets the material area, operation area and correct order of production line material picking for production line personnel; a material detection module, which monitors materials in real time, detects the images to be detected through a material detection model, and obtains detection information of all materials; a hand gesture estimation module, which estimates the hand gesture of the images to be detected through a YOLO algorithm to obtain hand key point information; a material discrimination module, which obtains the detection information of all materials in the images to be detected and the hand key point information of each production line personnel, and confirms whether the production line personnel have picked up the materials based on preset discrimination standards; if the production line personnel have picked up the materials, whether the operation of the production line personnel is correct is judged according to the correct order of production line material picking, and the detection result is output.

Description

Technical Field

[0001] The present invention relates to the technical field of production line monitoring and material management, and in particular to a material taking specification detection system, method and storage medium. Background Art

[0002] With the rapid development of industrial automation and intelligent manufacturing, the product quality and production efficiency of production lines have become the focus of enterprises. In the production process of the production line, the standardized operation of the production line personnel is the key factor to ensure product quality and smooth production. At present, most of the product quality problems of the production line are caused by the production line personnel missing, taking the wrong materials, taking more materials, etc. during the production process. Therefore, the use of material monitoring systems came into being in response to the standardization of material use in the production process.

[0003] The existing material handling standard detection system has improved the production quality qualification rate of products to a certain extent. However, the existing material handling standard detection system usually only relies on the results of material detection, with little judgment information, and is prone to misjudgment and omission.

[0004] That is, the existing material handling specification detection system has the problem of low judgment accuracy. Summary of the invention

[0005] The main purpose of the present invention is to provide a material handling specification detection system, method and storage medium, aiming to solve the technical problem of low judgment accuracy of the existing material handling specification detection system.

[0006] To achieve the above object, the present invention provides a material picking specification detection system, which includes:

[0007] An image acquisition module is used to acquire an initial image frame and to acquire an image to be detected in real time;

[0008] The system preset module is used to set the material area and operation area for production line personnel to work according to the initial image frame, and to set the correct order for picking up materials on the production line;

[0009] The material detection module is used to monitor materials in real time. The pre-trained material detection model is used to detect the image to be detected to obtain the detection information of all materials in the image to be detected. The detection information includes location information and category information.

[0010] The hand gesture estimation module is used to estimate the hand gesture of the image to be detected by using the YOLO algorithm to obtain the key point information of the hand of each production line personnel in the image to be detected;

[0011] The material identification module is used to obtain the detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel, and confirm whether the production line personnel have taken the materials based on the preset identification criteria; if the production line personnel have taken the materials, it is judged whether the operation of the production line personnel is correct based on the correct order of taking the materials on the production line, and the detection results are output, which are correct or abnormal.

[0012] Optionally, the system also includes a control module for controlling the detection state, including at least a start detection button, a pause detection button and a reset button.

[0013] Optionally, the system also includes an audio-visual alarm module for indicating the current detection status. When the current detection status is normal, a first indicator light is displayed; when the current detection status is paused, a second indicator light is displayed; when the current detection status is abnormal, a third indicator light is displayed and a sound prompt is given; the three indicator lights have different colors.

[0014] Optionally, the material detection model is obtained by the following steps:

[0015] Use the pre-training data set to pre-train the target detection model to obtain the weight of the target detection model;

[0016] Freeze the parameters of the target detection model except the detection head, use the production line material image set to fine-tune the target detection model, and obtain the weight of the fine-tuned target detection model;

[0017] The knowledge distillation method is used to migrate and lightweight the material detection capabilities of the fine-tuned target detection large model to obtain a lightweight student model, which is used as the material detection model.

[0018] Optionally, the pre-training dataset is obtained by collecting various types of target detection public datasets and performing a first pre-processing;

[0019] The production line material image set is obtained by collecting various material images required for the operation of the detected workstation and performing a second preprocessing;

[0020] The various material images include images of the same material under different backgrounds, different angles, different placement states and different light intensities, and the various material images are manually annotated using annotation software.

[0021] Optionally, pre-training the large object detection model using a pre-training dataset includes at least the following steps:

[0022] Input the data image in the pre-training dataset into the embedding layer for encoding to obtain the Embedding input vector corresponding to the dataset image;

[0023] Input the Embedding input vector to the position encoding layer and use absolute position encoding to add position information to the dataset image;

[0024] The embedding input vector with position information is input into the Transformer feature extraction layer for feature extraction and enhancement to obtain feature maps of different scales;

[0025] The feature maps of different scales are input into the feature pyramid network respectively, and multi-scale fusion is performed through the top-down path and the bottom-up path to obtain the fused multi-scale feature map;

[0026] Input the multi-scale feature map into the bounding box predictor and category classifier in the detection head to obtain the predicted detection information;

[0027] Calculate the loss value of the predicted detection information and the corresponding label in the pre-training dataset, update the weights through back propagation, and save the trained model.

[0028] Optionally, obtaining the detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel, and confirming whether the production line personnel have taken the materials based on a preset judgment standard, at least includes the following steps:

[0029] Select key points for judgment from the key point information of the hand of a production line worker for calculation, and judge whether the hand of the production line worker is in an open state or a closed state based on the preset hand closed state threshold and hand open state threshold;

[0030] When the hand of a production line worker changes from an open state to a closed state, and the key point of the hand is within the material area, the first detection timestamp is recorded;

[0031] Based on the preset distance threshold for picking up materials, determine whether the distance between the hand of the production line personnel and the center point of each material frame is less than the preset distance threshold for picking up materials; if so, obtain the number of image frames in which the distance between the hand of the production line personnel and the center point of each material frame is less than or equal to the preset distance threshold for picking up materials;

[0032] When the hands of the production line personnel are open and in the operating area, a second detection timestamp is recorded;

[0033] Obtaining the total number of image frames between the first detection timestamp and the second detection timestamp, and calculating the proportion of the number of image frames in the total number of image frames;

[0034] Determine whether the proportion is greater than or equal to the preset proportion. If so, it is determined that the production line personnel have taken the material and the category information of the material is recorded. If not, it is determined to be a false detection.

[0035] Optionally, it also includes: an expansion module for expanding the material handling specification detection system when new materials need to be tested.

[0036] Corresponding to the material picking specification detection system, the present invention provides a material picking specification detection method, which uses any of the above material picking specification detection systems for detection, and at least includes the following steps:

[0037] Collect the initial image frame, and set the material area and operation area for production line personnel to work according to the initial image frame, as well as set the correct order for taking materials from the production line;

[0038] Collect the images to be inspected in real time, monitor the materials in real time, detect the images to be inspected through the pre-trained material detection model, and obtain the detection information of all materials in the images to be inspected, including location information and category information;

[0039] The hand gestures of the image to be detected are estimated by the YOLO algorithm to obtain the key point information of the hands of each production line personnel in the image to be detected;

[0040] The detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel are obtained, and based on the preset judgment criteria, whether the production line personnel have taken the materials is confirmed; if the production line personnel have taken the materials, whether the operation of the production line personnel is correct is judged according to the correct order of taking the materials on the production line, and the detection results are output, which are correct or abnormal.

[0041] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer-readable storage medium, on which a material picking specification detection program is stored. When the material picking specification detection program is executed by a processor, the steps of the material picking specification detection method described above are implemented.

[0042] The beneficial effects of the present invention are:

[0043] (1) Compared with the prior art, the present invention improves the accuracy of material picking by integrating visual detection and hand key points as discrimination information, and can achieve comprehensive monitoring and evaluation of the material picking behavior of production line personnel, ensuring that materials are picked up correctly and efficiently, thereby significantly improving the automation level and operating efficiency of the production line;

[0044] (2) Compared with the prior art, the present invention, through the control module, allows the user to conveniently start the system by pressing the start detection button, pause the detection by pressing the pause detection button, and reset the system state by pressing the reset button, thereby improving the usability and flexibility of the system and enabling the user to accurately control the system according to actual needs;

[0045] (3) Compared with the prior art, the present invention uses an audible and visual alarm module to enable the system to intuitively display the current detection status. Through indicator lights of different colors and sound prompts, users can quickly understand the operating status of the system and respond to possible abnormal situations in a timely manner, thereby enhancing the reliability of the system and user experience;

[0046] (4) Compared with the prior art, the present invention can make full use of existing data and model resources to quickly build a lightweight model suitable for production line material detection, which not only improves the detection accuracy and speed of the model, but also reduces the overall cost of the system;

[0047] (6) Compared with the prior art, the present invention provides rich and high-quality data support for the training of material detection models by collecting and processing various types of target detection public data sets and production line material image sets, ensuring the generalization ability and accuracy of the model, enabling it to cope with various complex production line environments;

[0048] (7) Compared with the prior art, the present invention can fully utilize the advantages of advanced technologies such as Transformer to improve the feature extraction and detection capabilities of the model through the steps of encoding, position encoding, feature extraction and enhancement, multi-scale fusion, and output of the detection head; at the same time, through the top-down and bottom-up multi-scale fusion strategies, the model can more accurately identify materials of different scales;

[0049] (8) Compared with the prior art, the system of the present invention can accurately identify the material picking behavior of production line personnel by judging the hand state, recording the timestamp, calculating the distance and proportion, etc.; it not only improves the detection accuracy and robustness of the system, but also provides strong support for subsequent anomaly detection and behavior analysis;

[0050] (9) Compared with the prior art, the present invention uses an expansion module to enable the system to easily respond to the detection needs of new materials; users can include new materials in the monitoring scope of the system through simple configuration and training; this design improves the scalability and flexibility of the system, enabling it to adapt to ever-changing production needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0052] Figure 1 It is a framework diagram of an embodiment of a material taking standard detection system of the present invention;

[0053] Figure 2 This is a simplified flow chart of an embodiment of a material handling standard detection method of the present invention. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical scheme and advantages of the embodiments of the present invention clearer, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0055] like Figure 1 As shown, a material taking specification detection system of the present invention comprises:

[0056] The image acquisition module 10 is used to acquire the initial image frame and to acquire the image to be detected in real time;

[0057] The system preset module 20 is used to set the material area and operation area for the production line personnel to work according to the initial image frame, and to set the correct order for taking the production line materials based on the Standard Operating Procedure (SOP);

[0058] The material detection module 30 is used to monitor the material in real time, detect the image to be detected by using the pre-trained material detection model, and obtain the detection information of all materials in the image to be detected, and the detection information includes position information and category information;

[0059] The hand gesture estimation module 40 is used to estimate the hand gesture of the image to be detected by using the YOLO algorithm to obtain the key point information of the hand of each production line personnel in the image to be detected;

[0060] The material identification module 50 is used to obtain the detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel, and confirm whether the production line personnel have taken the materials based on the preset identification criteria; if the production line personnel have taken the materials, it is judged whether the operation of the production line personnel is correct based on the correct order of taking the materials on the production line, and the detection result is output, which is correct or abnormal.

[0061] In this embodiment, the initial image frame is an image frame captured when the system is started, and the user sets the material area and operation area for the production line personnel to work by selecting the material area and operation area on the initial image frame.

[0062] Preferably, the position information includes the coordinates of the upper left corner and the lower right corner of the material detection frame.

[0063] In this embodiment, the YOLO algorithm is specifically the YOLOv8-pose algorithm, and the hand key point information is the coordinates of 21 skeletal joints of the hand.

[0064] It can be understood that since the image acquisition module 10 described in the present invention acquires the image to be detected in real time (that is, the acquired image to be detected is a continuous video frame), the material detection module 30, the gesture posture estimation module 40 and the material discrimination module 50 will make judgments frame by frame until the system is shut down.

[0065] The present invention improves the accuracy of material picking judgment by integrating visual detection and hand key points as discrimination information, and can realize comprehensive monitoring and evaluation of the material picking behavior of production line personnel, ensuring that materials are picked up correctly and efficiently, thereby significantly improving the automation level and operating efficiency of the production line.

[0066] In this embodiment, the system further includes a control module for controlling the detection state, which at least includes a start detection button, a pause detection button and a reset button.

[0067] The present invention uses a control module to allow users to conveniently start the system through a start detection button, pause detection through a pause detection button, and reset the system state through a reset button, thereby improving the availability and flexibility of the system and enabling users to accurately control the system according to actual needs.

[0068] In this embodiment, the system also includes an audio-visual alarm module for prompting the current detection status. When the current detection status is normal, a first indicator light is displayed; when the current detection status is paused, a second indicator light is displayed; when the current detection status is abnormal, a third indicator light is displayed and a sound prompt is given; the three indicator lights have different colors.

[0069] Preferably, the first indicator light is green, the second indicator light is yellow, the third indicator light is red, and the sound prompt is given by a buzzer. When the system of the present invention is turned on, the current detection state is normal and a green light is displayed.

[0070] The present invention uses the sound and light alarm module to enable the system to intuitively display the current detection status. Through indicator lights of different colors and sound prompts, users can quickly understand the operating status of the system and respond to possible abnormal situations in a timely manner, thereby enhancing the reliability of the system and user experience.

[0071] In this embodiment, the material detection model is obtained by the following steps:

[0072] Use the pre-training dataset to pre-train the target detection model (do not freeze any weights of the network at this time) to obtain the weights of the target detection model;

[0073] Freeze the parameters of the target detection model except the detection head, use the production line material image set to fine-tune the target detection model, and obtain the weight of the fine-tuned target detection model;

[0074] The knowledge distillation method is used to migrate and lightweight the material detection capabilities of the fine-tuned target detection large model to obtain a lightweight student model, which is used as the material detection model.

[0075] In this embodiment, the target detection large model is fine-tuned using the production line material image set to obtain the fine-tuned target detection large model weight, which specifically includes the following steps:

[0076] The model weights obtained by back-propagation updating weights are used as pre-training weights for fine-tuning. The dataset used at this time is the pre-processed production line material image set.

[0077] Freeze the parameters except the detection head in the network structure of the target detection large model. The frozen parameters do not participate in the back propagation update. At this time, the training data set is the production line material image set, and the training process is the same as the pre-training process.

[0078] Calculate the loss value of the material detection information predicted by the bounding box predictor and category classifier in the detection head and the corresponding label in the material image set, update the weight parameters of the detection head through back propagation, and save the fine-tuned model.

[0079] It can be understood that the corresponding labels in the pre-training data set are the labels manually annotated in the pre-training data set, and the corresponding labels in the material image set are the labels manually annotated in the material image set.

[0080] The present invention can make full use of existing data and model resources to quickly build a lightweight model suitable for production line material detection, which not only improves the detection accuracy and speed of the model, but also reduces the overall cost of the system.

[0081] Knowledge distillation is a technique for transferring the knowledge learned by one model (teacher model) to another model (student model). Usually, the teacher model is a complex and accurate model with stronger feature extraction capabilities, and the student model is a lightweight version. The teacher model can guide the student model to perform regression and prediction, so that the student model can maintain good performance while reducing model complexity and has better generalization capabilities. In this embodiment, the teacher model is a trained large target detection model, and the student model is a simplified version consisting of an encoder, FPN, and a detection head. The knowledge distillation method is used to transfer and lightweight the material detection capabilities of the fine-tuned large target detection model to obtain a lightweight student model, which specifically includes the following steps:

[0082] The production line material image set is sent to the teacher model network and the student model network, where the teacher model weight is fixed, and the teacher model output and the student model output are obtained.

[0083] Calculate the classification loss between the student model and the true label and the classification loss between the student model and the teacher model output. The loss function is:

[0084]

[0085] Calculate the confidence loss between the student model and the true label and the confidence loss between the student model and the teacher model output. The loss function is:

[0086]

[0087] Calculate the regression loss between the student model and the true label and the regression loss between the student model and the teacher model output. The loss function is:

[0088]

[0089] Among them, f cl is the classification cross entropy loss function, specifically the binary cross entropy loss function; Represents the loss of measuring the student model and the true label separately, represents the classification loss between the output of the student model and the teacher model, The representation combines the difference between the student model and the true label and the difference between the student model and the teacher model. D is a weighting factor used to adjust the weight between the teacher model output and the student model output. obj is the confidence loss function, which measures whether there is a prediction error of the target; Represents the loss of measuring the student model and the true label separately, represents the classification loss between the output of the student model and the teacher model, The representation combines the difference between the student model and the true label and the difference between the student model and the teacher model. bb is the regression loss function, specifically the intersection-over-union loss between the predicted box and the true box; Represents the loss of measuring the student model and the true label separately, represents the classification loss between the output of the student model and the teacher model, The representation combines the difference between the student model and the true label and the difference between the student model and the teacher model. are the confidence, class probability and coordinates of the detection box of the student model. is the true label, true probability and true coordinate of the material, are the confidence, class probability and coordinates of the detection box of the teacher model.

[0090] Through the chain rule, forward propagation calculates the gradient of the loss function with respect to each neuron;

[0091] Using the Adam optimizer, the model loss value is reduced based on the calculated gradient backpropagation, so that the output of the student model is closer to the output of the teacher model;

[0092] After training is completed, the student model weights are obtained for subsequent material detection reasoning and deployment.

[0093] In this embodiment, the model deployment step specifically includes the following steps: converting the student model using the ONNX open ML model standard tool to obtain the ONNX format model weights, and performing reasoning verification of the ONNX model;

[0094] Use TensorRT's free conversion tool to convert the ONNX model into an engine file supported by TensorRT

[0095] Install and configure the TensorRT model acceleration tool on the Jetson Xavier embedded system;

[0096] Use the TensorRT API to load and utilize the TensorRT-optimized material detection model on a Jetson Xavier embedded system.

[0097] In this embodiment, the pre-training data set is obtained by collecting various types of target detection public data sets (such as mainstream target detection data sets such as COCO, Object365, Caltech, etc.) and performing a first preprocessing; preferably, the first preprocessing is performed using data enhancement methods such as Moscia, random horizontal flipping, random blurring, and random cropping.

[0098] The production line material image set is obtained by collecting various material images required for the operation of the detected workstation and performing a second preprocessing; preferably, the second preprocessing at least includes: data cleaning, removing duplicate, blurred or low-quality images, ensuring that there are no erroneous or inconsistent labels in the production line material image set; increasing the amount and diversity of data through data enhancement techniques such as rotation, flipping, scaling, and brightness adjustment to help model generalization; dividing the image set into a training set, a validation set, and a test set in a ratio of 7:2:1 to obtain a production line material image set.

[0099] In this embodiment, the various material images include images of the same material under different backgrounds, different angles, different placement states and different light intensities, and the various material images are manually annotated using annotation software.

[0100] Preferably, different placement states include different placement positions, different directions and different stacking states, and about 500 images are collected for each material.

[0101] In this embodiment, the labeling software specifically adopts labelImg.

[0102] By collecting and processing various types of target detection public data sets and production line material image sets, the present invention provides rich and high-quality data support for the training of material detection models, ensuring the generalization ability and accuracy of the model, enabling it to cope with various complex production line environments.

[0103] In this embodiment, the target detection model includes an embedding layer, a position encoding layer, a Transformer feature extraction layer, and a detection head. Pre-training the target detection model using a pre-training dataset includes at least the following steps:

[0104] Input the data image in the pre-training dataset into the embedding layer for encoding to obtain the Embedding input vector corresponding to the dataset image;

[0105] Input the embedding input vector to the position encoding layer, and use absolute position encoding to add position information to the dataset image; that is, directly use the pixel coordinates (x, y) as the position information;

[0106] The embedding input vector with position information is input into the Transformer feature extraction layer for feature extraction and enhancement to obtain feature maps of different scales;

[0107] The feature maps of different scales are input into the feature pyramid network respectively, and multi-scale fusion is performed through the top-down path and the bottom-up path to obtain the fused multi-scale feature map;

[0108] Input the multi-scale feature map into the bounding box predictor and category classifier in the detection head to predict the detection information;

[0109] Calculate the loss value of the predicted detection information and the corresponding label in the pre-training dataset, update the weights through back propagation, and save the trained model.

[0110] It can be understood that the feature maps of different scales above represent the operation of feature extraction, which is the process of extracting features from input data and gradually extracting high-level features from low-level features. Among them, low-level feature maps usually retain more original image details, such as edges, lines, and textures. These features are very important for identifying local structures in images. As the depth of the network increases, it can capture a wider range of information, that is, high-level features, such as the partial or overall shape of an object. These features help to understand the global information in the image.

[0111] And, the above-mentioned fused multi-scale feature map refers to a new feature map formed by fusing feature maps of different scales together through a feature pyramid network. It combines low-level detailed information and high-level abstract information. Specifically:

[0112] Spatial details: By fusing low-level feature maps, multi-scale feature maps retain rich spatial detail information, which is especially important for small object detection.

[0113] Semantic information: By fusing high-level feature maps, multi-scale feature maps enhance the ability to understand complex scenes and can better identify and classify large objects or elements in scenes.

[0114] Scale invariance: The multi-scale feature map improves the model's ability to detect objects of different sizes by integrating information at different scales, making the model more robust.

[0115] In this embodiment, the embedding input vector with position information is input into the Transformer feature extraction layer for feature extraction and enhancement to obtain feature maps of different scales; specifically, each layer of encoder captures the dependencies between features through the self-attention mechanism and feedforward neural network, and gradually maps the features to different subspaces. This process enables the model to effectively capture long-distance dependencies within and between images, thereby improving the performance of object detection.

[0116] In this embodiment, the upsampled high-level feature map is directly added to the underlying feature map of the corresponding size through upsampling and feature fusion by the feature pyramid network (FPN) to generate feature maps of multiple scales. These feature maps can be used to detect objects of different sizes, thereby enhancing the detection capability of the model.

[0117] In this embodiment, the bounding box predictor generates a bounding box offset for each anchor point through a convolutional layer to adjust the position and size of the anchor point and generate the final predicted bounding box. The category classifier generates a probability value of each anchor point belonging to each category through a convolutional layer, and the output dimension is K+1, where K is the number of categories and the additional category represents the background.

[0118] The present invention can fully utilize the advantages of advanced technologies such as Transformer to improve the feature extraction and detection capabilities of the model through the steps of encoding, position encoding, feature extraction and enhancement, multi-scale fusion and output of the detection head; at the same time, through top-down and bottom-up multi-scale fusion strategies, the model can more accurately identify materials of different scales.

[0119] In this embodiment, the detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel are obtained, and based on the preset judgment criteria, whether the production line personnel have taken the materials is confirmed, which at least includes the following steps:

[0120] Select key points for judgment from the key point information of the hand of a production line worker for calculation, and judge whether the hand of the production line worker is in an open state or a closed state based on the preset hand closed state threshold and hand open state threshold;

[0121] When the hand of a production line worker changes from an open state to a closed state, and the key point of the hand is within the material area, the first detection timestamp is recorded;

[0122] Based on the preset distance threshold for picking up materials, determine whether the distance between the hand of the production line personnel and the center point of each material frame is less than the preset distance threshold for picking up materials; if so, obtain the number of image frames in which the distance between the hand of the production line personnel and the center point of each material frame is less than or equal to the preset distance threshold for picking up materials;

[0123] When the hands of the production line personnel are open and in the operating area, a second detection timestamp is recorded;

[0124] Obtaining the total number of image frames between the first detection timestamp and the second detection timestamp, and calculating the proportion of the number of image frames in the total number of image frames;

[0125] Determine whether the proportion is greater than or equal to the preset proportion. If so, it is determined that the production line personnel have taken the material and the category information of the material is recorded. If not, it is determined to be a false detection.

[0126] Preferably, the preset proportion is 60%, and the key points to be judged are specifically the fingertip key points of the five fingers and the wrist base point.

[0127] In this embodiment, key points are selected from the key point information of the hand of a production line employee for calculation, specifically, the Euclidean distances between the key points of the fingertips of the five fingers and the wrist base point are calculated respectively, and the average is taken.

[0128] The Euclidean distance calculation formula is as follows:

[0129]

[0130] Among them, (x1, y1) respectively represent the x-coordinate and y-coordinate of the key point of the fingertip of a certain finger.

[0131] (x2, y2) represent the x-coordinate and y-coordinate of the wrist base point respectively.

[0132] Furthermore, based on the preset hand closed state threshold (preferably 0.3) and the hand open state threshold (preferably 0.7), it is determined whether the hands of the production line personnel are in an open state or a closed state; if the above mean is greater than 0.7, the hands of the production line personnel are in an open state; if the above mean is less than 0.3, the hands of the production line personnel are in a closed state.

[0133] In this embodiment, it is determined whether the position distance between the hand of the production line personnel and the center point of each material frame is less than the preset distance threshold for picking up the material. Specifically, the Euclidean distance between the key points of the fingertips of the five fingers and the center point of the material detection frame, and the Euclidean distance between the wrist base point and the center point of the material detection frame are calculated respectively, and the average of all calculation results is taken. The preset distance threshold is preferably 0.7. If the mean is less than 0.7, it is determined that the hand of the production line personnel is together with the material, and the number of image frames in which the position distance between the hand of the production line personnel and the center point of each material frame is less than or equal to the preset distance threshold for picking up the material is further obtained.

[0134] The Euclidean distance calculation formula is as follows:

[0135]

[0136] Among them, (x3, y3) respectively represent the x-coordinate and y-coordinate of the key point of the fingertip of a certain finger.

[0137] (x4, y4) represent the x-coordinate and y-coordinate of the center point of the material box respectively.

[0138] By judging the hand state, recording the timestamp, calculating the distance and proportion, etc., the system of the present invention can accurately identify the material picking behavior of the production line personnel; it not only improves the detection accuracy and robustness of the system, but also provides strong support for subsequent anomaly detection and behavior analysis.

[0139] In this embodiment, it also includes: an expansion module, which is used to expand the material handling specification detection system when new materials need to be detected.

[0140] Preferably, when new materials need to be tested, the material handling specification testing system is expanded, which can be achieved by the following steps:

[0141] Capture images of new materials and create new production line material image sets;

[0142] Use the new production line material image set to fine-tune the weights of the target detection model. During fine-tuning, freeze the parameters except the detection head part in the weights, and do not update the parameters. Only unfreeze and update the parameters of the detection head part. After fine-tuning, get the new target detection model weights. At this time, the basic model already has the ability to detect new materials.

[0143] The weights of the new large target detection model are migrated and lightweight using the knowledge distillation method;

[0144] Redeploy the lightweight weights to the edge computing development board to complete the expansion of the detection system.

[0145] The present invention enables the system to easily cope with the detection requirements of new materials through expansion modules; users can include new materials into the monitoring scope of the system through simple configuration and training; this design improves the scalability and flexibility of the system, enabling it to adapt to ever-changing production needs.

[0146] Furthermore, the system of the present invention also includes a display module for displaying real-time images and detection results during the detection process.

[0147] like Figure 2 As shown, the present invention also provides a material picking specification detection method, which at least includes the following steps:

[0148] Collect the initial image frame, and set the material area and operation area for production line personnel to work according to the initial image frame, as well as set the correct order for taking materials from the production line;

[0149] Collect the images to be inspected in real time, monitor the materials in real time, detect the images to be inspected through the pre-trained material detection model, and obtain the detection information of all materials in the images to be inspected, including location information and category information;

[0150] The hand gestures of the image to be detected are estimated by the YOLO algorithm to obtain the key point information of the hands of each production line personnel in the image to be detected;

[0151] The detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel are obtained, and based on the preset judgment criteria, whether the production line personnel have taken the materials is confirmed; if the production line personnel have taken the materials, whether the operation of the production line personnel is correct is judged according to the correct order of taking the materials on the production line, and the detection results are output, which are correct or abnormal.

[0152] The embodiment of the present invention further provides a computer-readable storage medium, which may be a computer-readable storage medium included in the memory in the above embodiment; or a computer-readable storage medium that exists independently and is not installed in a device. The computer-readable storage medium stores at least one instruction, which is loaded and executed by a processor to implement Figure 2 The material picking standard detection method shown. The computer readable storage medium can be a read-only memory, a disk or an optical disk, etc.

[0153] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same or similar parts between the various embodiments can be referred to each other. For the system embodiment and the storage medium embodiment, since they are basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0154] Furthermore, in this document, the terms "comprises," "comprising," or any other variation thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements, but also other elements not explicitly listed, or elements inherent to such process, method, article, or device. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the element.

[0155] The above description shows and describes the preferred embodiments of the present invention. It should be understood that the present invention is not limited to the form disclosed herein, and should not be regarded as excluding other embodiments, but can be used in various other combinations, modifications and environments, and can be modified within the scope of the invention, through the above teachings or the technology or knowledge of the relevant field. The changes and modifications made by those skilled in the art shall not depart from the spirit and scope of the present invention, and shall be within the scope of protection of the claims attached to the present invention.

Claims

1. A material handling specification detection system, characterized in that: include: An image acquisition module is used to acquire an initial image frame and to acquire an image to be detected in real time; The system preset module is used to set the material area and operation area for production line personnel to work according to the initial image frame, and to set the correct order for picking up materials on the production line; The material detection module is used to monitor materials in real time. The pre-trained material detection model is used to detect the image to be detected to obtain the detection information of all materials in the image to be detected. The detection information includes location information and category information. The hand gesture estimation module is used to estimate the hand gesture of the image to be detected by using the YOLO algorithm to obtain the key point information of the hand of each production line personnel in the image to be detected; The material identification module is used to obtain the detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel, and confirm whether the production line personnel have taken the materials based on the preset identification criteria; if the production line personnel have taken the materials, it is judged whether the operation of the production line personnel is correct based on the correct order of taking the materials on the production line, and the detection results are output, which are correct or abnormal.

2. The material handling specification detection system according to claim 1 is characterized in that: It also includes a control module for controlling the detection state, which at least includes a start detection button, a pause detection button and a reset button.

3. The material handling specification detection system according to claim 1 is characterized in that: It also includes an audio-visual alarm module for prompting the current detection status. When the current detection status is normal, the first indicator light is displayed; when the current detection status is paused, the second indicator light is displayed; when the current detection status is abnormal, the third indicator light is displayed and a sound prompt is given; the three indicator lights have different colors.

4. The material handling specification detection system according to claim 1 is characterized in that: The material detection model is obtained through the following steps: Use the pre-training data set to pre-train the target detection model to obtain the weight of the target detection model; Freeze the parameters of the target detection model except the detection head, use the production line material image set to fine-tune the target detection model, and obtain the weight of the fine-tuned target detection model; The knowledge distillation method is used to migrate and lightweight the material detection capabilities of the fine-tuned target detection large model to obtain a lightweight student model, which is used as the material detection model.

5. The material handling specification detection system according to claim 4 is characterized in that: The pre-training dataset is obtained by collecting various types of target detection public datasets and performing the first pre-processing; The production line material image set is obtained by collecting various material images required for the operation of the detected workstation and performing a second preprocessing; The various material images include images of the same material under different backgrounds, different angles, different placement states and different light intensities, and the various material images are manually annotated using annotation software.

6. The material handling specification detection system according to claim 4 is characterized in that: Pre-training the target detection model using the pre-training dataset includes at least the following steps: Input the data image in the pre-training dataset into the embedding layer for encoding to obtain the Embedding input vector corresponding to the dataset image; Input the Embedding input vector to the position encoding layer and use absolute position encoding to add position information to the dataset image; The embedding input vector with position information is input into the Transformer feature extraction layer for feature extraction and enhancement to obtain feature maps of different scales; The feature maps of different scales are input into the feature pyramid network respectively, and multi-scale fusion is performed through the top-down path and the bottom-up path to obtain the fused multi-scale feature map; Input the multi-scale feature map into the bounding box predictor and category classifier in the detection head to obtain the predicted detection information; Calculate the loss value of the predicted detection information and the corresponding label in the pre-training dataset, update the weights through back propagation, and save the trained model.

7. The material handling specification detection system according to claim 1 is characterized in that: Obtaining the detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel, and confirming whether the production line personnel have taken the materials based on the preset judgment criteria, at least includes the following steps: Select key points for judgment from the key point information of the hand of a production line worker for calculation, and judge whether the hand of the production line worker is in an open state or a closed state based on the preset hand closed state threshold and hand open state threshold; When the hand of a production line worker changes from an open state to a closed state, and the key point of the hand is within the material area, the first detection timestamp is recorded; Based on the preset distance threshold for picking up materials, determine whether the distance between the hand of the production line personnel and the center point of each material frame is less than the preset distance threshold for picking up materials; if so, obtain the number of image frames in which the distance between the hand of the production line personnel and the center point of each material frame is less than or equal to the preset distance threshold for picking up materials; When the hands of the production line personnel are open and in the operating area, a second detection timestamp is recorded; Obtaining the total number of image frames between the first detection timestamp and the second detection timestamp, and calculating the proportion of the number of image frames in the total number of image frames; Determine whether the proportion is greater than or equal to the preset proportion. If so, it is determined that the production line personnel have taken the material and the category information of the material is recorded. If not, it is determined to be a false detection.

8. The material handling specification detection system according to claim 1 is characterized in that: Also includes: The expansion module is used to expand the material handling specification detection system when new materials need to be tested.

9. A material handling standard detection method, characterized in that: The material handling specification detection system according to any one of claims 1 to 8 is used for detection, and at least the following steps are included: Collect the initial image frame, and set the material area and operation area for production line personnel to work according to the initial image frame, as well as set the correct order for taking materials from the production line; Collect the images to be inspected in real time, monitor the materials in real time, detect the images to be inspected through the pre-trained material detection model, and obtain the detection information of all materials in the images to be inspected, including location information and category information; The hand gestures of the image to be detected are estimated by the YOLO algorithm to obtain the key point information of the hands of each production line personnel in the image to be detected; The detection information of all materials in the image to be detected and the key point information of the hands of each production line personnel are obtained, and based on the preset judgment criteria, whether the production line personnel have taken the materials is confirmed; if the production line personnel have taken the materials, whether the operation of the production line personnel is correct is judged according to the correct order of taking the materials on the production line, and the detection results are output, which are correct or abnormal.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a material handling specification detection program, and when the material handling specification detection program is executed by the processor, the steps of the material handling specification detection method according to claim 9 are implemented.

Images