Manufacturing scene action process detection method based on large model

Abstract

The invention discloses a manufacturing scene action process detection method based on a large model, and the method comprises the following steps: when a mechanical arm gripper works normally, a camera obtains multiple frames of positions of the mechanical arm gripper and a clamped workpiece, forms a coordinate data set, builds a YOLO model, and inputs the coordinate data set for pre-training; the method comprises the following steps: respectively acquiring an image of a mechanical arm gripper and an image of a clamped workpiece in operation through a camera, inputting the acquired images into a trained YOLO model to obtain position coordinates of the mechanical arm gripper and the clamped workpiece, selecting a detection frame of the mechanical arm gripper and the clamped workpiece according to a confidence degree pre-trained by the trained YOLO model, and detecting the mechanical arm gripper and the clamped workpiece according to the detection frame. The method comprises the steps that a mechanical arm gripper is used for clamping a workpiece, position coordinates of the two detection frames are output, an IoU value is calculated according to the position coordinates of the mechanical arm gripper and the two detection frames for clamping the workpiece, the working state of the mechanical arm gripper is judged according to the IoU value, and a teaching state result prompt is output on a display screen. And the method is suitable for classroom display.

Description

Technical Field

[0001] This invention belongs to the field of teaching and training scenario technology, and specifically relates to a method for detecting the action flow of a manufacturing scenario based on a large model. Background Technology

[0002] Currently, vocational education and skills training are developing rapidly, with different training content tailored to various specific scenarios, each possessing distinct characteristics. A common problem across different vocational education scenarios is that in traditional or existing practical training environments, teachers struggle to observe each student's operational details in real-time, clearly, and comprehensively. Crucial information, such as whether students' actions are standardized and whether their procedures are correct, is often obscured by physical distance, limited perspective, large numbers of students, or equipment obstructions. This makes it difficult for teachers to accurately judge whether students' actions conform to operational standards, leading to low efficiency in skills training. Specifically, in intelligent manufacturing scenarios, due to the real-time nature of training, teachers also cannot observe each student's actions in real time.

[0003] Therefore, it is necessary to provide a method for detecting whether students' operational actions and procedures are correct and standardized during the training process. Summary of the Invention

[0004] Purpose of the invention: To address the problems existing in the above-mentioned background technology, the present invention provides a method for detecting the action flow of a manufacturing scene based on a large model. Technical solution

[0005] A method for detecting the action flow in a manufacturing scene based on a large model includes the following steps: S1: When the robotic arm gripper is working normally, the camera captures multiple frames of the position of the robotic arm gripper and the workpiece being held. The coordinates of the robotic arm gripper and the workpiece being held are then manually acquired to form a coordinate dataset. S2: Build a YOLO model and pre-train it using the input coordinate dataset; S3: Capture images of the robotic arm gripper in operation and images of the workpiece being held using cameras; S4: Input the acquired images into the trained YOLO model to obtain the position coordinates of the robotic arm gripper and the workpiece being held; S5: Select the detection boxes for the robotic arm gripper and the workpiece based on the confidence scores of the pre-trained YOLO model, and output the position coordinates of the two detection boxes as (X1, Y1, X2, Y2) and (A1, B1, A2, B2). Where (X1, Y1) are the coordinates of the upper left corner of the robotic arm gripper detection frame, (A1, B1) are the coordinates of the upper left corner of the workpiece clamping detection frame, (X2, Y2) are the coordinates of the lower right corner of the robotic arm gripper detection frame, and (A2, B2) are the coordinates of the lower right corner of the workpiece clamping detection frame. S6: Calculate the IoU value based on the position coordinates of the two detection frames of the robotic arm gripper and the workpiece: ; in: The area of ​​the intersection of the two detection boxes; Let be the area of ​​the union of the two detection boxes; S7: Determine the working status of the robotic arm gripper based on the IoU value and output the teaching status result prompt on the display screen.

[0006] The accuracy of the entire motion process is judged by calculating the IoU value. Compared with traditional detection methods, calculating the IoU value can not only determine whether the running status and running path of the robotic arm gripper are accurate, but also whether the position of the workpiece is correct.

[0007] Preferably, in step S1, the camera needs to be calibrated before capturing the position of the robotic arm gripper and the workpiece being held, to obtain accurate focal length and distortion parameters, ensuring the clarity of the images acquired by the camera.

[0008] Preferably, the method for the camera to acquire images of the robotic arm gripper and the workpiece in S2 is as follows: The camera captures images to generate MIPI images, which are then used to determine whether a robotic arm gripper has been detected. If detected, proceed to the next step; otherwise, the camera tracks frame by frame until the robotic arm gripper is detected, then proceed to the next step. The camera detects the position of the workpiece being held, ensuring that the camera can simultaneously capture the robotic arm gripper and the workpiece being held.

[0009] Preferably, in S4, the YOLO segmentation algorithm is used to process and output the position coordinates of the robotic arm gripper and the workpiece, as well as the position coordinates of the two detection frames of the robotic arm gripper and the workpiece, to facilitate the subsequent calculation of the IoU value.

[0010] Preferably, the coordinates in S1-S6 are located in a coordinate system with the same origin to ensure the correctness of the IoU value.

[0011] Preferably, the detection box selection method in S5 further includes: Calculate the IoU value between two detection frames that meet the confidence level, and further confirm the correctness of the two detection frame selections based on the IoU value. Perform a second confirmation on the selection of the detection frames for the robotic arm gripper and the workpiece clamping to ensure the accuracy of the detection results.

[0012] Preferably, the manufacturing scene motion process detection method based on a large model further includes comparing the position coordinates of the robotic arm gripper and the position coordinates of the workpiece with the position coordinates during normal operation to determine whether the robotic arm gripper's running path is correct, and synchronously detecting the running path of the robotic arm gripper to improve accuracy.

[0013] Preferably, the confidence level of the trained YOLO model is between 0% and 100% for selecting detection boxes.

[0014] Compared with existing technical solutions, the technical solution adopted in this invention has the following advantages: 1. By using a motion flow detection system and cameras to detect the entire process of the robotic arm gripper holding the workpiece, the possibility of errors in manual monitoring is reduced; 2. By identifying the coordinates of the detection frames of the robotic arm gripper and the clamping operation, and calculating the IoU value of the two detection frames, the working status of the robotic arm gripper during the clamping process is reflected more directly, thus improving the efficiency of the entire process. 3. It solves the problem of difficulty in observation due to too many students in the classroom. Furthermore, by training a large model, the movement of the robotic arm gripper can be detected in real time, and the robotic arm gripper can be stopped in advance before an error occurs, reducing the possibility of damage to the robotic arm gripper during the demonstration. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the method steps of this invention application; Figure 2 This is a flowchart illustrating the camera data acquisition process of this invention. Detailed Implementation

[0016] This invention provides a method for detecting the action flow in a manufacturing scene based on a large model. To address the issue of inconvenience for teachers when demonstrating robotic arm gripper operation in a manufacturing scenario within a classroom, this invention provides a method for real-time monitoring of the operation scenario using a large model.

[0017] A method for detecting the action flow in a manufacturing scene based on a large model, according to Figures 1-2 As shown, it includes the following steps: S1: When the robotic arm gripper is working normally, the camera captures multiple frames of the position of the robotic arm gripper and the workpiece being held. The coordinates of the robotic arm gripper and the workpiece being held are then manually acquired to form a coordinate dataset. S2: Build a YOLO model and pre-train it using the input coordinate dataset; S3: Capture images of the robotic arm gripper in operation and images of the workpiece being held using cameras; S4: Input the acquired images into the trained YOLO model to obtain the position coordinates of the robotic arm gripper and the workpiece being held; S5: Select the detection boxes for the robotic arm gripper and the workpiece based on the confidence scores of the pre-trained YOLO model, and output the position coordinates of the two detection boxes as (X1, Y1, X2, Y2) and (A1, B1, A2, B2). Where (X1, Y1) are the coordinates of the upper left corner of the robotic arm gripper detection frame, (A1, B1) are the coordinates of the upper left corner of the workpiece clamping detection frame, (X2, Y2) are the coordinates of the lower right corner of the robotic arm gripper detection frame, and (A2, B2) are the coordinates of the lower right corner of the workpiece clamping detection frame. S6: Calculate the IoU value based on the position coordinates of the two detection frames of the robotic arm gripper and the workpiece: ; in: The area of ​​the intersection of the two detection boxes; Let be the area of ​​the union of the two detection boxes; S7: Determine the working status of the robotic arm gripper based on the IoU value and output the teaching status result prompt on the display screen.

[0018] The camera position can be adjusted according to the position of the robotic arm gripper and the workpiece being held, ensuring that the camera can capture the robotic arm gripper and the workpiece being held during the movement of the robotic arm gripper.

[0019] Before the S1 camera acquires images, the camera's focal length and distortion parameters need to be calibrated to ensure the clarity of the images acquired by the camera during the process of capturing the manufacturing scene's action flow, thereby reducing the deviation of the output detection results.

[0020] The positions of the robotic arm gripper and the workpiece in the image acquired in S1 are manually marked to obtain the coordinates of the robotic arm gripper and the workpiece when the robotic arm gripper is working normally. The coordinates of the robotic arm gripper and the workpiece are selected from a point where the camera is installed as the origin, forming a coordinate dataset for training the YOLO model.

[0021] After the camera acquires images in S1 and S3, it performs preprocessing such as denoising and enhancement to ensure that clear images can output more accurate coordinates after being input into the YOLO large model, thereby improving the accuracy of the entire detection method.

[0022] The YOLO model established in S2 of this application is the YOLO11 model. The YOLO11 segmentation algorithm is used to segment the image to identify detection boxes and output their coordinates.

[0023] The confidence level of a trained YOLO 11 model is between 0 and 1.

[0024] The method for capturing images of the robotic arm gripper and the workpiece held by the camera in S2 is as follows: The camera captures images to generate MIPI images. The MIPI images are used to determine whether the robotic arm gripper is detected. If it is detected, the next step is performed. If it is not detected, the camera tracks the robotic arm gripper frame by frame until it is detected. The camera then detects the position of the workpiece being gripped.

[0025] The detection box selection method in S5 also includes calculating the IoU value between two detection boxes that meet the confidence level, and further confirming the correctness of the selection of the two detection boxes based on the IoU value.

[0026] The selection of the detection frame for the robotic arm gripper and the detection frame for clamping the workpiece is reconfirmed to ensure the accuracy of the detection results.

[0027] The manufacturing scene motion flow detection method based on large models also includes comparing the position coordinates of the robotic arm gripper and the position coordinates of the workpiece with the position coordinates during normal operation to determine whether the robotic arm gripper's running path is correct, and simultaneously detecting the running path of the robotic arm gripper to improve accuracy.

[0028] In this embodiment, after the camera transmits the captured image to the computer device via a wireless network, the computer device implements the subsequent detection method.

[0029] In this embodiment, the images captured by the camera, the coordinate data obtained through the YOLO model, and the calculated IoU value can all be displayed in real time on a screen connected to a computer device.

[0030] In this embodiment, a frame of an image captured by a camera during the process of the robotic arm gripper holding the cylinder is selected, and the image is input into the trained YOLO model to output the position coordinates of the robotic arm gripper and the position coordinates of the cylinder.

[0031] At this time, the detection boxes appearing in the camera include: detection box 1 with a confidence level of 90%, detection box 2 with a confidence level of 60%, and detection box 3 with a confidence level of 90%.

[0032] Select box 1 according to confidence level, and calculate the IoU value between box 1 and detection box 2 and the IoU value between detection box 1 and detection box 3 respectively.

[0033] Based on the coordinate dataset of the robotic arm gripper working normally, the IoU value between the robotic arm gripper and the cylinder should be in the range of 0.2-0.6.

[0034] The areas of detection boxes 1 and 3 are 100 pixels and 120 pixels respectively, and their IoU value is 0.375, which is within the range, so detection boxes 1 and 3 are selected.

[0035] Therefore, the IoU value of the selected box and the box is used to determine the working state of the robotic arm gripper.

[0036] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A large model-based manufacturing scene action flow detection method, characterized in that, The method comprises the following steps: S1: When the mechanical arm gripper is normally working, the camera acquires multiple frames of positions of the mechanical arm gripper and the clamped workpiece, and manually acquires coordinates of the mechanical arm gripper and the clamped workpiece to form a coordinate data set; S2: A YOLO model is established, and the coordinate data set is inputted for pre-training; S3: The camera respectively acquires images of the running mechanical arm gripper and the clamped workpiece; S4: The acquired images are inputted into the trained YOLO model to obtain position coordinates of the mechanical arm gripper and the clamped workpiece; S5: According to the confidence of the pre-training of the trained YOLO model, a detection frame of the mechanical arm gripper and the clamped workpiece is selected, and position coordinates of the two detection frames are outputted, which are (X1, Y1, X2, Y2) and (A1, B1, A2, B2); Wherein (X1, Y1) is the upper left corner coordinate of the mechanical arm gripper detection frame, (A1, B1) is the upper left corner coordinate of the clamped workpiece detection frame, (X2, Y2) is the right lower corner coordinate of the mechanical arm gripper detection frame, and (A2, B2) is the right lower corner coordinate of the clamped workpiece detection frame; S6: According to the position coordinates of the two detection frames of the mechanical arm gripper and the clamped workpiece, an IoU value is calculated; wherein: is the intersection area of the two bounding boxes; is the union area of the two detection boxes; S7: According to the size of the IoU value, the working state of the mechanical arm gripper is judged, and a teaching state result prompt is outputted on the display screen.

2. The large model-based manufacturing scene action flow detection method of claim 1, wherein: Before the camera in S1 shoots the positions of the mechanical arm gripper and the clamped workpiece, the camera needs to be calibrated to obtain accurate focal length and distortion parameters.

3. The big model based manufacturing scene action flow detection method of claim 1, wherein: The method for the camera to acquire images of the mechanical arm gripper and the clamped workpiece in S2 is as follows: The camera shoots an image to generate a MIPI picture, and whether the mechanical arm gripper is detected is judged through the MIPI picture, If detected, the next step is performed, if not detected, the camera is tracked frame by frame until the mechanical arm gripper is detected, and then the next step is performed, The camera detects the position of the clamped workpiece.

4. The large model-based manufacturing scene action flow detection method of claim 1, wherein: In S4, the position coordinates of the mechanical arm gripper and the clamped workpiece and the position coordinates of the two detection frames of the mechanical arm gripper and the clamped workpiece are outputted through the YOLO segment algorithm.

5. The large model-based manufacturing scene action flow detection method of claim 1, wherein: The coordinates in S1-S6 are located in the same origin coordinate system.

6. The large model-based manufacturing scene action flow detection method of claim 1, wherein: The detection frame selection method in S5 further comprises: The IoU value between the two detection frames with the confidence is calculated, and the correctness of the selection of the two detection frames is further confirmed according to the IoU value.

7. The large model-based manufacturing scene action flow detection method of claim 1, wherein: It also includes comparing the position coordinates of the mechanical arm gripper and the clamped workpiece with the position coordinates when normally working to judge whether the running route of the mechanical arm gripper is correct.

8. The large model-based manufacturing scene action flow detection method of claim 1, wherein: The confidence of the trained YOLO model is between 0% and 100%.

Images