A method and system for monitoring the condition of an assembly line

Abstract

This application proposes a method for monitoring the status of an assembly line, including: S1, using a track-mounted robot to acquire multiple consecutive images of the assembly line; S2, using a trained target detection model to determine whether a target exists in the image; if so, continue the judgment; otherwise, end the judgment; S3, performing a difference operation on two consecutive images of the assembly line to obtain a difference image, judging the differences between consecutive assembly line images based on the difference image, and combining the judgment results of multiple consecutive assembly line images to determine the motion state of the assembly line. The above scheme proposes a method for detecting the operating status of an assembly line using a robot combined with computer vision. It utilizes a track-mounted robot to acquire assembly line images over a period of time and performs similarity judgments to determine the status of materials on the assembly line, enabling timely observation of abnormal states such as material stagnation, thus making the operation of the assembly line more intelligent.

Description

Technical Field

[0001] This application belongs to the field of intelligent manufacturing technology, specifically relating to a method and system for monitoring the status of an assembly line. Background Technology

[0002] In recent years, with the development of automation, a type of rail-mounted robot has been developed and used in the operation of factory production lines to provide fully automated, 24 / 7 status monitoring and ensure the normal operation of the production line.

[0003] In assembly line operations, material accumulation and blockages frequently occur, preventing normal flow and operation. In traditional production environments, workers often need to observe from designated locations or use traditional sensors for detection. Manual observation is time-consuming, labor-intensive, and slow to react; traditional sensor detection suffers from limited detection methods, fixed detection locations, and poor portability. Summary of the Invention

[0004] To address the aforementioned problems, the first aspect of this application proposes a method for monitoring the status of an assembly line, comprising the following steps:

[0005] S1, using a rail-mounted robot to collect multiple consecutive images of the production line;

[0006] S2, use the trained object detection model to determine whether there is an object in the image. If yes, continue the judgment; otherwise, end the judgment.

[0007] S3: Perform a difference operation on two consecutive images in the pipeline image to obtain a difference operation image. Determine the differences between consecutive pipeline images based on the difference operation image, and combine the judgment results of multiple consecutive pipeline images to determine the pipeline motion state.

[0008] The above-mentioned solution proposes a method for detecting the operating status of an assembly line using a combination of robots and computer vision. It utilizes a track-mounted robot to collect images of the assembly line over a period of time and perform similarity judgments to determine the status of materials on the assembly line. This allows for timely observation of abnormal states such as material stagnation, making the operation of the assembly line more intelligent.

[0009] Preferably, the S2 target detection model is built on the YOLO algorithm framework and incorporates a self-attention (SE) module and a CBAM module. Furthermore, it also includes: obtaining image detection boxes based on the first N images, cropping the images from the Nth image onwards based on the image detection boxes, and detecting the target using fixed detection boxes, which reduces the computational load.

[0010] Preferably, S3 further includes:

[0011] Based on the difference operation image, obtain the difference operation image matrix data;

[0012] For each difference operation image matrix data, a threshold judgment is performed. The threshold judgment includes: calculating the sum of the elements of each difference operation image matrix data and comparing it with the set threshold. If it is greater than the threshold, the motion judgment count is incremented by one; otherwise, the motion judgment count remains unchanged.

[0013] A threshold judgment is performed on multiple difference operation image matrix data. If the motion judgment count value is greater than the set motion judgment threshold, the pipeline is judged to be moving; otherwise, the pipeline is judged to be stagnant.

[0014] The above scheme compares the similarity of images through difference operations and combines the similarity judgment results of multiple collected images with the count value. It can realize the motion state determination of the production line and effectively reduce false alarms caused by robot shaking or other interference, with an accuracy rate of over 90%.

[0015] Preferably, S3 further includes:

[0016] Perform a difference operation on two consecutive difference images to obtain a quadratic difference image;

[0017] The similarity of two consecutive quadratic difference images is judged. If they are not similar, the motion judgment count is incremented by one; otherwise, the motion judgment count remains unchanged.

[0018] The above scheme uses quadratic difference operation to compare the similarity of images, which further improves the accuracy of motion state determination.

[0019] Furthermore, similarity judgment is achieved through a trained Siamese neural network; the Siamese neural network consists of two VGG16 neural networks with shared weights, ensuring that images are only judged to be similar if most pixels are the same.

[0020] Furthermore, the Siamese neural network is a three-body Siamese neural network. The input to the three-body Siamese neural network includes two sets of positive samples and one set of negative samples. The positive samples are quadratic difference images obtained when the pipeline is stationary, and the negative samples are quadratic difference images obtained when the pipeline is in motion. The three-input Siamese neural network algorithm can better distinguish negative samples and differentiate between static and moving images with a smaller training amount, reducing training costs.

[0021] Preferably, the method further includes image processing of the difference operation image and / or the quadratic difference operation image, including grayscale conversion and erosion.

[0022] The second aspect of this application proposes a production line status monitoring system, comprising:

[0023] The image acquisition module is configured to acquire multiple consecutive images of the production line using a rail-mounted robot.

[0024] The object detection module is configured to use a trained object detection model to determine whether an object exists in an image. If it does, the detection continues; otherwise, the detection ends.

[0025] The state detection module is configured to perform a difference operation on two consecutive images in the pipeline image to obtain a difference operation image. Based on the difference operation image, the module determines the differences between consecutive pipeline images and combines the judgment results of multiple consecutive pipeline images to determine the pipeline motion state.

[0026] This application proposes a method and system for monitoring the status of a production line. Utilizing a rail-mounted robot and image recognition technology, it enables the detection of material operation status on the production line, allowing for the timely detection of anomalies such as material stagnation. This method and system can be integrated into existing rail-mounted robot systems and can be adapted to monitor various production line statuses, further expanding the functionality of the rail-mounted robot and enhancing the intelligence level of the production line. Attached Figure Description

[0027] The accompanying drawings are provided to aid in further understanding of this application. The elements in the drawings are not necessarily to scale. For ease of description, only the parts relevant to the invention are shown in the drawings.

[0028] Figure 1 This is a flowchart illustrating a pipeline status monitoring method in one embodiment of this application;

[0029] Figure 2 This is a flowchart illustrating a method for monitoring the status of an electricity meter in another embodiment of this application.

[0030] Figure 3 This is an image of an assembly line taken in another embodiment of this application;

[0031] Figure 4 This is a cropped and denoised image of an electricity meter, as described in another embodiment of this application.

[0032] Figure 5 This is a test sample of the difference operation image under motion state in another embodiment of this application;

[0033] Figure 6 This is a test sample of the difference operation image in a static state in another embodiment of this application;

[0034] Figure 7 This is a flowchart illustrating a method for monitoring the status of an electricity meter in another embodiment of this application.

[0035] Figure 8 This is a schematic diagram of a three-body twin neural network structure in another embodiment of this application;

[0036] Figure 9This is a schematic diagram of the structure of a production line status monitoring system in another embodiment of this application. Detailed Implementation

[0037] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the invention.

[0038] Figure 1 This is a flowchart illustrating a pipeline status monitoring method according to an embodiment of this application. The process includes:

[0039] S1, using a rail-mounted robot to collect multiple consecutive images of the production line;

[0040] S2, use the trained object detection model to determine whether there is an object in the image. If yes, continue the judgment; otherwise, end the judgment.

[0041] In a preferred embodiment, the YOLO algorithm with added self-attention (SE) and CBAM modules is used for target detection.

[0042] S3: Perform a difference operation on two consecutive images in the pipeline image to obtain a difference operation image. Determine the differences between consecutive pipeline images based on the difference operation image, and combine the judgment results of multiple consecutive pipeline images to determine the pipeline motion state.

[0043] Figure 2 This is a flowchart illustrating a method for monitoring the status of an electricity meter in a production line, as described in another embodiment of this application. The process includes:

[0044] 1. Control the robot to reach the designated position, and adjust the gimbal to point directly above the production line at the designated position;

[0045] 2. Acquire image stream, capturing 1 image per second for a total of 50 seconds; Figure 3 This is an image of the production line taken in this embodiment;

[0046] 3. Use the YOLO model to recognize the first 10 images. If the target is recognized, continue the process; if the target is not recognized, end the process.

[0047] 4. Based on the model detection results, obtain the top, bottom, left, and right border coordinates of the meter, and use the detection results as image detection boxes. Fix the position of the detection boxes and perform cropping preprocessing on the subsequent images.

[0048] 5. Apply Gaussian filtering to the cropped image to remove noise, and convert the image in RGB space to grayscale; Figure 4 This is an image of an electricity meter that has undergone cropping and noise reduction processing in this embodiment;

[0049] 6. In the image stream, subtract each pair of images sequentially to form a difference image. Figure 5 This is a test sample of a difference operation image under a moving state. Figure 6 This is a test sample of a difference operation image in a static state;

[0050] 7. Perform image erosion operation on the difference image to filter out noise caused by robot vibration and changes in ambient light and shadow;

[0051] 8. Set a threshold; clear the motion judgment count value and set a motion judgment threshold; calculate the sum of the elements of the difference operation image matrix data. If it is greater than the set threshold, the image is determined to be a moving image, and the motion judgment count value is incremented by one.

[0052] 9. After completing the calculation of all images in the image stream, if the accumulated motion judgment count is greater than the set motion judgment threshold, the pipeline is judged to be in motion; if it is not greater than the set motion judgment threshold, the pipeline is judged to be stationary and an alarm is sent to the background.

[0053] Figure 7 This is a flowchart illustrating a method for monitoring the status of an electricity meter in a production line, as described in another embodiment of this application. The process includes:

[0054] 1. In the training phase, for the collected electricity meter dataset, the self-attention mechanism (SE) module and the CBAM module were added to the YOLOv5 model to enhance the training results, resulting in a model with an accuracy of 97%.

[0055] 2. Control the robot to reach the designated position, and adjust the gimbal to point directly above the production line at the designated position;

[0056] 3. Obtain the image stream, use the imread function to load the image into memory, and use Torch.hub.load to call the trained model to perform YOLOv5 inference to obtain the detection results;

[0057] 4. Obtain the top, bottom, left, and right border coordinates of the meter, and use the detection results as the region of interest (ROI) of the image. Perform preprocessing and cropping on the ROI to obtain the processed image.

[0058] 5. Apply GaussianBlur function with 3x3 convolution kernel to the cropped image to remove image noise, and use CvtColor function with COLOR_BGR2RGAY parameter to convert the image in RGB space to grayscale.

[0059] 6. Use the Absdiff function in the image stream to subtract each pair of grayscale images in sequence to form a difference image; filter the difference image using the Threshod function, select a grayscale threshold of 60, and change the grayscale function value greater than the threshold to 255;

[0060] 7. Based on the difference operation image, construct a 5x5 convolution kernel using the Ones function, and then perform image erosion operation using the erode function to obtain a second difference operation image, which filters out environmental noise caused by robot vibration and changes in ambient light shadows.

[0061] 8. Perform the same difference operation and processing on the next two images to obtain two consecutive quadratic difference images;

[0062] 9. Use Torch.hub.load to call the three-body twin network training model, perform three-body twin neural network inference on the two difference operation images, and obtain the similarity confidence P. If the similarity confidence P is less than 15%, the motion judgment count value is incremented by one.

[0063] 10. Set the motion judgment threshold to 10. If the cumulative result of the motion judgment count is greater than the motion judgment threshold, the pipeline is judged to be moving and the process ends. If the motion judgment count is less than the motion judgment threshold and the judgment is not completed, the next round of judgment continues until all images are judged. If the motion judgment count is still less than the motion judgment threshold, the pipeline is judged to be stationary.

[0064] In one specific embodiment, based on a two-body twin neural network constructed using a VGG16 backbone network architecture, a negative sample input head, also part of the VGG16 backbone network, is added to construct a three-body twin neural network. This network is used to determine whether two positive sample input heads are similar and to provide a final similarity probability. Simultaneously, a shared weight method is employed to ensure that image similarity is only determined when most pixels are identical. Figure 9 This is a schematic diagram of the three-body Siamese neural network structure with an added negative sample input head in this embodiment. During training, images with quadratic differences in a static state are used as positive examples, and images with quadratic differences in a moving state are used as negative examples. After feature extraction by the VGG16 backbone network, multidimensional features from the three input heads are obtained, which are then converted into one-dimensional vectors. The L1 norm of the corresponding feature vector interpolation is obtained by summing the absolute values ​​of the differences between each pair of the three one-dimensional vectors. Compared to the two-body Siamese neural network, the improved three-body Siamese network algorithm can better distinguish negative example image samples and can differentiate between static and moving images with a smaller training load, reducing training costs.

[0065] Figure 9 This is a schematic diagram of the structure of a production line status monitoring system 900 according to one embodiment of the present application, including:

[0066] Image acquisition module 901 is configured to acquire multiple consecutive images of the production line using a rail-mounted robot.

[0067] The target detection module 902 is configured to use a trained target detection model to determine whether there is a target in the image; if so, the judgment continues, otherwise the judgment ends.

[0068] The state detection module 903 is configured to perform a difference operation on two consecutive images in the pipeline image to obtain a difference operation image, determine the difference between consecutive pipeline images based on the difference operation image, and determine the pipeline motion state by combining the judgment results of multiple consecutive pipeline images.

[0069] The method and system for monitoring the status of an assembly line using a track-mounted robot proposed in this application determine the motion status of the assembly line through image recognition algorithms. It can be used to detect the status of most assembly lines and is more flexible and intelligent than traditional sensor monitoring.

[0070] Although the contents of this application have been specifically shown and described in conjunction with preferred embodiments, those skilled in the art should understand that any changes in form and detail made to this application without departing from the spirit and scope of this application as defined by the appended claims and without inventive effort are within the scope of protection of this application.

Claims

1. A method for monitoring the status of an assembly line, characterized in that, Includes the following steps: S1, using a rail-mounted robot to collect multiple consecutive images of the production line; S2, use the trained target detection model to determine whether there is a target in the image; if yes, continue the judgment; otherwise, end the judgment. S3, perform a difference operation on two consecutive images in the pipeline image to obtain a difference operation image, determine the difference between consecutive pipeline images based on the difference operation image, and determine the pipeline motion state by combining the judgment results of multiple consecutive pipeline images. Perform a difference operation on two consecutive difference operation images to obtain a quadratic difference operation image; The similarity of two consecutive quadratic difference operation images is judged by a trained Siamese neural network. If they are not similar, the motion judgment count is incremented by one; otherwise, the motion judgment count remains unchanged. A threshold judgment is performed on multiple quadratic difference image matrix data. If the motion judgment count value is greater than the set motion judgment threshold, the pipeline is judged to be moving; otherwise, the pipeline is judged to be stagnant.

2. The method for monitoring the status of an assembly line according to claim 1, characterized in that, The target detection model described in S2 is built on the YOLO algorithm framework and incorporates the self-attention mechanism (SE) module and the CBAM module.

3. The method for monitoring the status of an assembly line according to claim 2, characterized in that, S2 further includes: obtaining image detection boxes based on the first N images, and cropping the images after the Nth image based on the image detection boxes.

4. The method for monitoring the status of an assembly line according to claim 1, characterized in that, S3 further includes: Based on the difference operation image, obtain the difference operation image matrix data; A threshold judgment is performed on each difference operation image matrix data. The threshold judgment includes: calculating the sum of elements of each difference operation image matrix data and comparing it with a set threshold. If it is greater than the threshold, the motion judgment count value is incremented by one; otherwise, the motion judgment count value remains unchanged. A threshold judgment is performed on multiple difference operation image matrix data. If the motion judgment count value is greater than the set motion judgment threshold, the pipeline is judged to be moving; otherwise, the pipeline is judged to be stagnant.

5. The method for monitoring the status of an assembly line according to claim 1, characterized in that, The twin neural network comprises two VGG16 neural networks with shared weights.

6. The method for monitoring the status of an assembly line according to claim 5, characterized in that, The twin neural network is a three-body twin neural network. The input of the three-body twin neural network includes two sets of positive samples and one set of negative samples. The positive samples are quadratic difference operation images obtained when the pipeline is stationary, and the negative samples are quadratic difference operation images obtained when the pipeline is in motion.

7. A method for monitoring the status of an assembly line according to claim 1 or 4, characterized in that, It also includes image processing of the difference operation image and / or the quadratic difference operation image, the image processing including grayscale conversion and erosion.

8. A production line status monitoring system, characterized in that, include: The image acquisition module is configured to acquire multiple consecutive images of the production line using a rail-mounted robot. The target detection module is configured to use a trained target detection model to determine whether a target exists in the image; if so, the determination continues; otherwise, the determination ends. The state detection module is configured to perform a difference operation on two consecutive images in the pipeline image to obtain a difference operation image, determine the difference between consecutive pipeline images based on the difference operation image, and determine the pipeline motion state by combining the judgment results of multiple consecutive pipeline images. Perform a difference operation on two consecutive difference operation images to obtain a quadratic difference operation image; The similarity of two consecutive quadratic difference operation images is judged by a trained Siamese neural network. If they are not similar, the motion judgment count is incremented by one; otherwise, the motion judgment count remains unchanged. A threshold judgment is performed on multiple quadratic difference image matrix data. If the motion judgment count value is greater than the set motion judgment threshold, the pipeline is judged to be moving; otherwise, the pipeline is judged to be stagnant.

Images