A method, system and medium for remote real-time monitoring of welding based on NB-IoT

Abstract

The application relates to a method, system and medium for remotely monitoring welding in real time based on NB-IoT, which comprises the following steps: U1. starting a welding operation, acquiring welding seam image data information of a welding part in real time based on a camera between welding operations, performing image labeling, and outputting the labeled welding seam image data information; U2. dividing the labeled welding seam image data information into a welding seam image training data set and a welding seam image test data set, inputting the welding seam image training data set into an improved YOLOv5 network model for training and learning, and outputting the trained improved YOLOv5 network model; and U3. based on the trained improved YOLOv5 network model, inputting the welding seam image test data set, predicting the image of the welding seam, and obtaining predicted welding seam image data information. The application can not only accurately detect the image of the welding seam, but also has high automation and can be applied to different complex environments.

Description

Technical Field

[0001] This invention relates to the field of weld seam image detection technology, and in particular to a method, system and medium for remote real-time monitoring of welding based on NB-IoT. Background Technology

[0002] Image detection technology is constantly developing and being applied to various fields, continuously improving the automation of these fields. Therefore, how to better utilize image detection technology has become an urgent problem for us to solve.

[0003] As people's demand for automated weld defect detection continues to increase, visual inspection of welds will become a long-standing topic in the field of nondestructive testing. However, the problems of low detection accuracy and semi-automation of inspection systems have not yet been effectively solved. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method, system and medium for remote real-time monitoring of welding based on NB-IoT, which can not only accurately detect the image of the weld, but also has a high degree of automation and can be applied to different complex environments.

[0005] To achieve the above and other related objectives, the present invention provides the following technical solution:

[0006] A method for remote real-time monitoring of welding based on NB-IoT, the method comprising:

[0007] U1. Welding operation begins. Based on the camera in the welding operation room, the weld seam image data information of the workpiece is acquired in real time, and the image is annotated and the annotated weld seam image data information is output.

[0008] U2. Divide the labeled weld image data into a weld image training dataset and a weld image test dataset. Input the weld image training dataset into the improved YOLOv5 network model for training and learning, and output the trained improved YOLOv5 network model.

[0009] U3. Based on the trained improved YOLOv5 network model, input the weld seam image test dataset, predict the weld seam image, and obtain the predicted weld seam image data information;

[0010] U4. Based on the predicted weld image data, the weld IoU detection algorithm is used to compare with the transmitted weld image, and the detection data of the weld image is output.

[0011] Furthermore, in step U1, the image annotation includes:

[0012] U11. Based on the weld seam image data information of the weldment, a bilateral filtering algorithm is used to perform image noise reduction processing, and the noise-reduced weld seam image data information is output.

[0013] U12. Based on the weld image data information after noise reduction, establish an image enhancement function F.

[0014] ,

[0015] r∈[0,1],

[0016] Where c is the enhancement coefficient, v is the weld image data information, and r is the image expansion factor, the enhanced weld image data information is obtained;

[0017] U13. Based on the enhanced weld image data information, the image contour points are annotated to obtain the annotated weld image data information.

[0018] Furthermore, the annotation of the image contour points is based on the enhanced weld image data information, obtaining the pixel value data information of each pixel in the weld image, setting a preset threshold, and if the pixel value of a pixel exceeds the preset threshold, it is annotated; if the pixel value of a pixel does not exceed the preset threshold, it is not annotated.

[0019] Furthermore, in step U2, the improved YOLOv5 network model includes an Input module, a Backbone module, a Neck module, and a Prediction module. The Input module is connected to the Backbone module, the Backbone module is connected to the Neck module, and the Neck module is connected to the Prediction module. A CBAMC3 module is added between the Backbone module and the Neck module for data transfer between each connected module and parameter aggregation between non-connected modules.

[0020] Furthermore, the Neck module employs a bounding box regression algorithm to detect small targets in the weld seam image. The bounding box regression algorithm includes:

[0021] L1. Obtain the data information of the weld seam image, perform bounding box labeling for each target in the image, define the four-dimensional vector of the bounding box (x,y,w,h), where (x,y) are the coordinates of the center point of the bounding box, w is the width of the bounding box, and h is the height of the bounding box, and output the four-dimensional vector data information of the bounding box of the target.

[0022] L2. Based on the four-dimensional vector data of the target's bounding box, establish a bounding box regression function G.

[0023] ,

[0024] p i =(x i ,y i ,w i ,h i ),

[0025] Where, p i Let x be the i-th bounding box vector of the image target. i ,y i Let w be the coordinates of the center point of the i-th bounding box of the image target. i h is the width of the i-th bounding box of the image target. i Let λ be the height of the i-th bounding box of the image target. i Let be the weight coefficient of the i-th bounding box of the image target, and n be the total number of samples;

[0026] L3. Based on the bounding box regression function G, detect the target in the image and output the detection data information of the target in the image.

[0027] Furthermore, the constraint condition for the weight coefficient λi of the i-th bounding box of the image target is:

[0028] .

[0029] Furthermore, in step U4, the comparison between the weld IoU detection algorithm and the transmitted weld image includes:

[0030] U41. Perform border calibration on the predicted weld seam image data information to obtain predicted weld seam image border calibration data information, and perform border calibration on the transmitted image to obtain transmitted image border calibration data information;

[0031] U42. Based on the border calibration data of the transmitted image and the predicted weld seam image border calibration data, a weld seam detection function D is established.

[0032] ,

[0033] Wherein, B1 is the border calibration data of the transmitted image, and B2 is the border calibration data of the predicted weld image;

[0034] U43. Based on the weld detection function D, obtain the detection data information of the weld image.

[0035] To achieve the above and other related objectives, the present invention also provides a system for implementing the NB-IoT-based remote real-time monitoring welding method as described in any one of the claims, comprising a terminal layer, a transmission layer, and a platform layer, characterized in that: the terminal layer includes a UI interface, a main controller, and a camera; the UI interface is connected to the main controller; the main controller is connected to the camera; the camera is used to acquire weld seam image data information of the weldment in real time; the main controller is used to receive instruction data information for acquiring weld seam images; and the UI interface is used to display weld seam image data information in real time.

[0036] The transmission layer includes a wireless radio frequency module, which is communicatively connected to the main controller and used to transmit instruction data information.

[0037] The platform layer includes a host computer interface, which is connected to the wireless radio frequency module and is used to send weld seam image acquisition command data information.

[0038] Furthermore, the system also includes a voice broadcasting system, which is connected to the host computer interface and is used to broadcast the data information of the issued instructions.

[0039] To achieve the above and other related objectives, the present invention also provides a computer-readable storage medium storing a computer program programmed or configured to perform any of the methods for remote real-time monitoring of welding based on NB-IoT.

[0040] The present invention has the following positive effects:

[0041] 1. This invention divides the labeled weld image data into a weld image training dataset and a weld image test dataset. The weld image training dataset is input into an improved YOLOv5 network model for training and learning, resulting in a well-trained improved YOLOv5 network model for predicting weld images. This not only enables accurate detection of weld images but also has a high degree of automation and can be applied to various complex environments.

[0042] 2. This invention uses a weld IoU detection algorithm to compare the predicted weld image with the transmitted weld image, thereby obtaining the detection data of the weld image. This not only facilitates the actual improvement work of welders in the later stage, but also allows managers to filter out unqualified welds by pressing a button, which is convenient and simple to operate. Attached Figure Description

[0043] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0044] Figure 2This is a diagram of the improved YOLOv5 network model architecture of the present invention;

[0045] Figure 3 This is a schematic diagram of the transmitted weld image of the present invention;

[0046] Figure 4 This is a schematic diagram of the predicted weld image of the present invention;

[0047] Figure 5 This is a schematic diagram of the system framework of the present invention. Detailed Implementation

[0048] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0049] Example 1: As Figure 1 As shown, a method for remote real-time monitoring of welding based on NB-IoT is provided, the method comprising:

[0050] U1. Welding operation begins. Based on the camera in the welding operation room, the weld seam image data information of the workpiece is acquired in real time, and the image is annotated and the annotated weld seam image data information is output.

[0051] U2. Divide the labeled weld image data into a weld image training dataset and a weld image test dataset. Input the weld image training dataset into the improved YOLOv5 network model for training and learning, and output the trained improved YOLOv5 network model.

[0052] U3. Based on the trained improved YOLOv5 network model, input the weld seam image test dataset, predict the weld seam image, and obtain the predicted weld seam image data information;

[0053] U4. Based on the predicted weld image data, the weld IoU detection algorithm is used to compare with the transmitted weld image, and the detection data of the weld image is output.

[0054] In this embodiment, step U1, the image annotation includes:

[0055] U11. Based on the weld seam image data information of the weldment, a bilateral filtering algorithm is used to perform image noise reduction processing, and the noise-reduced weld seam image data information is output.

[0056] U12. Based on the weld image data information after noise reduction, establish an image enhancement function F.

[0057] ,

[0058] r∈[0,1],

[0059] Where c is the enhancement coefficient, v is the weld image data information, and r is the image expansion factor, the enhanced weld image data information is obtained;

[0060] U13. Based on the enhanced weld image data information, the image contour points are annotated to obtain the annotated weld image data information.

[0061] In this embodiment, the annotation of the image contour points is achieved by obtaining the pixel value data information of each pixel in the weld image based on the enhanced weld image data information, setting a preset threshold, and annotating if the pixel value of a pixel exceeds the preset threshold, and not annotating if the pixel value of a pixel does not exceed the preset threshold.

[0062] Example 2: Based on the method for remote real-time monitoring of welding based on NB-IoT in Example 1, the present invention will be further described and explained below.

[0063] like Figure 1 As shown, a method for remote real-time monitoring of welding based on NB-IoT is provided, the method comprising:

[0064] U1. Welding operation begins. Based on the camera in the welding operation room, the weld seam image data information of the workpiece is acquired in real time, and the image is annotated and the annotated weld seam image data information is output.

[0065] U2. Divide the labeled weld image data into a weld image training dataset and a weld image test dataset. Input the weld image training dataset into the improved YOLOv5 network model for training and learning, and output the trained improved YOLOv5 network model.

[0066] U3. Based on the trained improved YOLOv5 network model, input the weld seam image test dataset, predict the weld seam image, and obtain the predicted weld seam image data information;

[0067] U4. Based on the predicted weld image data, the weld IoU detection algorithm is used to compare with the transmitted weld image, and the detection data of the weld image is output.

[0068] In this embodiment, as Figure 2As shown, in step U2, the improved YOLOv5 network model includes an Input module, a Backbone module, a Neck module, and a Prediction module. The Input module is connected to the Backbone module, the Backbone module is connected to the Neck module, and the Neck module is connected to the Prediction module. A CBAMC3 module is added between the Backbone module and the Neck module for data transfer between each connected module and parameter aggregation between non-connected modules.

[0069] In this embodiment, the Neck module uses a bounding box regression algorithm to detect small targets in the weld seam image. The bounding box regression algorithm includes:

[0070] L1. Obtain the data information of the weld seam image, perform bounding box labeling for each target in the image, define the four-dimensional vector of the bounding box (x,y,w,h), where (x,y) are the coordinates of the center point of the bounding box, w is the width of the bounding box, and h is the height of the bounding box, and output the four-dimensional vector data information of the bounding box of the target.

[0071] L2. Based on the four-dimensional vector data of the target's bounding box, establish a bounding box regression function G.

[0072] ,

[0073] p i =(x i ,y i ,w i ,h i ),

[0074] Where, p i Let x be the i-th bounding box vector of the image target. i ,y i Let w be the coordinates of the center point of the i-th bounding box of the image target. i h is the width of the i-th bounding box of the image target. i Let λ be the height of the i-th bounding box of the image target. i Let be the weight coefficient of the i-th bounding box of the image target, and n be the total number of samples;

[0075] L3. Based on the bounding box regression function G, detect the target in the image and output the detection data information of the target in the image.

[0076] Furthermore, the constraint condition for the weight coefficient λi of the i-th bounding box of the image target is:

[0077] .

[0078] In this embodiment, as Figure 3 or Figure 4 As shown, in step U4, the comparison between the weld IoU detection algorithm and the transmitted weld image includes:

[0079] U41. Perform border calibration on the predicted weld seam image data information to obtain predicted weld seam image border calibration data information, and perform border calibration on the transmitted image to obtain transmitted image border calibration data information;

[0080] U42. Based on the border calibration data of the transmitted image and the predicted weld seam image border calibration data, a weld seam detection function D is established.

[0081] ,

[0082] Wherein, B1 is the border calibration data of the transmitted image, and B2 is the border calibration data of the predicted weld image;

[0083] U43. Based on the weld detection function D, obtain the detection data information of the weld image.

[0084] like Figure 5 As shown, the present invention provides a system for implementing any of the methods for remote real-time monitoring of welding based on NB-IoT, comprising a terminal layer, a transmission layer, and a platform layer, characterized in that: the terminal layer includes a UI interface, a main controller, and a camera; the UI interface is connected to the main controller, the main controller is connected to the camera; the camera is used to acquire weld seam image data information of the weldment in real time; the main controller is used to receive instruction data information for acquiring weld seam images; and the UI interface is used to display weld seam image data information in real time.

[0085] The transmission layer includes a wireless radio frequency module, which is communicatively connected to the main controller and used to transmit instruction data information.

[0086] The platform layer includes a host computer interface, which is connected to the wireless radio frequency module and is used to send weld seam image acquisition command data information.

[0087] In this embodiment, the system further includes a voice broadcasting system, which is connected to the host computer interface and is used to broadcast the data information of the issued instructions.

[0088] The present invention provides a computer-readable storage medium storing a computer program programmed or configured to perform any of the methods for remote real-time monitoring of welding based on NB-IoT.

[0089] Any references to memory, storage, database, or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0090] In summary, this invention not only enables accurate detection of weld seam images, but also boasts a high degree of automation, making it applicable to various complex environments.

[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for remote real-time monitoring of welding based on NB-IoT, characterized in that, The method includes: U1. Welding operation begins. Based on the camera in the welding operation room, the weld seam image data information of the workpiece is acquired in real time, and the image is annotated and the annotated weld seam image data information is output. U2. Divide the labeled weld image data into a weld image training dataset and a weld image test dataset. Input the weld image training dataset into the improved YOLOv5 network model for training and learning, and output the trained improved YOLOv5 network model. U3. Based on the trained improved YOLOv5 network model, input the weld seam image test dataset, predict the weld seam image, and obtain the predicted weld seam image data information; U4. Based on the predicted weld image data, the weld IoU detection algorithm is used to compare with the transmitted weld image, and the detection data information of the weld image is output. In step U2, the improved YOLOv5 network model includes an Input module, a Backbone module, a Neck module, and a Prediction module. The Input module is connected to the Backbone module, the Backbone module is connected to the Neck module, and the Neck module is connected to the Prediction module. A CBAMC3 module is added between the Backbone module and the Neck module for data transfer between connected modules and parameter aggregation between unconnected modules. The Neck module uses a bounding box regression algorithm to detect small targets in the weld seam image. The bounding box regression algorithm includes: L1. Obtain the data information of the weld seam image, perform bounding box labeling for each target in the image, define the four-dimensional vector of the bounding box (x, y, w, h), where (x, y) are the coordinates of the center point of the bounding box, w is the width of the bounding box, and h is the height of the bounding box, and output the four-dimensional vector data information of the bounding box of the target. L2. Based on the four-dimensional vector data of the target's bounding box, establish a bounding box regression function G. ， p i =(x i ,y i ,w i ,h i ), where p i Let x be the i-th bounding box vector of the image target. i ,y i Let w be the coordinates of the center point of the i-th bounding box of the image target. i h is the width of the i-th bounding box of the image target. i Let λ be the height of the i-th bounding box of the image target. i Let be the weight coefficient of the i-th bounding box of the image target, and n be the total number of samples; L3. Based on the bounding box regression function G, detect the image target and output the detection data information of the image target; the weight coefficient λ of the i-th bounding box of the image target. i The constraints are: 。 2. The method for remote real-time monitoring of welding based on NB-IoT according to claim 1, characterized in that, In step U1, the image annotation includes: U11. Based on the weld seam image data information of the weldment, a bilateral filtering algorithm is used to perform image noise reduction processing, and the noise-reduced weld seam image data information is output. U12. Based on the weld image data information after noise reduction, establish an image enhancement function F. ， r∈[0,1]， Where c is the enhancement coefficient, v is the weld image data information, and r is the image expansion factor, the enhanced weld image data information is obtained; U13. Based on the enhanced weld image data information, the image contour points are annotated to obtain the annotated weld image data information.

3. The method for remote real-time monitoring of welding based on NB-IoT according to claim 2, characterized in that: The annotation of the image contour points is based on the enhanced weld image data information, which obtains the pixel value data information of each pixel in the weld image, sets a preset threshold, and if the pixel value of a pixel exceeds the preset threshold, it is annotated; if the pixel value of a pixel does not exceed the preset threshold, it is not annotated.

4. The method for remote real-time monitoring of welding based on NB-IoT according to claim 1, characterized in that, In step U4, the comparison between the weld IoU detection algorithm and the transmitted weld image includes: U41. Perform border calibration on the predicted weld image data information to obtain the predicted weld image border calibration data information, and perform border calibration on the transmitted weld image to obtain the transmitted weld image border calibration data information. U42. Based on the transmitted weld seam image border calibration data and the predicted weld seam image border calibration data, a weld seam detection function D is established. ， Wherein, B1 is the border calibration data of the transmitted weld image, and B2 is the border calibration data of the predicted weld image; U43. Based on the weld detection function D, obtain the detection data information of the weld image.

5. A system for implementing the NB-IoT-based remote real-time monitoring welding method according to any one of claims 1-4, comprising a terminal layer, a transmission layer, and a platform layer, characterized in that: The terminal layer includes a UI interface, a main controller, and a camera. The UI interface is connected to the main controller, and the main controller is connected to the camera. The camera is used to acquire weld seam image data information of the weldment in real time. The main controller is used to receive instruction data information for acquiring weld seam images. The UI interface is used to display weld seam image data information in real time. The transmission layer includes a wireless radio frequency module, which is communicatively connected to the main controller and used to transmit instruction data information. The platform layer includes a host computer interface, which is connected to the wireless radio frequency module and is used to send weld seam image acquisition command data information.

6. The system according to claim 5, characterized in that: The system also includes a voice broadcasting system, which is connected to the host computer interface and is used to broadcast the data information of the issued instructions.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that is programmed or configured to perform the NB-IoT-based remote real-time monitoring welding method as described in any one of claims 1 to 4.

Images