A method for real-time segmentation of video of tool breakage
By combining a deep learning-based tool condition monitoring method with image enhancement, classification, object detection, and a lightweight real-time semantic segmentation network, the problems of real-time tool breakage monitoring and high computational burden in traditional methods are solved, and real-time segmentation and efficient processing under machine tool conditions are realized.
Patent Information
- Application Number
- CN202410400180.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-03
- Publication Date
- 2026-11-17
- Estimated Expiration
- 2044-04-03
AI Technical Summary
Traditional sensor monitoring methods are difficult to implement in real time monitoring of tool breakage, and deep learning networks face high computational pressure under machine tool conditions, making it difficult to guarantee real-time monitoring.
A deep learning-based tool condition monitoring method is adopted, including image enhancement and augmentation processing, tool working condition classification, target detection of damaged areas and image segmentation. A lightweight real-time semantic segmentation network is used for real-time segmentation, and VGG-16D, YOLO and super-resolution technology are used to optimize image processing.
It enables real-time segmentation of tool breakage under machine tool conditions, improves operating speed and reduces resource consumption, and can perform real-time processing of precision machining tools.
Smart Images

Figure CN118334331B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of tool breakage monitoring and relates to a real-time segmentation method for tool breakage videos. Background Technology
[0002] In traditional machining, tool breakage and cutting edge conditions directly affect whether the parts can meet the customer's requirements for product dimensional accuracy and surface quality.
[0003] However, due to the complex machining conditions, traditional sensor monitoring methods often require or necessitate machine shutdown for inspection, making real-time monitoring difficult; or, due to frequent parameter changes, it is difficult to obtain accurate data to predict tool condition. These problems make it difficult to independently apply traditional sensor monitoring methods in actual production.
[0004] Current widely researched machine vision and deep learning methods often struggle to obtain the necessary clear images for training under machine tool operating conditions. Furthermore, when current deep learning networks are directly used for segmenting damaged areas, the sheer number of parameters creates immense computational pressure, making it difficult to widely undertake this task under current machine tool conditions, thus compromising the real-time performance of monitoring. Summary of the Invention
[0005] To address the aforementioned problems in the background art, the present invention provides a real-time video segmentation method for tool breakage.
[0006] The objective of this invention is achieved through the following technical solution:
[0007] A method for real-time segmentation of video of tool breakage includes the following steps:
[0008] Step S1: Collect tool images after machining for training and tool machining videos for testing. Perform image enhancement and augmentation processing on the collected images to obtain the original tool dataset.
[0009] Step S2: Perform image classification and annotation of tool working status, target detection and annotation of tool broken area, and image segmentation and annotation of tool broken area on the original tool dataset obtained in step S1 to construct tool working status classification dataset, tool broken area target detection dataset, and tool broken area image segmentation dataset;
[0010] Step S3: Decompose the tool processing video used for testing in step S1 into continuous images frame by frame to obtain a tool breakage video decomposition image dataset.
[0011] Step S4: Build a deep learning-based tool condition monitoring training module. Use this module to train the tool working state classification dataset, tool damage area target detection dataset, and tool damage area image segmentation dataset from Step S2 in sequence to obtain the training model.
[0012] Step S5: Build a deep learning-based tool condition monitoring and inference module. Use this module to learn the model trained in step S4 and perform inference on the tool damage video decomposition dataset obtained in step S3.
[0013] Compared with the prior art, the present invention has the following advantages:
[0014] 1. It features faster processing speed and lower resource consumption. In the real-time segmentation method for tool breakage video of this invention, the real-time segmentation implementation part uses a window of the same resolution extracted from the tool breakage area in the previous step. Therefore, the segmentation area size is significantly smaller compared to directly segmenting the entire image. Furthermore, the tool breakage area localization in the previous step uses target detection technology, which has significantly better resource consumption and processing speed than image segmentation technology. Therefore, this invention is superior to conventional direct image segmentation techniques in terms of processing speed and resource consumption.
[0015] 2. Real-time processing of precision-machined cutting tools. The method of this invention incorporates super-resolution technology, which performs super-resolution reconstruction based on video sequences. In current equipment environments, industrial cameras often struggle to capture clear video of precision-machined cutting tools, hindering subsequent deep learning image processing. By introducing image super-resolution technology, which effectively performs super-resolution reconstruction on images based on video sequences, the tool movement during machining is a simple cyclic process, enabling the images to meet the standards for subsequent deep learning processing. Attached Figure Description
[0016] Figure 1 A flowchart for a real-time video segmentation method for tool breakage;
[0017] Figure 2 A diagram of a lightweight real-time semantic segmentation network structure;
[0018] Figure 3 A diagram showing the cutting damage of the cutting tools used;
[0019] Figure 4 This is a schematic diagram of modules M1 and M2. Detailed Implementation
[0020] The technical solution of the present invention will be further described below with reference to the accompanying drawings, but it is not limited thereto. Any modifications or equivalent substitutions to the technical solution of the present invention that do not depart from the spirit and scope of the technical solution of the present invention should be covered within the protection scope of the present invention.
[0021] This invention provides a real-time segmentation method for tool breakage videos. The method includes the following modules: a deep learning-based tool state monitoring training module and a deep learning-based tool state monitoring inference module. These two modules are further composed of the following sub-modules: a VGG-16D-based tool working state classification module, used for rapid initial tool state classification to accelerate subsequent processing; a deep learning-based tool breakage region super-resolution module to address the blurring problem in tool images captured with high precision; a YOLO-based tool breakage region localization module, utilizing this technology to detect tool breakage regions and assist in subsequent segmentation; and a lightweight network-based region of interest (ROI) image segmentation module, performing image segmentation based on the previously detected ROI to obtain the final high-precision real-time segmentation data. The specific steps are as follows:
[0022] Step S1: Collect tool images after machining for training and tool machining videos for testing. Perform image enhancement and augmentation processing on the collected images to obtain the original tool dataset.
[0023] In this step, image enhancement and augmentation processes are performed using methods such as adding noise, increasing contrast, rotation, translation, and histogram equalization.
[0024] Step S2: Perform image classification and annotation of tool working status, target detection and annotation of tool broken area, and image segmentation and annotation of tool broken area on the original tool dataset obtained in step S1 to construct tool working status classification dataset, tool broken area target detection dataset, and tool broken area image segmentation dataset.
[0025] In this step, the images in the tool working status classification dataset are compressed to 224×224.
[0026] Step S3: Decompose the tool processing video used for testing in step S1 into continuous images frame by frame to obtain a tool breakage video decomposition image dataset.
[0027] Step S4: Build a deep learning-based tool condition monitoring training module. Use this module to train the tool working state classification dataset, tool damage area target detection dataset, and tool damage area image segmentation dataset from Step S2 in sequence to obtain the training model.
[0028] In this step, the structure of the deep learning-based tool condition monitoring training module is as follows:
[0029] A1. A tool working state classification module based on VGG16-D. This module uses the tool working state classification dataset constructed in step S2 to train an image classification model, thereby obtaining a tool working state classification model. The tool working state classification dataset is divided into a training set and a validation set in an 8:2 ratio. The training set is used to train the VGG16-D-based tool working state classification module, and the validation set is used to validate the trained tool working state classification model to determine if the function can be implemented.
[0030] A2. A YOLO-based tool breakage area localization module. This module uses the tool breakage area target detection dataset constructed in step S2 to train the target detection model and obtain the tool breakage area target detection model.
[0031] A3. A tool breakage region segmentation module based on a lightweight real-time semantic segmentation network. This module uses the tool breakage region image segmentation dataset constructed in step S2 to train the image segmentation model and obtain the tool breakage region image segmentation model.
[0032] Step S5: Build a deep learning-based tool condition monitoring and inference module. Use this module to learn the model trained in step S4 and perform inference on the tool breakage video decomposition dataset obtained in step S3.
[0033] In this step, the structure of the deep learning-based tool state detection and inference module is as follows:
[0034] B1. A tool working state classification module based on VGG16-D. This module learns the tool working state classification model obtained in A1, performs image classification reasoning on the tool damage video decomposition image dataset in step S3, obtains a classification reasoning result, and extracts the damage classification images as a damage classification dataset.
[0035] B2. A deep learning-based super-resolution processing module that processes the damaged classification dataset mentioned in B1 to improve the image perception quality and thus construct a super-resolution damaged dataset.
[0036] B3. A YOLO-based tool breakage area localization module. This module learns the tool breakage area target detection model obtained in A2, performs target detection inference on the super-resolution breakage classification dataset in B2, retains the location information of the detected parts and extracts them from the original image, thereby constructing a breakage interest dataset.
[0037] B4. A tool breakage region segmentation module based on a lightweight real-time semantic segmentation network. This module learns the tool breakage region image segmentation model obtained in A3, performs image segmentation inference on the breakage interest dataset in B3, and maps the obtained segmentation results back to the original image through the location information mentioned in B3 to obtain the tool breakage region segmentation result.
[0038] In this step, such as Figure 2 As shown, the lightweight real-time semantic segmentation network contains a symmetric encoder-decoder architecture consisting of 18 convolutional layers. These 18 convolutional layers incorporate two types of residual convolutional modules, where:
[0039] The structure of the residual module is as follows:
[0040] C1: Two residual convolutional modules are introduced. One uses factorization convolution to extract low-level features and is called the M1 module. The other uses multi-branch dilated convolution to expand the receptive field to capture high-level semantic features and is called the M2 module.
[0041] C2: such as Figure 4 As shown in (a), the M1 module is designed using a Non-bt-1D module, which helps to significantly reduce the number of parameters and accelerates the training and inference processes. Meanwhile, the kernel size is designed to be variable, allowing the module to adaptively expand its receptive field.
[0042] C3: such as Figure 4 As shown in (b), the M2 module is designed using a transform-split-transform-merge strategy, selecting parallel multi-branch dilated convolutions. Each branch's dilated convolution uses a different rate to expand the receptive field. This design allows the dilated convolutions to adaptively capture objects of different scales, ensuring the required accuracy while reducing computational complexity. At the start of each M2 module, the input is first passed through a dedicated 1D convolution, and the output is passed through three parallel dilated convolutions at three different rates. After an identity mapping, the outputs of the three convolutions are merged with the initial input and passed to the next M2 module. Within the same M2 module, each branch shares the same convolutional feature map.
[0043] The encoder-decoder architecture is structured as follows:
[0044] D1: The encoder consists of three parts. The first two parts introduce the M1 module, and the third part introduces the M2 module. The first part uses a smaller convolutional kernel to extract low-level image features, while the second part uses a larger convolutional kernel to capture a wider range of contextual information.
[0045] D2: The decoder consists of three parts. The first two parts introduce the M1 module, and the third part introduces the fully convolutional module. The first part uses a larger convolutional kernel to collect long-range information to improve prediction accuracy, while the second part uses a smaller convolutional kernel to recover image details through the response of neighboring pixels.
Claims
1. A method for real-time segmentation of video of tool breakage, characterized in that... The method includes the following steps: Step S1: Collect tool images after machining for training and tool machining videos for testing. Perform image enhancement and augmentation processing on the collected images to obtain the original tool dataset. Step S2: Perform image classification and annotation of tool working status, target detection and annotation of tool broken area, and image segmentation and annotation of tool broken area on the original tool dataset obtained in step S1 to construct tool working status classification dataset, tool broken area target detection dataset, and tool broken area image segmentation dataset; Step S3: Decompose the tool processing video used for testing in step S1 into continuous images frame by frame to obtain a tool breakage video decomposition image dataset. Step S4: Build a deep learning-based tool condition monitoring training module. Use this module to train the tool working state classification dataset, tool damage area target detection dataset, and tool damage area image segmentation dataset from Step S2 in sequence to obtain the training model. The structure of the deep learning-based tool condition monitoring training module is as follows: A1. A tool working state classification module based on VGG16-D. This module uses the tool working state classification dataset constructed in step S2 to train the image classification model and obtain the tool working state classification model. A2. A YOLO-based tool breakage area localization module. This module uses the tool breakage area target detection dataset constructed in step S2 to train the target detection model and obtain the tool breakage area target detection model. A3. A tool breakage region segmentation module based on a lightweight real-time semantic segmentation network. This module uses the tool breakage region image segmentation dataset constructed in step S2 to train the image segmentation model and obtain the tool breakage region image segmentation model. The tool working state classification dataset is divided into a training set and a validation set in an 8:2 ratio. The training set is used to train the tool working state classification module based on VGG16-D, and the validation set is used to verify the trained tool working state classification model to determine whether the function can be implemented. Step S5: Build a deep learning-based tool condition monitoring inference module, and use this module to learn the model trained in step S4 to infer the tool damage video decomposition dataset obtained in step S3. The structure of the deep learning-based tool condition monitoring and inference module is as follows: B1. A tool working state classification module based on VGG16-D. This module learns the tool working state classification model obtained in A1, performs image classification reasoning on the tool damage video decomposition image dataset in step S3, obtains a classification reasoning result, and extracts the damage classification images as a damage classification dataset. B2. A super-resolution processing module based on deep learning. This module processes the damaged classification dataset mentioned in B1 to improve the image perception quality and thus construct a super-resolution damaged dataset. B3. A YOLO-based tool breakage area localization module. This module learns the tool breakage area target detection model obtained in A2, performs target detection inference on the super-resolution breakage classification dataset in B2, retains the location information of the detected parts and extracts them from the original image, thereby constructing a breakage interest dataset. B4. A tool breakage region segmentation module based on a lightweight real-time semantic segmentation network. This module learns the tool breakage region image segmentation model obtained in A3, performs image segmentation inference on the breakage interest dataset in B3, and maps the obtained segmentation results back to the original image through the location information mentioned in B3 to obtain the tool breakage region segmentation result.
2. The real-time video segmentation method for tool breakage according to claim 1, characterized in that... In step S1, image enhancement and augmentation processes are performed using methods such as adding noise, increasing contrast, rotation, translation, and histogram equalization.
3. The real-time video segmentation method for tool breakage according to claim 1, characterized in that... In step S2, the images in the tool working status classification dataset are compressed to 224×224.
Citation Information
Patent Citations
Cutter surface defect on-machine detection method
CN116539620A
