Ring water inlet area alarm method based on deep neural network

Abstract

The application relates to a deep neural network-based ring water inlet area alarm method, which comprises the following steps: based on a new yo lov7 algorithm in which the original ELAN module is replaced by a swin-transformer module, a target detection model is constructed by adding a DeepSORT tracking algorithm. The application converts the picture format of the spiral discharge pipe related to label processing into a coco data set format, constructs a target detection model based on the new yo lov7 algorithm, trains the coco data set to obtain a weight file, detects the spiral discharge pipe based on the weight file, and obtains a detection result. The DeepSORT tracking algorithm is added to the target detection model, a mask operation is used to cover the area outside the ring water inlet area, only the ring water inlet area is detected, the spiral discharge pipe is tracked, and when the target appears in the area, an alarm is given. The application introduces the attention mechanism and the idea of regional intrusion detection, and realizes the function of giving an alarm when the spiral pipe target detection frame collides with the human-set ring water inlet area.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of visual image processing, in particular to a ring water gap area alarm method based on a deep neural network. BACKGROUND

[0002] Steel casting is a crucial link in the production of steel enterprises. In 1963, submerged entry nozzle protection pouring technology was adopted in Europe, which was rapidly popularized around the world due to its unique advantages. In order to solve the lubrication and heat transfer problem between the continuous casting shell and the mold, researchers developed a protective slag application technology, which filled the gap between the shell and the mold with a liquid slag film formed by the melting of the protective slag, effectively solving the problems of poor lubrication and cracks in open pouring. In 1965, Japan applied powder protective slag pouring on continuous casting machines. In the 1980s, due to the need for hot charging or direct rolling in high-speed continuous casting and high-temperature continuous casting, as well as the performance and adding method of protective slag for special steel continuous casting, higher requirements were put forward.

[0003] In the steel industry, mold protective slag is a very important additive material in continuous casting steel production. The steel liquid surface of the mold rectangular melting cavity needs to be uniformly covered with a layer of mold protective slag to stabilize the pouring operation and improve the quality of the steel billet. At present, leading steel enterprises have begun to explore robot slagging technology, but the existing slagging robot discharge outlet is too close to the ring water gap, which may cause a series of safety problems, and in order to prevent this problem from occurring, workers need to supervise nearby, which leads to high risk and high labor intensity of continuous casting production technology. SUMMARY

[0004] Therefore, the present application provides a ring water gap area alarm method based on a deep neural network to overcome the problem of dangerous contact between the discharge outlet and the ring water gap in the prior art.

[0005] To achieve the above purpose, the present application provides a ring water gap area alarm method based on a deep neural network, comprising:

[0006] A coco format spiral discharge pipe data set is made and divided into a training set and a test set according to a predetermined proportion. An industrial camera is used to shoot the spiral discharge pipe area to obtain image information of the spiral discharge pipe, and the obtained image information is used as the spiral discharge pipe data set.

[0007] Further, a target detection model is constructed, and a yolov7 algorithm is used to improve the target detection model. The improvements include introducing an attention mechanism in the detection head, replacing the original ELAN module with a swin-transformer module, replacing the ELAN module in the 56th layer, the 61st layer, the 67th layer and the 73rd layer of the detection head with a swin-transformer module respectively, and performing concat fusion on the 56th layer and the upper layer output in the 66th layer to obtain semantic information. Finally, target detection is performed through the [61, 67, 73] layer. The schematic diagram of the swin-transformer algorithm introduced in the present application is shown in Figure 4 Specifically, first, it is input into the patch partition module for blocking, that is, every 4x4 adjacent pixels are a patch, and it is flattened in the channel direction. After the patch partition layer, the image form is changed from [H, W, 3] to [H / 4, W / 4, 48]; then, 4 layer stacking modules are used to construct feature maps of different sizes, and each layer stacking module reduces the resolution of the input feature map. The layer stacking module 1Stage1 includes 1 linear embedding and 2 swin-transformer modules, and the layer stacking module 2 to the layer stacking module 4 each include 1 patch merging layer and different numbers of swin-transformer modules. There are two structures of the rotation conversion block, and they are used in pairs. One structure uses a W-MSA (Window Multi-head Self-Attention) structure, and the other uses a SW-MSA (Shifted Window Multi-head Self-Attention) structure.

[0008] Further, the yolov7 algorithm is set to single type detection, and the target detection model is trained after the setting to obtain a weight file.

[0009] Further, the target detection result of the spiral discharge pipe is obtained by target detection of the spiral discharge pipe according to the weight file.

[0010] Further, a video related to the spiral discharge pipe is introduced, and a DeepSORT tracking algorithm module is added on the basis of the yolov7 algorithm training. The weight path and the path of the test video are modified to train the spiral discharge pipe related video, and the Kalman filter algorithm is used to predict the state of the detection frame in the next frame. The formula of the Kalman filter algorithm is as follows:

[0011] Prediction:

[0012]

[0013] P t - = FPt-1 F T +Q

[0014] Wherein in the priori estimation, the predicted position state is represented, containing position, velocity and other information, F is the state transition matrix, describing how the previous frame affects the frame, ut-1 is the control amount, which can be considered as acceleration, B is the control matrix, indicating how the control ut-1 acts on the current state, P- is the predicted covariance matrix of the current frame, which is described as the change relationship, Q represents noise, which obeys normal distribution.

[0015] Update:

[0016] K t =P t - H T (HP t - H T +R) -1

[0017]

[0018] P t =(I-K t H)P t -

[0019] In the posteriori estimation, the measurement matrix is represented by artificial setting, R is the observation noise covariance matrix, the observation noise obeys normal distribution, K is the Kalman gain, Z represents the observation value, x is the state estimation of the current frame, which is the final value of the prediction, P is the error covariance matrix of the current frame.

[0020] The state is matched with the detection result of the next frame, then all the Kalman filtering algorithm predicted matrix is taken as the input of the Hungarian algorithm, the linear matching result is obtained, the tracking weight file is obtained, finally the spiral discharge pipe is allocated with corresponding ID based on the tracking weight file obtained by training and real-time tracking is realized.

[0021] Further, the ring water inlet detection area is masked to cover the area outside the ring water inlet. Different ring water inlet areas exist in different application scenarios. For specific scenarios, the ring water inlet area is manually set. First, the camera shooting area is determined, the ring water inlet monitoring area is found in the camera shooting area, and manual setting is performed: for the collection area, the upper left vertex of the monitoring area is hl1 proportion of the picture top and wl1 proportion of the picture left, the upper right vertex of the monitoring area is hl2 proportion of the picture top and wl2 proportion of the picture left, the lower right vertex of the monitoring area is hl3 proportion of the picture top and wl3 proportion of the picture left, and the lower left vertex of the monitoring area is hl4 proportion of the picture top and wl4 proportion of the picture left. That is, other areas outside the above specified area become black areas. Since the RGB value of black is 0, this can be used to make the camera shooting area black according to the mask chart. In this way, yolo only detects the area and the black area, and does not detect the content outside the area. Since the spiral discharge pipe cannot touch the ring water inlet area, the area is slightly expanded when the area is divided, so that the spiral discharge pipe is a certain distance away from the real ring water inlet area to alarm.

[0022] Further, the area where the spiral discharge pipe is located is recorded as a target area and the target area is tracked. When the target area contacts the area where the ring water inlet is located, an alarm is issued to remind. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 The whole flowchart of the present application;

[0024] Figure 2 The detection effect diagram of the improved yolov7 target detection model of the present application;

[0025] Figure 3 The architecture schematic diagram of the improved yolov7 target detection model of the present application;

[0026] Figure 4 The schematic diagram of the swin-transformer algorithm introduced by the present application;

[0027] Figure 5 The flowchart of the improved yolov7+DeepSort target tracking algorithm of the present application;

[0028] Figure 6 The area mask schematic diagram of the present application; DETAILED DESCRIPTION

[0029] In order to make the objects and advantages of the present application clearer, the following further describes the present application with reference to examples; it should be understood that the specific examples described herein are merely intended to explain the present application, and are not intended to limit the present application.

[0030] The preferred embodiments of the present application are described below with reference to the accompanying drawings. Those skilled in the art should understand that these embodiments are merely intended to explain the technical principles of the present application, and are not intended to limit the protection scope of the present application.

[0031] It should be noted that, in the description of the present application, the terms indicating the direction or positional relationship of "upper", "lower", "left", "right", "inner", "outer" and the like are based on the direction or positional relationship shown in the drawings, which is merely for the convenience of description, and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present application.

[0032] In addition, it should also be noted that, in the description of the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connection" should be understood broadly, for example, it can be fixed connection, or detachable connection, or integrally connected; it can be mechanical connection, or electrical connection; it can be directly connected, or indirectly connected through an intermediate medium, or the internal communication of two elements. Those skilled in the art can understand the specific meaning of the above terms in the present application according to the specific circumstances.

[0033] Please refer to Figure 1 The specific implementation steps of the ring water inlet area alarm method based on a deep neural network provided by the embodiment are as follows:

[0034] A coco format spiral discharge pipe dataset is made. An industrial camera is used to shoot the spiral discharge pipe area to obtain image information of the spiral discharge pipe, and the obtained image information is used as the spiral discharge pipe dataset; the spiral discharge pipe dataset is divided into a training set and a test set with a division ratio of 9:1. A polygon annotation tool of labelme is used to annotate different colors for each spiral discharge pipe, and after the annotation is completed, a coco format json annotation file is generated, and the json file is converted into a txt format file.

[0035] Specifically, the target detection model is improved based on the yolov7 algorithm. The improved yolov7 target detection model detection effect diagram is as shown in Figure 2 ; and the architecture schematic diagram of the improved yolov7 target detection model is as shown in Figure 3 .

[0036] The yolov7 network comprises a backbone network and a detection head, the backbone network is used for feature extraction, the detection head comprises two parts, namely neck and prediction; the features extracted by the backbone network are subjected to multi-scale fusion processing by using a top-down and bottom-up connection mode, and are then sent to a detection layer.

[0037] The improvements include introducing an attention mechanism in the detection head, replacing the original ELAN module with a swin-transformer module, replacing the ELAN modules in the 56th layer, the 61st layer, the 67th layer and the 73rd layer of the detection head with swin-transformer modules respectively, and performing concat fusion on the outputs of the 56th layer and the upper layer in the 66th layer to obtain semantic information, and finally performing target detection through the [61, 67, 73] layers, and a schematic diagram of the swin-transformer algorithm introduced in the application is as shown in Figure 4 Specifically, the input is first divided into patches in a patch division module, that is, every 4x4 adjacent pixels are a patch, and the image is flattened in the channel direction, and after the patch division layer, the image form is changed from [H, W, 3] to [H / 4, W / 4, 48]; then, 4 stacked modules are used to construct feature maps of different sizes, and each stacked module reduces the resolution of the input feature map. The stacked module 1 Stage 1 comprises one linear embedding and two swin-transformer modules, and the stacked module 2 to the stacked module 4 each comprise one patch merging layer and different numbers of swin-transformer modules. There are two structures of the rotation conversion block and they are used in pairs, one of which uses a W-MSA (Window Multi-head Self-Attention) structure, and the other uses a SW-MSA (Shifted Window Multi-head Self-Attention) structure.

[0038] Specifically, the yolov7 algorithm is set to single type detection, and the target detection face model is trained after the setting to obtain a weight file; in this embodiment, the number of training iterations is set to 200, the batchsize is set to 16, the weight file is obtained through training, and the experimental results of mAP, precision and recall show the superiority of the algorithm, and the calculation formulas of the evaluation indexes are as follows:

[0039]

[0040] Wherein, AP represents average precision, TP represents true positive, FN represents false negative, FP represents false positive, and TN represents true negative.

[0041] Specifically, the target detection is performed on the spiral discharge pipe based on the weight file to obtain a target detection result, a DeepSort target tracking module code is added based on the improved yolov7, the weight path and the path of the test video are modified according to the detection weight file to train the spiral discharge pipe related video, and a Kalman filtering algorithm is used to predict the state of the detection frame in the next frame. The Kalman filtering algorithm formula is as follows:

[0042] Prediction:

[0043]

[0044] P t - = FP t-1 F T + Q

[0045] In the priori estimation, that is, the prediction part, represents the predicted position state, including position, speed and other information, F is a state transition matrix, describing how the previous frame affects the frame, ut-1 is a control amount, which can be considered as an acceleration, B is a control matrix, representing how ut-1 acts on the current state, P- is the predicted covariance matrix of the current frame, which describes the change relationship, and Q represents noise, which obeys a normal distribution.

[0046] Update:

[0047] K t = P t - H T (HP t - H T + R) -1

[0048]

[0049] P t = (I-K t H)P t -

[0050] In the posteriori estimation, that is, the update part, is a measurement matrix, which is artificially set, R is an observation noise covariance matrix, and the observation noise obeys a normal distribution, represents a Kalman gain, represents an observation value, represents a state estimation of the current frame, is a final value of the prediction, and is an error covariance matrix of the current frame.

[0051] The state is matched with the detection result of the next frame, then all the Kalman filtering algorithm predicted matrices are taken as inputs of the Hungarian algorithm to obtain a linear matching result, a tracking weight file is obtained, and finally the spiral discharge pipe is assigned with a corresponding ID based on the trained tracking weight file and real-time tracking is realized, and the specific process is as followsFigure 5 As shown.

[0052] Specifically, such as Figure 6 As shown, different ring inlet areas exist in different application scenarios. For specific scenarios, the ring inlet area is manually set. First, the camera shooting area is determined, and then the ring inlet monitoring area is found within the camera shooting area and manually set as follows: the ratio of the top left vertex of the monitoring area to the top of the image is hl 1 and the ratio of the top right vertex to the left side of the image is wl 1; the ratio of the top right vertex to the top of the image is hl2 and the ratio of the top right vertex to the left side of the image is wl2; the ratio of the bottom right vertex to the top of the image is hl3 and the ratio of the bottom left vertex to the left side of the image is wl3; and the ratio of the bottom left vertex to the top of the image is hl4 and the ratio of the bottom left vertex to the left side of the image is wl4. That is, areas other than the above-mentioned designated areas become black areas. Since the RGB value of black is 0, this can be used to blacken the area captured by the camera according to the mask image. In this way, YOLO will only detect the area and the black area and will not detect the content outside the area. Since the spiral discharge pipe cannot touch the ring inlet area, the area is slightly enlarged when dividing the area so that the spiral discharge pipe is a certain distance away from the actual ring inlet area to trigger an alarm.

[0053] Specifically, when the target detection frame for detecting the spiral discharge pipe intersects with the manually set frame of the annular water inlet area, it is determined that the spiral discharge pipe has entered the annular water inlet area. At this time, the operation stops and an alarm is issued to remind the staff to remove the spiral discharge pipe.

[0054] In summary, the application provides a deep neural network-based ring water inlet area alarm method. The method comprises the following steps: step S1, making a spiral discharge pipe dataset into a coco format dataset, and dividing the dataset into a training set, a validation set and a test set; step S2, constructing a target detection model, the target detection model of the application is improved based on the yolov7 algorithm, the improvement includes introducing a swin-transformer module in the detection head, and replacing the original ELAN module with a swin-transformer module; step S3, setting the improved algorithm as single type detection, training the target detection model and obtaining a weight file; step S4, performing target detection of the spiral discharge pipe based on the weight file to obtain the target detection result; step S5, adding a DeepSORT tracking algorithm module based on the improved yolov7 algorithm training, first training the spiral discharge pipe related video according to the detection weight obtained by training, predicting the state of the detection frame in the next frame using the Kalman filter algorithm, matching the state with the detection result of the next frame, then taking all the cost matrices predicted by the Kalman filter algorithm as the input of the Hungarian algorithm to obtain the linear matching result, obtaining the tracking weight file, assigning the corresponding ID to the spiral discharge pipe based on the tracking weight file obtained by training and realizing real-time tracking; step S6, performing mask operation on the ring water inlet detection area to cover the area other than the ring water inlet; step S7, performing target tracking on the spiral discharge pipe, and when the target area collides with the ring water inlet area, an alarm is given.

[0055] So far, the technical solutions of the application have been described in combination with the preferred embodiments shown in the drawings, but those skilled in the art can easily understand that the protection scope of the application is obviously not limited to these specific embodiments. Those skilled in the art can make equivalent changes or replacements to related technical features without departing from the principles of the application, and the technical solutions after the changes or replacements will fall within the protection scope of the application.

[0056] The above description is only the preferred embodiments of the application and is not intended to limit the application; for those skilled in the art, the application can have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the application shall be included in the protection scope of the application.

Claims

1. A method for alarming the ring water inlet area based on deep neural networks, characterized in that, include: Step S1: Create a Cocoa format spiral discharge tube dataset and divide the spiral discharge tube dataset into a training set and a test set according to a preset ratio. An industrial camera is used to capture images of the spiral discharge tube area to obtain image information of the spiral discharge tube, and the acquired image information is used as the spiral discharge tube dataset. Step S2: Construct an object detection model and improve it using the YOLOv7 algorithm. Improvements include the introduction of an attention mechanism into the detection head and the replacement of the original ELAN module with the swin-transformer module; Step S3: Set the YOLOv7 algorithm to single-type detection, and train the target detection model to obtain a weight file after setting. Step S4: Perform target detection on the spiral discharge pipe according to the weight file to obtain the target detection result; Step S5: Introduce videos related to the spiral discharge pipe, add the DeepSORT tracking algorithm module based on the YOLOv7 algorithm training, and train the introduced videos according to the weight file to complete fast tracking of the weight file. Step S6: Perform a masking operation on the detection area of ​​the ring water inlet to cover the area outside the ring water inlet; Step S7: The area where the spiral discharge pipe is located is recorded as the target area and the target area is tracked. When the target area comes into contact with the area where the ring water inlet is located, an alarm is issued.

2. The alarm method for the ring water inlet area based on deep neural networks according to claim 1, characterized in that, In step S1, the labelme annotation tool is used to mark each of the spiral discharge tubes and different colors are used to mark the corresponding spiral discharge tubes. When the marking of each spiral discharge tube is completed, a JSON annotation file in Coco format is generated, and each JSON file is converted into a TXT format file in turn.

3. The alarm method for the ring water inlet area based on deep neural networks according to claim 2, characterized in that, The structure of the YOLOv7 algorithm includes a backbone network and a head for feature extraction. The head includes a neck and a prediction. The YOLOv7 algorithm uses the detection head to perform multi-scale fusion processing on the features extracted by the backbone network in a top-down and bottom-up connection manner, and then sends the fused features into the detection layer.

4. The alarm method for the ring water inlet area based on deep neural networks according to claim 3, characterized in that, In step S2, when replacing the ELAN module with the swin-transformer module, the ELAN modules in layers 56, 61, 67 and 73 of the detection head are replaced with the swin-transformer module, and the output of layer 56 and the upper layer are concat-fused in layer 66 to obtain semantic information. Finally, target detection is performed through layers [61, 67, 73].

5. The alarm method for the ring water inlet area based on a deep neural network according to claim 4, characterized in that, In step S3, the process of obtaining the weight file using the swin-transformer module includes: Step S31: Input it into the patch partitioning module to divide it into blocks so that each 4×4 adjacent pixels form a patch, flatten it in the channel direction, and transform the image shape from [H,W,3] to [H / 4,W / 4,48] after patching the partitioning layer; Step S32: Feature maps of different sizes are constructed through four stacked modules. The first stacked module contains a linear embedding and two swin-transformer modules. The second to fourth stacked modules each contain a patch merging layer and a corresponding number of swin-transformer modules. The two structures of the rotation transformation block are used in pairs. One structure uses the W-MSA structure and the other structure uses the SW-MSA structure. Step S33: Set the number of training iterations and batch size, and then perform object detection training to obtain the weight file. Validate the algorithm's performance using experimental results of mAP, precision, and recall. The calculation formulas for each evaluation metric are as follows: Where AP represents average precision, TP represents true positives, FN represents false negatives, FP represents false positives, and TN represents true negatives.

6. The alarm method for the ring water inlet area based on a deep neural network according to claim 5, characterized in that, In step S5, when the DeepSORT tracking algorithm module is added to the YOLOv7 algorithm training, the weight path and the test video path are modified according to the weight file to train the video related to the spiral discharge pipe, and the Kalman filter algorithm is used to predict the state of the detection box in the next frame, wherein the Kalman filter algorithm formula is as follows: In the prior estimation part P t - =FP t-1 F T +Q In the prior estimation part, The predicted position state is represented by F, which is the state transition matrix, describing how the previous frame affects the current frame. ut-1 is the control variable, including acceleration. B is the control matrix, indicating how ut-1 is controlled to act on the current state. P - This is the prediction covariance matrix for the current frame, used to describe the changing relationship. Q represents noise, which follows a normal distribution. In the posterior estimation section: K t =P t - H T (HP t - H T +R) -1 P t =(I-K t H)P t - In the posterior estimation part, H is the measurement matrix, R is the observation noise covariance matrix, the observation noise follows a normal distribution, and K... t Indicates the Kalman gain, z t Represents the observed value. P represents the state estimate of the current frame. t The error covariance matrix of the current frame; The state is matched with the detection result of the next frame, and the matrices predicted by all Kalman filters are used as input to the Hungarian algorithm to obtain a linear matching result and a tracking weight file. Finally, based on the tracking weight file obtained from the training, the spiral discharge pipe is assigned a corresponding ID to achieve real-time tracking.

7. The alarm method for the ring water inlet area based on deep neural networks according to claim 1, characterized in that, In step S6, when determining the detection area of ​​the ring inlet under different application scenarios, the shooting area of ​​the industrial camera is determined, the ring inlet monitoring area is found in the shooting area of ​​the industrial camera, the collection area is extracted from the ring inlet monitoring area, and the area outside the collection area is blacked out according to the mask image. For the collection area, the ratio of its upper left vertex to the top of the image height of the monitoring area is hl 1 and the ratio of its upper right vertex to the left of the image height of the monitoring area is wl 1, the ratio of its upper right vertex to the top of the image height of the monitoring area is hl2 and the ratio of its upper right vertex to the left of the image height of the monitoring area is wl2, the ratio of its lower right vertex to the top of the image height of the monitoring area is hl3 and the ratio of its lower left vertex to the left of the image height of the monitoring area is wl3, and the ratio of its lower left vertex to the top of the image height of the monitoring area is hl4 and the ratio of its lower left vertex to the left of the image height of the monitoring area is wl4.

8. The alarm method for the ring water inlet area based on a deep neural network according to claim 7, characterized in that, In step S7, when the target detection frame of the spiral discharge pipe intersects with the manually set frame of the annular water inlet area, it is determined that the spiral discharge pipe has entered the annular water inlet area. At this time, the work is stopped and an alarm is issued to remind the staff to remove the spiral discharge pipe.

9. The alarm method for the ring water inlet area based on deep neural networks according to claim 1, characterized in that, In step S7, the ratio of the training set to the test set is 9:1.

Images