An intelligent management system for smart factories

Abstract

The present invention relates to the field of factory management, and provides an intelligent management system for smart factories, including a smoke foreground and background distinguishing module, a smoke detection module, a smoke area thermal recognition module and an alarm management module mounted on an MES platform; the foreground and background distinguishing module distinguishes the smoke foreground and the workshop background of the workshop video image collected by the visible light camera; the smoke detection module generates a smoke recognition frame from the smoke foreground image through a smoke detection model based on an improved YOLOv5, wherein the smoke detection model combines an attention mechanism and adopts an improved loss function combined with an improved frame difference method; the smoke area thermal recognition module generates a smoke thermal map representing the smoke temperature distribution from the smoke recognition frame through a smoke segmentation network based on UNet; the alarm management module displays the smoke thermal map and manages the fire alarm area and level according to the smoke thermal map. The present invention realizes the visual intelligent management of the smoldering stage of fire in smart factories.

Description

Technical Field

[0001] The present invention relates to the field of factory management, and in particular to an intelligent management system for a smart factory. Background Art

[0002] Industry 4.0 is an era of intelligence. It is based on the Internet of Things and big data, and uses artificial intelligence as a carrier for machine judgment, making the production process highly intelligent. Among them, intelligent manufacturing has attracted special attention from manufacturers. It deeply combines modern sensor technology, Internet technology, automation technology and artificial intelligence technology to realize the informatization and intelligence of product design and production, enterprise management and services.

[0003] Among them, the application of artificial intelligence has greatly reduced the cost of management and monitoring in factories. In particular, the combined application of machine learning and computer vision has reduced the need for workshops to install multiple sensors for smoke detection and management.

[0004] At present, for example, a patent document with application number 202211014890.2 discloses a smoke recognition method, a smoke recognition device and a storage medium. However, it does not perform smoke recognition through artificial intelligence methods, and therefore cannot meet the needs of today's Industrial 4.0 era.

[0005] Furthermore, existing image recognition methods for detecting flames or fires cannot identify smoke in the smoldering stage, which is not conducive to early warning and fire fighting.

[0006] Therefore, how to enable the MES (Manufacturing Execution System) platform of a smart factory to identify the smoke and heat in the workshop and the masks worn by personnel based on artificial intelligence technology and thus realize intelligent fire alarm management is a technical problem that needs to be solved at present. Summary of the invention

[0007] To this end, the present invention provides an intelligent management system for smart factories, which preliminarily screens suspected smoke areas according to the dynamic characteristics of smoke, performs secondary detection by improving the smoke detection model of YOLOv5, and distinguishes smoke areas through a segmentation network based on UNet, thereby achieving accurate discrimination of smoke areas and generating accurate workshop smoke heat maps. Through the smoke heat maps, visual intelligent management of the smoldering stage of fire is achieved, which greatly reduces the losses and monitoring costs of workshops.

[0008] To achieve the above-mentioned object, the present invention proposes an intelligent management system for a smart factory, comprising: an MES platform communicating with a visible light camera in a workshop, the MES platform being equipped with a smoke foreground and background distinction module, a smoke detection module, a smoke area thermal recognition module and an alarm management module;

[0009] The smoke foreground and background distinguishing module is used to distinguish the smoke foreground and the workshop background by using an improved frame difference method on the workshop video image collected by the visible light camera, and generate a smoke foreground image;

[0010] The smoke recognition module is connected to the smoke foreground and background distinguishing module, and is used to generate a smoke recognition frame from the smoke foreground image through a smoke detection model based on an improved YOLOv5, wherein the smoke detection model is combined with an attention mechanism and adopts an improved loss function combined with the improved frame difference method;

[0011] The smoke area thermal recognition module is connected to the smoke recognition module, and is used to generate a smoke existing area by using a smoke segmentation network based on UNet for the smoke recognition frame, and generate a smoke thermal map for characterizing smoke temperature distribution by using a smoke concentration model based on gray value for the smoke existing area;

[0012] The alarm management module is connected to the smoke area thermal recognition module to display the smoke thermal map and manage the fire alarm area and level according to the smoke thermal map.

[0013] Furthermore, the improved frame difference method is a five-frame difference accumulation algorithm, and the smoke foreground and background distinction module includes a difference image unit, a weighted accumulation unit and a distinction unit;

[0014] The differential image unit is used to perform differential operations on adjacent frames of the five frames of the workshop video images to generate four differential images;

[0015] The weighted accumulation unit is used to perform weighted accumulation on the four differential images to generate a cumulative differential image;

[0016] The distinguishing unit is used to distinguish the smoke foreground and the workshop background according to the comparison result of the cumulative difference image and the illumination threshold, and generate a smoke foreground image.

[0017] Further, the differentiation unit includes an illumination threshold calculation subunit;

[0018] The illumination threshold calculation subunit is used to calculate the illumination threshold according to a set threshold, an illumination suppression coefficient, and the four differential images.

[0019] In the above scheme, the background is removed by differentiating the continuous sequence frames in the video to conform to the dynamic characteristics of the smoke.

[0020] Furthermore, the smoke recognition module includes a foreground confidence unit, a real frame prediction frame intersection-over-union ratio unit and a loss function calculation unit;

[0021] The foreground confidence unit is used to determine whether the smoke prediction frame is the smoke foreground and generate a first intersection-and-union ratio according to the maximum intersection-and-union ratio between the prediction frame of the smoke foreground generated by the smoke detection model and the real frame;

[0022] The real frame prediction frame intersection-union ratio unit is used to calculate the distance penalty term, shape penalty term and classification confidence between the prediction frame of the smoke foreground and the real frame and generate a second intersection-union ratio;

[0023] The loss function calculation unit is used to calculate a loss function according to the first intersection-and-union ratio and the second intersection-and-union ratio.

[0024] Furthermore, the distance penalty term is calculated and determined based on the coordinate difference between the predicted box and the real box, and the shape value of the predicted box;

[0025] The shape penalty item is determined by calculating the shape difference between the predicted box and the true box.

[0026] In the above scheme, the classification loss, positioning loss and confidence loss of the YOLOv5 model loss function are improved by combining the improved loss function with the foreground image, thereby improving the detection accuracy and efficiency of the model.

[0027] Furthermore, the attention mechanism is set between the two convolutional pooling layers of the smoke detection model, and the attention mechanism includes a channel attention mechanism and a spatial attention mechanism;

[0028] The channel attention mechanism is used to perform weighted processing on the feature map output by the convolutional pooling layer;

[0029] The spatial attention mechanism is used to enable the smoke detection model to focus on the smoke feature area in the smoke foreground.

[0030] Furthermore, the smoke segmentation network includes at least four convolutional layers, all of which adopt LeakyReLU activation function and dynamic convolution blocks, and the number of convolution kernels of the dynamic convolution blocks of the four convolutional layers increases successively.

[0031] In the above scheme, the dynamic convolution block allows the smoke detection model to flexibly select the convolution kernel according to the characteristics of the input data, thereby greatly enhancing the network's expressive ability and enabling it to better extract effective features when processing input data with different characteristics. The attention mechanism enables the model to learn richer image features, thereby improving the model's adaptability and generalization ability to complex scenes in the workshop.

[0032] Furthermore, the smoke density model includes a gray value calculation unit, a color calibration determination unit and a color filling unit;

[0033] The gray value calculation unit is used to convert the three primary colors of the pixel points in the smoke existing area to calculate the corresponding gray value;

[0034] The color calibration determination unit is used to normalize the characteristic grayscale value to generate a grayscale normalization value, and determine the color filling calibration according to the grayscale normalization value;

[0035] The color filling unit is used to determine a color filling area according to the pixel points, and fill the color filling area according to the color filling calibration to generate the smoke heat map.

[0036] Further, the alarm management module includes a fire smoldering situation calculation unit and an alarm level management unit;

[0037] The fire smoldering situation calculation unit is used to obtain a weighted average value based on the color filling calibration maximum value and the maximum value area ratio of a plurality of the smoke thermograms at a first set time interval to determine the fire smoldering degree value and the fire smoldering trend value;

[0038] The alarm level management unit is used to calculate and determine the alarm level according to the fire smoldering degree value and the fire smoldering trend value, and control the fire fighting equipment and alarm equipment in the workshop to operate at the alarm level.

[0039] Furthermore, the alarm management module also includes a fire smoldering area prediction unit and an alarm area management unit;

[0040] The fire smoldering area prediction unit is used to calculate the fire smoldering prediction area by fitting according to the area proportion of the color filling calibration median interval of a plurality of the smoke thermograms at a second set time interval;

[0041] The alarm area management unit is used to determine the alarm area according to the fire smoldering prediction area, and control the fire-fighting equipment and alarm equipment in the alarm area of ​​the workshop to operate at the alarm level.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] 1. Preliminary screening of suspected smoke areas is carried out according to the dynamic characteristics of smoke, and secondary detection is performed by improving the smoke detection model of YOLOv5. The smoke area is distinguished by the UNet-based segmentation network, which achieves accurate identification of the smoke area and generates an accurate workshop smoke heat map. The smoke heat map realizes visual intelligent management of the smoldering stage of the fire, greatly reducing the loss and monitoring cost of the workshop.

[0044] 2. Remove the background by differentiating the continuous sequence frames in the video to match the dynamic characteristics of the smoke.

[0045] 3. By combining the improved loss function with the foreground image, the classification loss, positioning loss and confidence loss of the YOLOv5 model loss function are improved, thereby improving the detection accuracy and efficiency of the model.

[0046] 4. The dynamic convolution block allows the smoke detection model to flexibly select the convolution kernel according to the characteristics of the input data, which greatly enhances the network's expressiveness and enables it to better extract effective features when processing input data with different characteristics. The attention mechanism enables the model to learn richer image features, improving the model's adaptability and generalization ability to complex scenes in the workshop. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] Figure 1 This is a schematic diagram of the structure of an intelligent management system for a smart factory according to an embodiment of the present invention;

[0048] Figure 2 This is a schematic diagram of the identification process of generating a smoke heat map for an intelligent management system for a smart factory according to an embodiment of the present invention;

[0049] Figure 3 A schematic diagram of the structure of a backbone network of a smoke detection model based on an improved YOLOv5 for an intelligent management system for a smart factory according to an embodiment of the present invention;

[0050] Figure 4 It is a structural diagram of a smoke segmentation network based on UNet for an intelligent management system of a smart factory according to an embodiment of the present invention;

[0051] Figure 5 This is a schematic diagram of a smoke heat map generation process for an intelligent management system for a smart factory according to an embodiment of the present invention. DETAILED DESCRIPTION

[0052] In order to make the objects and advantages of the present invention more clearly understood, the present invention is further described below in conjunction with embodiments; it should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0053] The preferred embodiments of the present invention are described below with reference to the accompanying drawings. It should be understood by those skilled in the art that these embodiments are only used to explain the technical principles of the present invention and are not intended to limit the protection scope of the present invention.

[0054] It should be noted that, in the description of the present invention, terms such as "up", "down", "left", "right", "inside" and "outside" indicating directions or positional relationships are based on the directions or positional relationships shown in the drawings. This is merely for the convenience of description and does not indicate or imply that the device or element must have a specific orientation, be constructed and operated in a specific orientation. Therefore, it cannot be understood as a limitation on the present invention.

[0055] In addition, it should be noted that in the description of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection, or it can be indirectly connected through an intermediate medium, or it can be the internal communication of two components. For those skilled in the art, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0056] like Figures 1 to 5 As shown, the present invention provides an intelligent management system for smart factories, which preliminarily screens suspected smoke areas according to the dynamic characteristics of smoke, performs secondary detection by improving the smoke detection model of YOLOv5, and distinguishes smoke areas through a segmentation network based on UNet, thereby achieving accurate discrimination of smoke areas, and then generating an accurate workshop smoke heat map, and using the foreground image of the workshop video image to identify the masks worn by personnel through the residual network, thereby improving the model efficiency through model reuse, and realizing intelligent alarm management of early fires in smart factories.

[0057] Embodiment 1:

[0058] like Figures 1 to 5 As shown, this embodiment proposes an intelligent management system for a smart factory, wherein the MES platform for fire safety management of the smart factory communicates with the visible light camera of its workshop, including a smoke foreground and background differentiation module, a smoke detection module, a smoke area thermal recognition module and an alarm management module mounted on the MES platform;

[0059] The smoke foreground and background distinguishing module is used to distinguish the smoke foreground and the workshop background of the workshop video image collected by the visible light camera through the improved frame difference method, and generate a smoke foreground image; the smoke recognition module is connected to the smoke foreground and background distinguishing module, and is used to generate a smoke recognition frame from the smoke foreground image through a smoke detection model based on the improved YOLOv5, wherein the smoke detection model is combined with an attention mechanism and adopts an improved loss function combined with the improved frame difference method; the smoke area thermal recognition module is connected to the smoke recognition module, and is used to generate a smoke existence area from the smoke recognition frame through a smoke segmentation network based on UNet, and generate a smoke thermal map for characterizing the smoke temperature distribution from the smoke existence area through a smoke concentration model based on gray value; the alarm management module is connected to the smoke area thermal recognition module, and is used to display the smoke thermal map, and manage the fire alarm area and level according to the smoke thermal map.

[0060] It should be noted that in order to adapt to the Industry 4.0 standard, most smart factories have set up an MES (Manufacturing Execution System) platform, which is a computerized system used to monitor and guide the factory production process. It is located between the planning layer and the control layer of the production enterprise to connect the upper-level planning management system with the underlying industrial control. The MES platform collects data from the production site in real time, schedules, executes, tracks and provides feedback on the production plan, thereby achieving comprehensive optimization and management of the production process. Its integrated fire alarm management system is used to ensure the safe progress of production tasks. Therefore, it is necessary to improve the safety of the factory and ensure the smooth completion of production tasks.

[0061] It should be noted that the YOLOv5 model has high real-time performance, but its accuracy in smoke detection still needs to be improved, so it needs to be improved. The specific architecture after improvement is as follows: Figure 3 shown.

[0062] Furthermore, the MES platform is also equipped with a personnel mask wearing identification module, which is connected to the smoke foreground and background differentiation module, and is used to generate personnel mask wearing information from the smoke foreground image through a mask recognition model based on a residual network; the alarm management module is connected to the personnel mask wearing identification module, and is used to determine whether to issue a personnel mask wearing alarm based on the smoke heat map and the safety equipment wearing information.

[0063] Furthermore, if Figure 2 As shown, the improved frame difference method is a five-frame difference accumulation algorithm, and the smoke foreground and background distinction module includes a difference image unit, a weighted accumulation unit and a distinction unit;

[0064] The differential image unit is used to perform differential operations on adjacent frames of the five frames of the workshop video image to generate four differential images; the weighted accumulation unit is used to perform weighted accumulation on the four differential images to generate a cumulative differential image; the distinguishing unit is used to distinguish the smoke foreground and the workshop background based on the comparison result of the cumulative differential image and the illumination threshold, and generate a smoke foreground image.

[0065] It can be understood that the frame difference method can effectively identify smoke moving targets and suppress background changes. The conventional frame difference method is a three-frame difference method. After testing, due to the slow flow of smoke in the factory workshop, the recognition accuracy of its motion changes at the 0.1 second level is not high. Therefore, this embodiment can improve the extraction and fusion accuracy of slow motion through the five-frame difference operation, which can be reflected in the recall rate (FR) and false positive rate (FN) of the smoke detection model based on the improved YOLOv5.

[0066] Specifically, the workshop video image is used to generate the images of the n+2th frame, the n+1th frame, the nth frame, the n-1th frame and the n-2th frame according to the video sequence. The grayscale values ​​of the pixels of the five frames are differentially calculated, which is expressed as:

[0067] D n+2 =|f n+2 -f n+1 |, D n+1 =|f n+1 -f n |, D n =|f n -f n-1 |, D n-1 =|f n-1 -f n-2 |

[0068] In the formula, f n-2 、f n-1 、f n 、f n+1 、f n+2 Denote the images of the n-2th frame, the n-1th frame, the nth frame, the n+1th frame, and the n+2th frame respectively, D n-1 , D n , D n+1 , D n+2 Represents four differential images obtained by performing differential operations on the grayscale values ​​of pixels of five frames of images.

[0069] The four differential images are weighted and accumulated to generate a cumulative differential image, which is expressed as:

[0070] D=k 1 D n-1 ∩k 2 D n ∩k 3 D n+1 ∩k 4 D n+2

[0071] k 1 +k 2 +k 3 +k 4 =1

[0072] In the formula, k 1 , k 2 , k 3 , k 4 Represents four weight coefficients, the sum of which is 1, so that the contribution of each difference image can be correctly reflected on a standardized scale, D n-1 , D n , D n+1 , D n+2represents four differential images, and D represents the cumulative differential image. Preferably, the weight coefficient k 1 , k 2 , k 3 , k 4 They are 0.53, 0.23, 0.12, and 0.12 respectively, so that the influence of the most recently acquired differential image is maximized and the algorithm has higher sensitivity.

[0073] Before the cumulative difference image is judged, a connectivity analysis is performed to identify the coherence between adjacent pixels to ensure that all pixels belonging to the same object are marked in the same area. This is a conventional process and will not be described in detail here.

[0074] Furthermore, the differentiation unit includes an illumination threshold calculation subunit; the illumination threshold calculation subunit is used to calculate the illumination threshold according to a set threshold, an illumination suppression coefficient, and the four differential images.

[0075] It is understandable that if the illumination threshold is a fixed value, the noise in the differential image cannot be suppressed if the value is too small. If the value is too large, some information of the smoke and personnel targets in the differential image will be covered up and holes will appear in the detection results. Therefore, in order to adapt to the changes in light in the workshop, it is dynamically valued.

[0076] Specifically, according to the comparison result of the accumulated difference image after the connectivity analysis and the illumination threshold calculated according to the set threshold, the illumination suppression coefficient, and the four difference images, the smoke foreground and the workshop background are distinguished to generate the smoke foreground image, which is expressed as:

[0077]

[0078] Where Q is the smoke foreground image, D is the cumulative difference image, is the illumination threshold, wherein T is the set threshold, which is used to ensure that the illumination threshold is not too small, preferably 0.6, λ is the illumination suppression coefficient, which is used to amplify the proportion of the differential image to the total number of pixels, preferably 1.3 to 1.5, A is the entire frame image, and D n-1 , D n , D n+1 , D n+2 Represents four difference images, N A is the total number of pixels in the area to be detected. The binary image generated is as follows Figure 5 (b) Figure 5 (a) is the original input workshop video image.

[0079] In the above scheme, the background is removed by differentiating the continuous sequence frames in the video to conform to the dynamic characteristics of the smoke.

[0080] Furthermore, if Figure 2 As shown, the smoke recognition module includes a foreground confidence unit, a real frame prediction frame intersection-over-union unit and a loss function calculation unit;

[0081] The foreground confidence unit is used to determine whether the smoke prediction frame is the smoke foreground and generate a first intersection-and-union ratio based on the maximum intersection-and-union ratio between the predicted frame of the smoke foreground generated by the smoke detection model and the real frame; the real frame prediction frame intersection-and-union ratio unit is used to calculate the distance penalty term, shape penalty term and classification confidence between the predicted frame of the smoke foreground and the real frame and generate a second intersection-and-union ratio; the loss function calculation unit is used to calculate the loss function based on the first intersection-and-union ratio and the second intersection-and-union ratio.

[0082] It should be noted that YOLOv5 uses the output cell to predict the location information of the target, where each cell predicts 5 prior frames, and each prior frame predicts 20 values, 15 of which are smoke and background classification categories, and the remaining 5 are coordinates x, y, w, h, t and confidence. When the confidence exceeds the confidence threshold, the prior frame predicts and frames the target. YOLOv5 quotes the Anchor idea in Faster-RCNN to predict the relative offset value of the center point of the bounding box relative to the upper left corner of the corresponding cell, so as to predict and generate a target box with a certain length and width ratio. In the bounding box calculation, it uses the sigmoid function to process the offset, so as to constrain the center point of the bounding box in the cell. In this process, the loss function represents the difference between the actual output and the expected output, which is used to measure the quality of the model prediction. The smaller the value, the better the training effect of the model. The YOLOv5 model has multiple tasks of classification and location prediction, so the expression of the improved loss function that integrates multiple tasks is set as:

[0083]

[0084] Where loss is the loss function, W and H are the width and height of the feature map, A represents the number of prediction boxes corresponding to each cell, preferably 3, and λ 1 , 2 , 3 , 4 are weights of multiple loss functions, 1 MaxIOU＜Thresh λ 1 (b ijk ) 2 is the first intersection-and-union ratio, where 1 MaxIOU＜Thresh Ensure that the maximum intersection-over-union ratio MAXIOU between the predicted box and the true box is less than the intersection-over-union ratio threshold when the predicted box b ijk Determine it as smoke foreground and accumulate variance error. is the second intersection-over-union ratio, where Ensure that the predicted box and the true box match and accumulate the weighted variance errors of the first to third sub-intersections to obtain the average value. Δ is the distance penalty term, Ω is the shape penalty term, C is the number of categories, and the number of prior box prediction values ​​mentioned above is 15. Represents the classification confidence of YOLOv5, where truth c is the classification of the real box, b c ijk Classification of the predicted box for the smoke foreground.

[0085] Furthermore, the distance penalty item is calculated and determined based on the coordinate difference between the predicted box and the real box, and the shape value of the predicted box; and the shape penalty item is calculated and determined based on the shape difference between the predicted box and the real box.

[0086] Specifically, the distance penalty term is calculated by the following formula:

[0087]

[0088]

[0089] In the formula, Δ is the distance penalty term, e is a natural constant, and P t is the coordinate distance penalty value, P x is the horizontal axis distance penalty term, P y is the vertical coordinate distance penalty term, x, y are the horizontal coordinate and vertical coordinate respectively, w, h are the width and height respectively, p rx and p ry are the horizontal and vertical coordinates of the center point of the real frame, respectively, fx and p fy is the horizontal and vertical coordinates of the prediction box, f w 、f h is the width and height of the prediction box.

[0090] Specifically, the shape penalty term is calculated by the following formula:

[0091]

[0092] Where Ω is the shape penalty term, e is a natural constant, and w t is the shape penalty value, θ is the adjustment coefficient, w w is the width shape penalty value, w h is the height shape penalty value, w rw and w rh are the width and height of the real frame, w fw and w fh are the width and height of the prediction box respectively.

[0093] In the above scheme, the classification loss, positioning loss and confidence loss of the YOLOv5 model loss function are improved by combining the improved loss function with the foreground image, thereby improving the detection accuracy and efficiency of the model.

[0094] Furthermore, if Figure 3 As shown, the attention mechanism is set between the two convolutional pooling layers of the smoke detection model, and the attention mechanism includes a channel attention mechanism and a spatial attention mechanism;

[0095] The channel attention mechanism is used to perform weighted processing on the feature map output by the convolutional pooling layer; the spatial attention mechanism is used to make the smoke detection model focus on the smoke feature area in the smoke foreground.

[0096] Specifically, the attention mechanism is CBAM (Convolutional Block Attention Module), which improves the network's representation ability by modeling spatial and channel attention. CBAM consists of two sub-modules: Figure 3 The Mc channel attention module (Channel Attention Module) and Ms spatial attention module (SpatialAttention Module) shown, the attention mechanism is set after the feature map C3 output by the second layer of convolution.

[0097] The expression of the channel attention mechanism is:

[0098] M c (F) = σ(W 2 (RELU(W 1 (F avg (W 01 (g)+b 1 ))))+W 1 (RELU(W 0 (F max (W 02 (g)+b 2 ))))+b 3 )

[0099] Where M c (F) is the channel attention weight, σ represents the Sigmoid activation function, ReLU represents the ReLU activation function, W 1 , W 2 , W 01 , W 01 , b 1 , b 2 , b 3 represents the learnable weights and learnable parameters, F avgrepresents global average pooling, F max Represents global maximum pooling.

[0100] The expression of the spatial attention mechanism is:

[0101]

[0102] Where M s (F′) is the spatial attention weight, σ represents the Sigmoid activation function, W 1 , W 2 represents the learnable weights, which are shared for the attention mechanism, is the global average pooling of spatial attention, It is the global maximum pooling of spatial attention.

[0103] In the above scheme, the image generated by the improved YOLOv5 model is as follows Figure 5 (c) shows the selection of the smoke area.

[0104] Furthermore, if Figure 4 As shown, the smoke segmentation network includes at least four convolutional layers, all of which adopt LeakyReLU activation function and dynamic convolution blocks, and the number of convolution kernels of the dynamic convolution blocks of the four convolutional layers increases successively.

[0105] Understandably, see Figure 4 Repeatedly increasing the number of convolution kernels of the dynamic convolution module after each convolution layer can improve model performance, obtain stronger feature expression capabilities, and make details and boundary information more obvious. At the same time, the high-dimensional features of multiple channels provide richer information and can better support the processing of steps.

[0106] Specifically, see Figure 4 , Figure 4 The specific architecture of the convolution activation function layer is as follows: a convolution operation with a 5×5 convolution kernel, a stride of 1, and an output channel number of 64; the activation function uses LeakyReLU, which adds nonlinear characteristics and allows a certain amount of negative value transmission, followed by another convolution operation with an output channel number of 128, using the same 1×1 convolution kernel and LeakyReLU activation function.

[0107] Therefore, through two convolutions, the underlying features of the image, including edges and shapes, are extracted layer by layer to ensure rich feature information. The LeakyReLU activation function enables the network to capture more complex feature patterns, avoids the problem of neuron death, and improves the learning ability of the network. Then the dynamic convolution block DynamicConv usually uses 1×1 convolution to generate multiple convolution kernels. The input feature map is convolved according to the generated convolution kernel to adapt to the differences in input features, thereby enhancing the adaptability of the network to features. The maximum pooling operation uses a 2×2 pooling window and a step size of 2 to reduce the spatial resolution of the feature map. By selecting the maximum value in the pooling window, the most significant feature information is retained and the representativeness of the feature is enhanced.

[0108] See also Figure 4 In the encoding stage, the smoke segmentation network has 4 convolution layers, and the number of convolution kernels in the dynamic convolution block of each layer is 64, 128, 256 and 512 respectively. It can be understood that the number of dynamic convolution kernels in the UNet model directly affects the performance, computational complexity and memory consumption of the model. In the encoder stage of UNet, as the image resolution gradually decreases and the feature abstraction level increases, setting a gradually increasing number of dynamic convolution kernels helps the model better capture the global information and features in the image.

[0109] In the above scheme, the dynamic convolution block allows the smoke detection model to flexibly select the convolution kernel according to the characteristics of the input data, thereby greatly enhancing the network's expressive ability and enabling it to better extract effective features when processing input data with different characteristics. The attention mechanism enables the model to learn richer image features, thereby improving the model's adaptability and generalization ability to complex scenes in the workshop.

[0110] Use ffmpeg to split the collected smoke video into 5300 pictures, which are divided into training set, validation set and testing set in a ratio of 6:2:2.

[0111] The smoke image generated by the above model is as follows Figure 5 As shown in the figure, the workshop video image collected by the visible light camera is as follows Figure 5 As shown in (a), the image generated by the smoke person foreground and background distinction module is as follows Figure 5 As shown in (b), the image generated by the smoke detection module is as follows Figure 5 As shown in (c), the image generated by the smoke area thermal recognition module is as follows Figure 5 (d) as shown.

[0112] Furthermore, the smoke density model includes a gray value calculation unit, a color calibration determination unit and a color filling unit;

[0113] The gray value calculation unit is used to convert the three primary colors of the pixel points in the smoke area to calculate the corresponding gray value, and the expression is:

[0114]

[0115] Where V xy is the grayscale value of the pixel (x, y), R, G, and B are the red channel value, green channel value, and blue channel value of the pixel (x, y), respectively.

[0116] The color calibration determination unit is used to normalize the characteristic grayscale value to generate a grayscale normalization value, and the expression is:

[0117]

[0118] Where V x ' y is the grayscale normalized value of the pixel (x, y), V xy is the gray value of the pixel (x, y), V xymin 、V xymax They are the minimum and maximum grayscale values ​​in the smoke area respectively.

[0119] The color fill calibration is determined according to the grayscale normalization value, specifically: the minimum value of the grayscale normalization value of all pixels in the smoke area corresponds to the blue channel value of the color fill calibration, the maximum value corresponds to the red channel value of the color fill calibration, and the difference between the minimum and maximum values ​​of the grayscale normalization value of all pixels in the smoke area corresponds to the green channel area, and then the grayscale normalization values ​​between the maximum and minimum values ​​can be corresponded one-to-one with the color fill calibration.

[0120] The color filling unit is used to determine the color filling area according to the pixel points, and fill the color filling area according to the color filling calibration to generate the smoke heat map, and its effect is as follows: Figure 5 (d) as shown.

[0121] In the above solution, the heat map provides visual and intuitive image support for intelligent alarm management in the early stage of fire.

[0122] Further, the alarm management module includes a fire smoldering situation calculation unit and an alarm level management unit;

[0123] The fire smoldering situation calculation unit is used to obtain a weighted average value based on the color filling calibration maximum value and the maximum area ratio of multiple smoke thermograms at a first set time interval to determine the fire smoldering degree value and the fire smoldering trend value, and the expression is:

[0124]

[0125] Where W is the fire smoldering degree value, C is the fire smoldering degree value, M and N are the weights of the two weighted sums, which have different values ​​for different types of workshops to make the results consistent with the ease of smoke generation in the workshop process. Indicates the maximum value of the color filling calibration of the i-th smoke heat map, It represents the maximum area ratio of the i-th smoke heat map, and I is the total number of smoke heat maps.

[0126] The alarm level management unit is used to calculate and determine the alarm level according to the fire smoldering degree value and the fire smoldering trend value, and control the fire fighting equipment and alarm equipment in the workshop to operate at the alarm level.

[0127] Specifically, the degree difference of the fire smoldering degree value exceeding the degree safety upper limit threshold and the trend difference of the fire smoldering trend value exceeding the trend safety upper limit threshold are calculated, the degree difference and the trend difference are summed and corresponded to the alarm level interval to determine the alarm level.

[0128] Furthermore, the alarm management module also includes a fire smoldering area prediction unit and an alarm area management unit; the fire smoldering area prediction unit is used to calculate the fire smoldering prediction area by fitting the area proportion of the median interval of the color filling calibration of the plurality of smoke thermograms at the second set time interval, and the expression is:

[0129]

[0130] Where F is the fire smoldering prediction area, are the maximum and minimum values ​​of the color filling calibration of the i-th and i-1-th smoke heat maps, respectively, and I is the total number of smoke heat maps.

[0131] The alarm area management unit is used to determine the alarm area according to the fire smoldering prediction area, and control the fire fighting equipment and alarm equipment in the alarm area of ​​the workshop to operate at the alarm level. Specifically, the fire smoldering prediction area is proportionally enlarged to determine the alarm area.

[0132] Preferably, the first set time interval is 2 seconds, and the second set time interval is 5 seconds.

[0133] Furthermore, the MES platform is mounted on a JetsonNano development board with an integrated OpenCV library.

[0134] In a possible implementation, each of the above models is stored in the SD card of the JetsonNano development board for offline video recognition.

[0135] In this embodiment, the suspected smoke area is preliminarily screened according to the dynamic characteristics of smoke, and secondary detection is performed by improving the smoke detection model of YOLOv5. The smoke area is distinguished by the segmentation network based on UNet, and the accurate discrimination of the smoke area is achieved, and then an accurate workshop smoke heat map can be generated. The smoke heat map realizes the visual intelligent management of the fire smoldering stage, which greatly reduces the loss and monitoring cost of the workshop. The background is removed by differentiating the continuous sequence frames in the video to meet the dynamic characteristics of the smoke. The classification loss, positioning loss and confidence loss of the YOLOv5 model loss function are improved by combining the foreground image with the improved loss function, thereby improving the detection accuracy and efficiency of the model. The dynamic convolution block allows the smoke detection model to flexibly select the convolution kernel according to the characteristics of the input data, thereby greatly enhancing the expression ability of the network, and enabling it to better extract effective features when processing input data with different characteristics, and the attention mechanism enables the model to learn richer image features, improving the adaptability and generalization ability of the model to complex scenes in the workshop.

[0136] So far, the technical solutions of the present invention have been described in conjunction with the preferred embodiments shown in the accompanying drawings. However, it is easy for those skilled in the art to understand that the protection scope of the present invention is obviously not limited to these specific embodiments. Without departing from the principle of the present invention, those skilled in the art can make equivalent changes or substitutions to the relevant technical features, and the technical solutions after these changes or substitutions will fall within the protection scope of the present invention.

[0137] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An intelligent management system for a smart factory, characterized in that: include: The MES platform communicates with the visible light camera in the workshop, and the MES platform is equipped with a smoke foreground and background differentiation module, a smoke recognition module, a smoke area thermal recognition module, and an alarm management module; The smoke foreground and background distinguishing module is used to distinguish the smoke foreground and the workshop background by using an improved frame difference method on the workshop video image collected by the visible light camera, and generate a smoke foreground image; The smoke recognition module is connected to the smoke foreground and background distinguishing module, and is used to generate a smoke recognition frame from the smoke foreground image through a smoke detection model based on an improved YOLOv5, wherein the smoke detection model is combined with an attention mechanism and adopts an improved loss function combined with the improved frame difference method; The smoke area thermal recognition module is connected to the smoke recognition module, and is used to generate a smoke existing area by using a smoke segmentation network based on UNet for the smoke recognition frame, and generate a smoke thermal map for characterizing smoke temperature distribution by using a smoke concentration model based on gray value for the smoke existing area; The alarm management module is connected to the smoke area thermal recognition module to display the smoke thermal map and manage the fire alarm area and level according to the smoke thermal map; The expression of the improved loss function is: Where, loss is the improved loss function, W and H are the width and height of the feature map, A represents the number of prediction boxes corresponding to each cell, λ1, λ2, λ3, and λ4 are multiple loss function weights, and 1 MaxIOU＜Thresh λ1(b ijk ) 2 is the first intersection ratio, where 1 MaxIOU＜Thresh Ensure that the maximum intersection-over-union ratio MaxIOU between the predicted box and the true box is less than the intersection-over-union ratio threshold when the predicted box b ijk Determine it as smoke foreground and accumulate variance error. is the second intersection-combination ratio, where Ensure that the predicted box and the real box match and accumulate the weighted variance errors of the first to third sub-intersections to obtain the average value. Δ is the distance penalty term, Ω is the shape penalty term, and C is the number of categories. Represents the classification confidence of YOLOv5, where truth c is the classification of the real box, b c ijk Classification of the predicted box for the smoke foreground.

2. The intelligent management system for a smart factory according to claim 1, characterized in that: The improved frame difference method is a five-frame difference accumulation algorithm, and the smoke foreground and background distinction module includes a difference image unit, a weighted accumulation unit and a distinction unit; The differential image unit is used to perform differential operations on adjacent frames of the five frames of the workshop video images to generate four differential images; The weighted accumulation unit is used to perform weighted accumulation on the four differential images to generate a cumulative differential image; The distinguishing unit is used to distinguish the smoke foreground and the workshop background according to the comparison result of the cumulative difference image and the illumination threshold, and generate a smoke foreground image.

3. The intelligent management system for a smart factory according to claim 2, characterized in that: The distinguishing unit includes an illumination threshold calculation subunit; The illumination threshold calculation subunit is used to calculate the illumination threshold according to a set threshold, an illumination suppression coefficient, and the four differential images.

4. The intelligent management system for a smart factory according to claim 2, characterized in that: The distance penalty item is calculated and determined according to the coordinate difference between the predicted frame and the real frame, and the shape value of the predicted frame; The shape penalty item is determined by calculating the shape difference between the predicted box and the true box.

5. The intelligent management system for a smart factory according to claim 2, characterized in that: The attention mechanism is set between the two convolutional pooling layers of the smoke detection model, and the attention mechanism includes a channel attention mechanism and a spatial attention mechanism; The channel attention mechanism is used to perform weighted processing on the feature map output by the convolutional pooling layer; The spatial attention mechanism is used to enable the smoke detection model to focus on the smoke feature area in the smoke foreground.

6. The intelligent management system for a smart factory according to claim 1, characterized in that: The smoke segmentation network includes at least four convolutional layers, all of which adopt LeakyReLU activation function and dynamic convolution blocks, and the number of convolution kernels of the dynamic convolution blocks of the four convolutional layers increases successively.

7. The intelligent management system for a smart factory according to any one of claims 1 to 6, characterized in that: The smoke density model includes a gray value calculation unit, a color calibration determination unit and a color filling unit; The gray value calculation unit is used to convert the three primary colors of the pixel points in the smoke existing area to calculate the corresponding gray value; The color calibration determination unit is used to normalize the characteristic grayscale value to generate a grayscale normalization value, and determine the color filling calibration according to the grayscale normalization value; The color filling unit is used to determine a color filling area according to the pixel points, and fill the color filling area according to the color filling calibration to generate the smoke heat map.

8. The intelligent management system for a smart factory according to claim 7, characterized in that: The alarm management module includes a fire smoldering situation calculation unit and an alarm level management unit; The fire smoldering situation calculation unit is used to obtain a weighted average value based on the color filling calibration maximum value and the maximum value area ratio of a plurality of the smoke thermograms at a first set time interval to determine the fire smoldering degree value and the fire smoldering trend value; The alarm level management unit is used to calculate and determine the alarm level according to the fire smoldering degree value and the fire smoldering trend value, and control the fire fighting equipment and alarm equipment in the workshop to operate at the alarm level.

9. The intelligent management system for a smart factory according to claim 8, characterized in that: The alarm management module also includes a fire smoldering area prediction unit and an alarm area management unit; The fire smoldering area prediction unit is used to calculate the fire smoldering prediction area by fitting according to the area proportion of the color filling calibration median value interval of a plurality of the smoke heat maps at a second set time interval; The alarm area management unit is used to determine the alarm area according to the fire smoldering prediction area, and control the fire-fighting equipment and alarm equipment in the alarm area of ​​the workshop to operate at the alarm level.

Images