Intelligent monitoring method and system for coke oven equipment
By collecting RGB images and thermal imaging on coke oven equipment and using recognition models to identify the state of exchange cocks and addition and subtraction cocks, the equipment monitoring problem in high-temperature environments is solved, automated monitoring is achieved, and equipment reliability and efficiency are improved.
Patent Information
- Application Number
- CN202411328607.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-24
- Publication Date
- 2024-11-22
- Estimated Expiration
- 2044-09-24
AI Technical Summary
It is difficult to effectively monitor coke oven equipment in high temperature environments, especially the state of exchange cocks and addition and subtraction cocks, resulting in low equipment reliability and efficiency. The existing technology mainly relies on artificial imaging recognition, which poses errors and safety hazards.
The imaging equipment is used to collect RGB images and thermal imaging, and the pre-trained coke oven device recognition model is used to identify the types and states of exchange cocks and addition and subtraction cocks. The images are processed in combination with the ORB algorithm to obtain rotation and scale unchanged feature points to realize the focus oven. Intelligent monitoring of equipment.
The automatic status monitoring of the coking oven equipment is realized, the reliability and efficiency of the equipment is improved, human error is reduced, and safety is enhanced.
Smart Images

Figure CN118865266B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of coke oven equipment monitoring, and in particular to a coke oven equipment intelligent monitoring method and system. Background Art
[0002] A coke oven is a kiln used to produce coke from coal, and is the main thermal equipment for coking. During the coking process, after the coal is loaded into the carbonization chamber, the coke oven gas is controlled to burn in the combustion chamber and then transfer heat to the carbonization chamber, so that the coal is closed and dry distilled, thereby forming a lump of carbon with various properties based on fixed carbon. Coking byproducts such as coke oven gas, hydrogen, coal tar and crude benzene are all important chemical raw materials or energy products. Modern coke ovens are mainly composed of carbonization chambers, combustion chambers, regenerators, chute areas, furnace tops, flues, etc. The coal in the carbonization chamber is heated to coke under air-tight conditions. A coke oven is usually composed of dozens of carbonization chambers and combustion chambers arranged alternately, separated by refractory materials (such as silica bricks). Each combustion chamber has 20 to 30 vertical fire channels. The preheated gas and air from the regenerator meet and burn at the bottom of the vertical fire channel, providing heat to the carbonization chamber from the side. The regenerator is located at the bottom of the coke oven, and uses high-temperature exhaust gas to preheat the coal gas and air used for heating. The inclined channel area is an inclined channel connecting the regenerator and the combustion chamber. The furnace body above the carbonization chamber and the combustion chamber is called the furnace roof, and its thickness is determined according to the strength of the furnace body and the need to reduce the surface temperature of the furnace roof. There are coal loading holes and riser holes in the furnace roof area leading to the carbonization chamber, which are used to load coal and export the raw coal gas generated during the dry distillation of coal. The exhaust gas exchange switch of the coke oven cooperates with the flue to control the amount of air used for heating the coke oven, control the introduction of lean coal gas and control the amount of exhaust gas discharged. The exhaust gas exchange switch is set between the small flue and the branch flue of the coke oven, and plays the role of switching the airflow. The commonly used lever-type weight-type exhaust gas switch is mainly composed of a cylinder, a transmission mechanism, an exhaust gas weight, a gas weight, an air port cover and a flue connecting pipe. When the lean coal gas is used to heat the coke oven, the air port cover is locked, the exhaust gas weight is lowered and the gas weight is lifted by the transmission mechanism, and the lean coal gas is introduced into the gas heat storage chamber. When the gas weight is lowered and the exhaust gas weight is lifted, the exhaust gas is introduced into the branch flue. The exchange cocks and the addition and subtraction cocks on the paired flues of the coke oven adjust the opening and closing states and the opening degree of the flues to control ventilation. It is an important component of the coke oven control. However, due to the limitations of the thermal environment of the coke oven, in order to ensure the reliability of the equipment, the exchange cocks, addition and subtraction cocks and the exhaust gas exchange switch are purely mechanically set. Since electronic sensors are easily damaged in the coke oven environment, the status of the three is often monitored manually through imaging recognition. Summary of the invention
[0003] In order to solve the above technical problems or at least partially solve the above technical problems, the present invention provides a method and system for intelligent monitoring of coke oven equipment.
[0004] In a first aspect, the present invention provides a coke oven equipment intelligent monitoring method, comprising:
[0005] Pre-collecting reference RGB images of the exchange cock and the addition and subtraction cock by an imaging device, and marking the rotation angles of the exchange cock and the addition and subtraction cock in the reference RGB images of the exchange cock and the addition and subtraction cock by a reference indication annotation in an indication annotation layer;
[0006] During monitoring, RGB images are collected through imaging devices installed at the coke oven exhaust gas exchange switch, exchange cock and addition and subtraction cock. and thermal imaging ;
[0007] The collected RGB image and thermal imaging Input the pre-trained coke oven equipment recognition model for processing, obtain the masks of the exhaust gas exchange switch, exchange cock and addition and subtraction cock of the coke oven equipment, and identify the types of the exchange cock and addition and subtraction cock, and the types of the transmission mechanism and the air port cover in the exhaust gas exchange switch;
[0008] The status of the exhaust gas weight and the gas weight is determined according to the posture of the transmission mechanism; after the exchange valve and the addition and subtraction valve are identified according to their masks, the correspondence between the exchange valve and the addition and subtraction valve in the reference RGB image and the identified exchange valve and addition and subtraction valve is determined; then the reference indication annotation in the reference RGB image is used to obtain the indication annotation of the exchange valve and the addition and subtraction valve in the RGB image during the actual monitoring process.
[0009] Furthermore, based on the ORB algorithm, the arbitrarily rotated and scaled RGB images of the exchange cock and the addition and subtraction cock are processed to obtain the rotation and scale-invariant feature point sets related to the exchange cock and the addition and subtraction cock in the RGB image; target points are respectively taken from the rotation and scale-invariant feature point sets to form different target geometric structures, and the target geometric structures indicate the rotation angles of the exchange cock and the addition and subtraction cock.
[0010] Furthermore, the coke oven equipment recognition model includes: a visual state space encoder, a visual state space feature fusion module and a visual state space decoder; the visual state space encoder extracts the RGB image through four feature processing stages and thermal imaging Features at different levels; RGB images at different levels of the four feature encoding stages of the visual state space encoder and thermal imaging The features are fused through the visual state space feature fusion module respectively; the visual state space decoder analyzes the fused features at each level and the decoded features of the previous visual state space decoder through four feature decoding stages; finally, the decoded features are input to the mask mapping layer to obtain the masks of the exhaust gas exchange switch, exchange valve and addition and subtraction valve; the decoded features are input to the fully connected layer to identify the types of exchange valve and addition and subtraction valve, and the types of postures of the transmission mechanism and air port cover in the exhaust gas exchange switch.
[0011] Furthermore, the first feature encoding stage of the visual state space encoder is implemented by a patch layer and a visual state space connected by the patch layer; each patch layer includes a convolution layer with a convolution kernel size of k×k and a convolution shift step of k and a normalization layer, and the two groups of patch layers respectively convert the dimension RGB image and dimensions are Thermal imaging Convert to patch and , and patch and Provided for two visual state space processing.
[0012] Furthermore, the visual state space includes: layer normalization, patches before layer normalization and They are divided into two branches, one of which is fused with subsequent features through skip chaining, and the other is processed by layer normalization. The result of layer normalization is divided into two branches, one of which is processed by linear layer to form dynamic weights, and the other is processed by linear layer and depth-separable convolution to obtain feature maps. , feature map Through the two-dimensional selective scanning process, the two-dimensional selective scanning first performs scanning expansion, and the feature map Scan forward from the upper left to the lower right horizontally and vertically, and then scan backward from the lower right to the upper left horizontally and vertically to expand the feature map into four sets of flat sequences. , taking four sets of flattened sequences as input, and passing them through three linear layers to obtain :
[0013] , ;
[0014] but, , ,in, is a matrix whose parameters can be learned; is the identity matrix; represents a linear layer, ;
[0015] The state equations for the four flat sequences are:
[0016] ; ,in, is a matrix whose parameters can be learned; is the intermediate state of the corresponding input, which is determined by the previous state of the input;
[0017] The output sequences corresponding to the four groups of flat sequences are obtained through the state equation, and the output sequences are sorted and fused in the same order to achieve scanning fusion;
[0018] The fusion result is processed by layer normalization and then weighted with dynamic weights. The weighted result is processed by a linear layer and then fused with skip chaining, and finally processed by a linear layer again.
[0019] Furthermore, the second to fourth feature encoding stages are implemented by two downsampling and cross-visual state spaces of the corresponding feature encoding stages;
[0020] The cross-vision state space includes:
[0021] Two layers of standardization, the input of each layer before standardization is divided into two branches, one of which is fused with subsequent features through skip chaining, and the other is processed through layer standardization. The results of each layer standardization are divided into two branches, one of which is processed by the linear layer to form dynamic weights, and the other of which is processed by the linear layer and depth-separable convolution to obtain feature maps; the two feature maps obtained by the two paths are set as feature maps ; Feature map Through two-dimensional selective scanning processing: two-dimensional selective scanning first performs scanning expansion, respectively, Scan forward from the upper left to the lower right in the horizontal and vertical directions, and then scan backward from the lower right to the upper left in the horizontal and vertical directions to obtain the feature map. Expand into four sets of flat sequences and ; Take two sets of four flattened sequences as input and pass them through the linear layer to obtain and :
[0022] , ;
[0023] , ;
[0024] but, , ,
[0025] , ,
[0026] in, , is a matrix whose parameters can be learned; is the identity matrix;
[0027] Swapping matrices in the state matrix , ;
[0028] Four flat sequences The state equation is:
[0029] ; ;
[0030] in, is a matrix with learnable parameters, and the corresponding four sets of flat sequences are obtained through the state equation The output sequences are sorted and merged in the same order;
[0031] Four flat sequences The state equation is:
[0032] ; ;
[0033] in, is a matrix with learnable parameters, and the corresponding four sets of flat sequences are obtained through the state equation The output sequences are sorted and merged in the same order;
[0034] The two fusion results are normalized by two layers and weighted with their respective dynamic weights. The weighted results are processed by two linear layers and fused with their respective skip chains, and finally processed by linear layers respectively.
[0035] Furthermore, the visual state space feature fusion module includes: four levels of fusion layers implemented based on the visual state space: fusion layer 1, fusion layer 2, fusion layer 3 and fusion layer 4; the structure of each fusion layer is consistent, including:
[0036] Two linear layers that process different received features respectively, the input features before each linear layer are divided into two branches, one branch is processed by a linear layer branch to form a dynamic weight, and the other branch is input into the linear layer and the depth-separable convolution layer; in the linear layer and the depth-separable convolution layer structure, the output of the linear layer is divided into two branches, one branch is input into the depth-separable convolution layer, and the other branch is processed by a convolution layer to obtain the weight, the output of the depth-separable convolution layer is processed by feature splicing, and the feature splicing splices the two received features in the channel dimension, and the result of the feature splicing is processed by the fused visual state space, and the fused visual state space expands the spliced features by forward scanning and reverse scanning to obtain two sets of spliced scan sequences, then the fused visual state space of each fusion layer takes the two spliced scan sequences as input respectively, and obtains them respectively through the linear layer :
[0037] , ,
[0038] but, , ,
[0039] in, is a matrix whose parameters can be learned; is the identity matrix;
[0040] The state equation of the two spliced scan sequences is:
[0041] ; ;
[0042] in, is a matrix with learnable parameters. The output sequence corresponding to the two concatenated scan sequences is obtained through the state equation. The two output sequences are sorted and fused in the same order.
[0043] The fusion results are split and normalized by two layers respectively, and then weighted with their respective weights. The weighted results are processed by two linear layers and weighted with their respective dynamic weights. The weighted results are concatenated in dimension and finally mapped by linear layer to reduce the channel dimension.
[0044] Furthermore, training data is constructed: RGB images and thermal images are collected by imaging devices arranged at the coke oven exhaust gas exchange switches, exchange plugs and addition and subtraction plugs; the masks of the coke oven exhaust gas exchange switches, exchange plugs and addition and subtraction plugs are used as the first label, and the types of the postures of the transmission mechanism and the air port cover in the exhaust gas exchange switches, and the types of the exchange plugs and addition and subtraction plugs are used as the second label to annotate the collected RGB images and thermal images to obtain training data; and the coke oven equipment recognition model is trained using the training data.
[0045] Furthermore, using training data to train the coke oven equipment identification model includes: taking the IOU loss between the first label and the mask predicted by the coke oven equipment identification model as the first loss function; freezing the fully connected layer parameters of the coke oven equipment identification model, and based on the first loss function, using the gradient descent algorithm to update the parameters of the coke oven equipment identification model except the fully connected layer; until the first loss function is less than a set threshold; freezing the parameters of the coke oven equipment identification model except the fully connected layer; using the cross entropy loss between the second label and the type of exchange valves, addition and subtraction valves predicted by the coke oven equipment identification model, and the type of posture of the transmission mechanism and the air port cover in the exhaust gas exchange switch as the second loss function, and based on the second loss function, using the gradient descent algorithm to update only the parameters of the fully connected layer of the coke oven equipment identification model until the second loss function is less than the set threshold.
[0046] Furthermore, the first three feature decoding stages of the visual state space decoder are implemented by decoding the visual state space and upsampling, and the last feature decoding stage is implemented by decoding the visual state space, a fully connected layer, and a mask mapping layer, wherein the decoded visual state space includes: a convolution layer, a visual state space, and the output of the visual state space is fused with subsequent features through skip chaining on the one hand, and is weighted with the channel attention after layer normalization on the other hand, and is fused with the output of the visual state space after weighting; the output of the visual state space is processed by a convolution layer after normalization, and the output of the convolution layer is divided into two branches, one is processed by a maximum pooling layer, and the other is processed by an average pooling layer, and the outputs of the maximum pooling layer and the average pooling layer are added and processed by a sigmoid activation function to obtain the channel attention.
[0047] In a second aspect, the present invention provides an intelligent monitoring device for coke oven equipment, comprising: at least one processing unit, wherein the processing unit, a storage unit and a collection unit are connected via a bus unit, the storage unit stores a computer program, and when the computer program is executed by the processing unit, the intelligent monitoring method for coke oven equipment as described above is implemented.
[0048] In a third aspect, the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the intelligent monitoring method for coke oven equipment is implemented.
[0049] The above technical solution provided by the embodiment of the present invention has the following advantages compared with the prior art:
[0050] The present application marks the rotation angles of the exchange valve and the addition and subtraction valve in the reference RGB image through reference indication annotation; during monitoring, RGB images and thermal images are collected by imaging devices arranged at the coke oven exhaust gas exchange switch, exchange valve and addition and subtraction valve; the collected RGB images and thermal images are input into the pre-trained coke oven equipment recognition model for processing to obtain the masks of the exhaust gas exchange switch, exchange valve and addition and subtraction valve of the coke oven equipment, and identify the types of the exchange valve and addition and subtraction valve, and the types of the postures of the transmission mechanism and the air port cover in the exhaust gas exchange switch; determine the status of the exhaust gas weight and the gas weight according to the posture of the transmission mechanism; after identifying the exchange valve and the addition and subtraction valve, use the reference indication annotation in the reference RGB image to obtain the indication annotation of the exchange valve and the addition and subtraction valve in the collected RGB image. This application uses the coke oven equipment recognition model to identify the exchange cock and the addition and subtraction cock, and combines the principle of the ORB (Oriented FAST and Rotated BRIEF) algorithm to indicate and mark the exchange cock and the addition and subtraction cock; this application uses the coke oven equipment recognition model to classify the types of the transmission mechanism and the air port cover in the exhaust gas exchange switch, and determines the status of the exhaust gas weight and the gas weight according to the posture of the transmission mechanism, so as to obtain the status of the coke oven exhaust gas exchange switch, exchange cock and addition and subtraction cock. Vision-based coke oven status monitoring is realized.
[0051] In the coke oven equipment recognition model, the first feature encoding stage of the visual state space encoder encodes the RGB image through the visual state space and thermal imaging When using the cross visual state space for long-range context modeling, the exchange matrix , By using the state matrix generated by the other party to construct the state equation and realize the cross-fusion of features, it is possible to effectively extract common linear invariant relationships from RGB images and thermal imaging for better modeling. The visual state space feature fusion module strengthens and fuses the features of each level of the visual state space encoder through the visual state space. The visual state space decoder analyzes the fused features at each level and the decoded features of the previous visual state space decoder through four feature decoding stages, and enhances the perception of the channel by introducing channel attention in the decoded visual state space. The coke oven equipment recognition model of this application adopts the visual state space solution as a whole, supports efficient channel calculation through deeply separable convolutional layers, has high computational efficiency and good real-time performance. BRIEF DESCRIPTION OF THE DRAWINGS
[0052] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0053] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0054] Figure 1 A flow chart of a coke oven equipment intelligent monitoring method provided by an embodiment of the present invention;
[0055] Figure 2 A schematic diagram of a coke oven equipment identification model provided by an embodiment of the present invention;
[0056] Figure 3 A schematic diagram of a visual state space provided by an embodiment of the present invention;
[0057] Figure 4 A schematic diagram of a scanning expansion method provided by an embodiment of the present invention;
[0058] Figure 5 A schematic diagram of a cross-vision state space provided by an embodiment of the present invention;
[0059] Figure 6 A schematic diagram of a fusion layer provided in an embodiment of the present invention;
[0060] Figure 7 A schematic diagram of decoding a visual state space provided by an embodiment of the present invention;
[0061] Figure 8 A schematic diagram of an indication mark provided in an embodiment of the present invention;
[0062] Fig. 9 A schematic diagram of an intelligent monitoring device for coke oven equipment provided in an embodiment of the present invention. DETAILED DESCRIPTION
[0063] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.
[0064] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.
[0065] Example 1
[0066] like Figure 1 As shown, the technology of the present invention realizes an intelligent monitoring method for coke oven equipment. This application mainly monitors the exhaust gas exchange switch, exchange cock and addition and subtraction cock status of the coke oven equipment. It includes:
[0067] The reference RGB images of the exchange cock and the addition and subtraction cock are pre-collected by an imaging device. For the reference RGB images of the exchange cock and the addition and subtraction cock, the rotation angles of the exchange cock and the addition and subtraction cock in the reference RGB image are marked by the reference indication annotation in the indication annotation layer.
[0068] The purpose of constructing reference indication annotations is to construct the indication annotations in the RGB image from the points in the actual acquired RGB image that match the reference indication annotations in the reference RGB image. Figure 8 As shown, Figure 8 The rotation angle of the addition and subtraction cock is marked by a triangular indication mark, and the rotation angle of the exchange cock is marked by a T-shaped indication mark. The principle is: based on the ORB (Oriented FAST and Rotated BRIEF) algorithm, the arbitrarily rotated and scaled RGB images (including reference RGB images) of the exchange cock and the addition and subtraction cock are processed to obtain the rotation and scale invariant feature point set related to the exchange cock and the addition and subtraction cock in the RGB image; target points are respectively taken from the rotation and scale invariant feature point set to form different target geometric structures, and the target geometric structures formed by the target points, such as triangles and T-shapes, can indicate the rotation angle of the exchange cock and the addition and subtraction cock; the RGB image in the actual monitoring process can be obtained by rotating and scaling the reference RGB image relative to the reference RGB image. Since the feature point matching from the rotation and scale invariant feature point set is not affected by rotation and scale transformation, the reference RGB image can be used to refer to the indication mark to obtain the indication mark of the RGB image in the actual monitoring process.
[0069] Although the ORB (Oriented FAST and Rotated BRIEF) algorithm can extract rotation and scale invariant features, it cannot identify the exchange plug and the addition and subtraction plug. Therefore, it is impossible to determine the correspondence between the target geometric structure shape and the exchange plug, and between the target geometric structure shape and the addition and subtraction plug.
[0070] Monitoring of the exhaust gas exchange switch requires: identification and analysis of the transmission mechanism and the posture of the air port cover of the exhaust gas exchange switch, so as to determine the status of the exhaust gas weight and the gas weight according to the posture of the transmission mechanism.
[0071] In view of the recognition requirements of exchange cocks, addition and subtraction cocks and exhaust gas exchange switches, the present application provides a coke oven equipment recognition model. The coke oven equipment recognition model includes: a visual state space encoder, a visual state space feature fusion module and a visual state space decoder.
[0072] Construct training data: collect RGB images and thermal images through imaging devices arranged at the coke oven exhaust gas exchange switch, exchange plug and addition and subtraction plug; use the masks of the coke oven exhaust gas exchange switch, exchange plug and addition and subtraction plug as the first label, use the type of the posture of the transmission mechanism and the air port cover in the exhaust gas exchange switch, and the type of the exchange plug and addition and subtraction plug as the second label, and annotate the collected RGB images and thermal images to obtain training data. Use the training data to train the coke oven equipment recognition model.
[0073] The visual state space encoder extracts RGB images from the training data through four feature processing stages. and thermal imaging Features at different levels. The visual state space encoder consists of one feature encoding stage based on the visual state space and three feature encoding stages based on the cross visual state space.
[0074] Specifically, the first feature encoding stage of the visual state space encoder is implemented by the patch layer and the visual state space connected by the patch layer. Each patch layer includes a convolution layer with a convolution kernel size of k×k and a convolution step size of k and a normalization layer. The two groups of patch layers respectively convert the dimension RGB image and dimensions are Thermal imaging Convert to patch and , and patch and Provided for two visual state space processing.
[0075] like Figure 3 As shown, the visual state space includes: layer normalization, patches before layer normalization and They are divided into two branches, one of which is fused with subsequent features through skip chaining, and the other is processed by layer normalization. The result of layer normalization is divided into two branches, one of which is processed by linear layer to form dynamic weights, and the other is processed by linear layer and depth-separable convolution to obtain feature maps. , feature map Through two-dimensional selective scanning processing, such as Figure 4 As shown, Figure 4 The left side of the figure shows the feature map. The different types of arrows in the feature map indicate the scanning trajectory. The right side shows four sets of flat sequences obtained according to the scanning trajectory. The two-dimensional selective scanning first performs scanning expansion to expand the feature map. Scan forward from the upper left to the lower right horizontally and vertically, and then scan backward from the lower right to the upper left horizontally and vertically to expand the feature map into four sets of flat sequences. , taking four sets of flattened sequences as input, and passing them through three linear layers to obtain :
[0076] , ;
[0077] but, , ,in, is a matrix whose parameters can be learned; is the identity matrix; represents a linear layer, ;
[0078] The state equations for the four flat sequences are:
[0079] ; ,in, is a matrix whose parameters can be learned; is the intermediate state of the corresponding input, which is determined by the previous state of the input;
[0080] The output sequences corresponding to the four groups of flat sequences are obtained through the state equation, and the output sequences are sorted and fused in the same order to achieve scanning fusion;
[0081] The fusion result is processed by layer normalization and then weighted with dynamic weights. The weighted result is processed by a linear layer and then fused with skip chaining, and finally processed by a linear layer again.
[0082] The first feature encoding stage encodes the RGB image through the visual state space and thermal imaging Each patch of is encoded with contextual long-range modeling.
[0083] The second feature encoding stage is implemented by two downsampling and cross visual state spaces of the second feature encoding stage. The output of the two visual state spaces of the first feature encoding stage and The two downsamples are respectively input into the second feature encoding stage, and the two downsamples are respectively and Height and width downsampling, and The channel expansion of Features and , downsampling is achieved through convolutional layers. The two downsampling layers respectively and The corresponding input of the cross-visual state space is provided to the second feature encoding stage.
[0084] The third feature encoding stage and the fourth feature encoding stage are similar to the second feature encoding stage.
[0085] The third feature encoding stage is implemented by two downsampling and the cross visual state space of the third feature encoding stage. The two outputs of the cross visual state space of the second feature encoding stage are and The two downsamples are respectively input into the third feature encoding stage, and the two downsamples are respectively and Height and width downsampling, and The channel expansion of Features and , downsampling is achieved through convolutional layers. The two downsampling layers respectively and The corresponding input of the cross-visual state space is provided to the third feature encoding stage.
[0086] The fourth feature encoding stage is implemented by two downsampling and the cross visual state space of the fourth feature encoding stage. The two outputs of the cross visual state space of the third feature encoding stage and The two downsamples are respectively input into the third feature encoding stage, and the two downsamples are respectively and Height and width downsampling, and The channel expansion of Features and , downsampling is achieved through convolutional layers. The two downsampling layers respectively and The corresponding input provided to the cross-visual state space of the fourth feature encoding stage is obtained after being processed by the cross-visual state space of the fourth feature encoding stage. and .
[0087] like Figure 5 As shown, the cross-visual state space includes:
[0088] Two layers of standardization, the input of each layer before standardization is divided into two branches, one of which is fused with subsequent features through skip chaining, and the other is processed through layer standardization. The results of each layer standardization are divided into two branches, one of which is processed by the linear layer to form dynamic weights, and the other of which is processed by the linear layer and depth-separable convolution to obtain feature maps; the two feature maps obtained by the two paths are set as feature maps . Feature map Through two-dimensional selective scanning processing: two-dimensional selective scanning first performs scanning expansion, respectively, Scan forward from the upper left to the lower right in the horizontal and vertical directions, and then scan backward from the lower right to the upper left in the horizontal and vertical directions to obtain the feature map. Expand into four sets of flat sequences and . Take two sets of four flattened sequences as input and pass them through the linear layer to obtain and :
[0089] , ;
[0090] , ;
[0091] but, , ,
[0092] , ,
[0093] in, , is a matrix whose parameters can be learned; is the identity matrix;
[0094] Swapping matrices in the state matrix , ;
[0095] Four flat sequences The state equation is:
[0096] ; ;
[0097] in, is a matrix with learnable parameters, and the corresponding four sets of flat sequences are obtained through the state equation The output sequences are sorted and fused in the same order.
[0098] Four flat sequences The state equation is:
[0099] ; ;
[0100] in, is a matrix with learnable parameters, and the corresponding four sets of flat sequences are obtained through the state equation The output sequences are sorted and fused in the same order.
[0101] The two fusion results are normalized by two layers and weighted with their respective dynamic weights. The weighted results are processed by two linear layers and fused with their respective skip chains, and finally processed by linear layers respectively.
[0102] When modeling long-range context using the cross-visual state space, the exchange matrix , By using the state matrix generated by the other party to construct the state equation and realize the cross-fusion of features, it can effectively extract the common linear invariant relationship from RGB images and thermal imaging, and achieve better modeling.
[0103] RGB images at different levels of the four feature encoding stages of the visual state space encoder and thermal imaging The features are fused by the visual state space feature fusion module. Specifically, the visual state space feature fusion module includes: four levels of fusion layers based on the visual state space, such as Figure 2 The middle fusion layer 1, fusion layer 2, fusion layer 3 and fusion layer 4.
[0104] The structures of all fusion layers are consistent, such as Figure 6 As shown, including:
[0105] Two linear layers that process different received features respectively. The input features before each linear layer are divided into two branches. One branch is processed by a linear layer branch to form a dynamic weight, and the other branch is input into the linear layer and the depthwise separable convolution layer. In the linear layer and the depthwise separable convolution layer structure, the output of the linear layer is divided into two branches. One branch is input into the depthwise separable convolution layer, and the other branch is processed by a convolution layer to obtain the weight. The output of the depthwise separable convolution layer is processed by feature splicing. The feature splicing splices the two received features in the channel dimension. The result of the feature splicing is processed by the fused visual state space. The fused visual state space expands the spliced features by forward scanning and reverse scanning to obtain two sets of spliced scan sequences. For example, the linear layer and depthwise separable convolution layer processing of fusion layer 1 and get and , feature splicing processing and get ,right Forward and reverse scans yield , ; Linear layer and depth-separable convolutional layer processing of fusion layer 2 and get and , feature splicing processing and get ,right Forward and reverse scans yield , ; Linear layer and depth-separable convolutional layer processing of fusion layer 3 and get and , feature splicing processing and get ,right Forward and reverse scans yield , ; Linear layer and depth-separable convolutional layer processing of fusion layer 4 and get and , feature splicing processing and get ,right Forward and reverse scans yield , Then the fused visual state space of each fusion layer takes the two spliced scan sequences as input and obtains them through the linear layer respectively. :
[0106] , ,
[0107] but, , ,
[0108] in, is a matrix whose parameters can be learned; is the identity matrix;
[0109] The state equation of the two spliced scan sequences is:
[0110] ; ;
[0111] in, is a matrix with learnable parameters. The output sequence corresponding to the two concatenated scan sequences is obtained through the state equation. The two output sequences are sorted in the same order and fused.
[0112] The fusion results are split and normalized by two layers respectively, and then weighted with their respective weights. The weighted results are processed by two linear layers and weighted with their respective dynamic weights. The weighted results are concatenated in dimension and finally mapped by linear layer to reduce the channel dimension.
[0113] The visual state space feature fusion module strengthens and fuses the features of each level of the visual state space encoder through the visual state space.
[0114] The visual state space decoder analyzes the fused features at each level and the decoded features of the previous visual state space decoder through four feature decoding stages. The visual state space decoder contains four feature decoding stages based on the decoded visual state space.
[0115] like Figure 2As shown, the output of the fusion layer 4 is provided to the decoding visual state space of the first feature decoding stage, the output of the decoding visual state space of the first feature decoding stage is spliced with the output of the fusion layer 3 after upsampling, and the splicing result is input to the decoding visual state space of the second feature decoding stage; the output of the decoding visual state space of the second feature decoding stage is spliced with the output of the fusion layer 2 after upsampling, and the splicing result is input to the decoding visual state space of the third feature decoding stage; the output of the decoding visual state space of the third feature decoding stage is spliced with the output of the fusion layer 1 after upsampling, and the splicing result is input to the decoding visual state space of the fourth feature decoding stage; the output of the decoding visual state space of the fourth feature decoding stage obtains the mask of the exhaust gas exchange switch, the exchange valve and the addition and subtraction valve through the mask mapping layer formed by the linear layer, and the decoding features of the decoding visual state space output of the fourth feature decoding stage are analyzed by the fully connected layer to identify the types of the exchange valve and the addition and subtraction valve, and the types of the postures of the transmission mechanism and the air port cover in the exhaust gas exchange switch. Among them, upsampling is realized by dilated convolution.
[0116] like Figure 7 As shown in the figure, decoding the visual state space includes: convolution layer, visual state space, the output of the visual state space is fused with subsequent features through skip chaining on the one hand, and is weighted with the channel attention after layer normalization on the other hand, and is fused with the output of the visual state space after weighting; the output of the visual state space is processed by the convolution layer after normalization, and the output of the convolution layer is divided into two branches, one is processed by the maximum pooling layer, and the other is processed by the average pooling layer. The outputs of the maximum pooling layer and the average pooling layer are added and processed by the sigmoid activation function to obtain the channel attention.
[0117] During monitoring, RGB images are collected through imaging devices installed at the coke oven exhaust gas exchange switch, exchange cock and addition and subtraction cock. and thermal imaging ; The coke oven equipment recognition model analyzes the RGB image and thermal imaging The output of the decoded visual state space of the fourth feature decoding stage obtains the masks of the exhaust gas exchange switch, the exchange valve and the addition and subtraction valve through the mask mapping layer formed by the linear layer. The output of the decoded visual state space of the fourth feature decoding stage is analyzed by the fully connected layer to identify the types of the exchange valve and the addition and subtraction valve, and the types of the postures of the transmission mechanism and the air port cover in the exhaust gas exchange switch.
[0118] In order to make the coke oven equipment recognition model based on the RGB image and thermal imaging Obtain the masks of the exhaust gas exchange switch, exchange cock and addition and subtraction cock of the coke oven equipment, and identify the types of the exchange cock and addition and subtraction cock, and the types of the transmission mechanism and the posture of the air port cover in the exhaust gas exchange switch.
[0119] The IOU loss between the first label and the mask predicted by the coke oven equipment identification model is used as the first loss function; first, the parameters of the fully connected layer are frozen, and based on the first loss function, the parameters of the coke oven equipment identification model except the fully connected layer are updated using the gradient descent algorithm; until the first loss function is less than the set threshold; the parameters of the coke oven equipment identification model except the fully connected layer are frozen; the cross entropy loss between the second label and the types of exchange valves, addition and subtraction valves, and the types of postures of the transmission mechanism and the air port cover in the exhaust gas exchange switch predicted by the coke oven equipment identification model is used as the second loss function, and based on the second loss function, only the parameters of the fully connected layer of the coke oven equipment identification model are updated using the gradient descent algorithm until the second loss function is less than the set threshold.
[0120] The status of the exhaust gas weight and the gas weight is determined according to the posture of the transmission mechanism; after the exchange valve and the addition and subtraction valve are identified according to their masks, the correspondence between the exchange valve and the addition and subtraction valve in the reference RGB image and the identified exchange valve and addition and subtraction valve is determined; then the reference indication annotation in the reference RGB image is used to obtain the indication annotation of the exchange valve and the addition and subtraction valve in the RGB image during the actual monitoring process.
[0121] Example 2
[0122] See also Fig. 9 As shown, an embodiment of the present invention provides an intelligent monitoring device for coke oven equipment, including: at least one processing unit, the processing unit, the storage unit and the collection unit are connected via a bus unit, the storage unit is a computer-readable storage medium, and can be used to store software programs, computer executable programs and modules, such as the software programs, computer executable programs and modules corresponding to an intelligent monitoring method for coke oven equipment in an embodiment of the present invention, which can be used to store data collected by the collection unit. The processing unit implements the above-mentioned intelligent monitoring method for coke oven equipment by running the software programs, computer executable programs and modules stored in the storage unit, including:
[0123] Pre-collecting reference RGB images of the exchange cock and the addition and subtraction cock by an imaging device, and marking the rotation angles of the exchange cock and the addition and subtraction cock in the reference RGB images of the exchange cock and the addition and subtraction cock by a reference indication annotation in an indication annotation layer;
[0124] During monitoring, RGB images are collected through imaging devices installed at the coke oven exhaust gas exchange switch, exchange cock and addition and subtraction cock. and thermal imaging ;
[0125] The collected RGB image and thermal imaging Input the pre-trained coke oven equipment recognition model for processing, obtain the masks of the exhaust gas exchange switch, exchange cock and addition and subtraction cock of the coke oven equipment, and identify the types of the exchange cock and addition and subtraction cock, and the types of the transmission mechanism and the air port cover in the exhaust gas exchange switch;
[0126] The status of the exhaust gas weight and the gas weight is determined according to the posture of the transmission mechanism; after the exchange valve and the addition and subtraction valve are identified according to their masks, the correspondence between the exchange valve and the addition and subtraction valve in the reference RGB image and the identified exchange valve and addition and subtraction valve is determined; then the reference indication annotation in the reference RGB image is used to obtain the indication annotation of the exchange valve and the addition and subtraction valve in the RGB image during the actual monitoring process.
[0127] Of course, the computer program stored in the storage unit of the device for implementing the intelligent monitoring method of coke oven equipment provided by an embodiment of the present invention is not limited to the method operations described above, and can also execute related operations in the intelligent monitoring method of coke oven equipment provided by any embodiment of the present invention.
[0128] Example 3
[0129] An embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program. When the computer program is executed, the coke oven equipment intelligent monitoring method is implemented, including:
[0130] Pre-collecting reference RGB images of the exchange cock and the addition and subtraction cock by an imaging device, and marking the rotation angles of the exchange cock and the addition and subtraction cock in the reference RGB images of the exchange cock and the addition and subtraction cock by a reference indication annotation in an indication annotation layer;
[0131] During monitoring, RGB images are collected through imaging devices installed at the coke oven exhaust gas exchange switch, exchange cock and addition and subtraction cock. and thermal imaging ;
[0132] The collected RGB image and thermal imaging Input the pre-trained coke oven equipment recognition model for processing, obtain the masks of the exhaust gas exchange switch, exchange cock and addition and subtraction cock of the coke oven equipment, and identify the types of the exchange cock and addition and subtraction cock, and the types of the transmission mechanism and the air port cover in the exhaust gas exchange switch;
[0133] The status of the exhaust gas weight and the gas weight is determined according to the posture of the transmission mechanism; after the exchange valve and the addition and subtraction valve are identified according to their masks, the correspondence between the exchange valve and the addition and subtraction valve in the reference RGB image and the identified exchange valve and addition and subtraction valve is determined; then the reference indication annotation in the reference RGB image is used to obtain the indication annotation of the exchange valve and the addition and subtraction valve in the RGB image during the actual monitoring process.
[0134] A computer-readable storage medium provided in an embodiment of the present invention stores a computer program which is not limited to the method operations described above, but can also execute related operations in a coke oven equipment intelligent monitoring method provided in any embodiment of the present invention.
[0135] In the embodiments provided by the present invention, it should be understood that the disclosed structures and methods can be implemented in other ways. For example, the structural embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, structures or units, which can be electrical, mechanical or other forms.
[0136] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0137] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0138] The foregoing is merely a specific embodiment of the present invention, which enables those skilled in the art to understand or implement the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features claimed herein.
Claims
1. A coke oven equipment intelligent monitoring method, characterized in that: include: Pre-collecting reference RGB images of the exchange cock and the addition and subtraction cock by an imaging device, and marking the rotation angles of the exchange cock and the addition and subtraction cock in the reference RGB images of the exchange cock and the addition and subtraction cock by a reference indication annotation in an indication annotation layer; During monitoring, RGB images are collected through imaging devices installed at the coke oven exhaust gas exchange switch, exchange cock and addition and subtraction cock. and thermal imaging ; The collected RGB image and thermal imaging The pre-trained coke oven equipment recognition model is input for processing to obtain the masks of the exhaust gas exchange switch, exchange plug and addition and subtraction plug of the coke oven equipment, and the types of the exchange plug and addition and subtraction plug, and the types of the transmission mechanism and the air port cover in the exhaust gas exchange switch are identified; wherein, the coke oven equipment recognition model includes: a visual state space encoder, a visual state space feature fusion module and a visual state space decoder; the visual state space encoder extracts the RGB image through four feature processing stages and thermal imaging Features at different levels; RGB images at different levels of the four feature encoding stages of the visual state space encoder and thermal imaging The features are fused by the visual state space feature fusion module respectively; the visual state space decoder analyzes the fused features at each level and the decoded features of the previous visual state space decoder through four feature decoding stages; finally, the decoded features are input to the mask mapping layer to obtain the masks of the exhaust gas exchange switch, the exchange cock and the addition and subtraction cock; the decoded features are input to the fully connected layer to identify the types of the exchange cock and the addition and subtraction cock, and the types of the postures of the transmission mechanism and the air port cover in the exhaust gas exchange switch; The status of the exhaust gas weight and the gas weight is determined according to the posture of the transmission mechanism; after the exchange valve and the addition and subtraction valve are identified according to their masks, the correspondence between the exchange valve and the addition and subtraction valve in the reference RGB image and the identified exchange valve and addition and subtraction valve is determined; then the reference indication annotation in the reference RGB image is used to obtain the indication annotation of the exchange valve and the addition and subtraction valve in the RGB image during the actual monitoring process.
2. The coke oven equipment intelligent monitoring method according to claim 1, characterized in that: Processing the arbitrarily rotated and scaled RGB images of the exchange cock and the addition and subtraction cock based on the ORB algorithm, obtaining the rotation and scale invariant feature point set related to the exchange cock and the addition and subtraction cock in the RGB image; Target points are respectively taken from the rotation and scale invariant feature point sets to form different target geometric structures, and the target geometric structures indicate the rotation angles of the exchange tap and the addition and subtraction tap.
3. The coke oven equipment intelligent monitoring method according to claim 1, characterized in that: The first feature encoding stage of the visual state space encoder is implemented by the patch layer and the visual state space connected by the patch layer; each patch layer includes a convolution layer with a convolution kernel size of k×k and a convolution shift step of k and a normalization layer, and the two groups of patch layers respectively convert the dimension RGB image and dimensions are Thermal imaging Convert to patch and , and patch and Provided for two visual state space processing.
4. The intelligent monitoring method for coke oven equipment according to claim 3, characterized in that: The visual state space includes: layer normalization, patches before layer normalization and They are divided into two branches, one of which is fused with subsequent features through skip chaining, and the other is processed by layer normalization. The result of layer normalization is divided into two branches, one of which is processed by linear layer to form dynamic weights, and the other is processed by linear layer and depth-separable convolution to obtain feature maps. , feature map Through the two-dimensional selective scanning process, the two-dimensional selective scanning first performs scanning expansion, and the feature map Scan forward from the upper left to the lower right horizontally and vertically, and then scan backward from the lower right to the upper left horizontally and vertically to expand the feature map into four sets of flat sequences. , taking four sets of flattened sequences as input, and passing them through three linear layers to obtain : , ; but, , ,in, is a matrix whose parameters can be learned; is the identity matrix; represents a linear layer, ; The state equations for the four flat sequences are: ; ,in, is a matrix whose parameters can be learned; is the intermediate state of the corresponding input, which is determined by the previous state of the input; The output sequences corresponding to the four groups of flat sequences are obtained through the state equation, and the output sequences are sorted and fused in the same order to achieve scanning fusion; The fusion result is processed by layer normalization and then weighted with dynamic weights. The weighted result is processed by a linear layer and then fused with skip chaining, and finally processed by a linear layer again.
5. The coke oven equipment intelligent monitoring method according to claim 1, characterized in that: The second to fourth feature encoding stages are implemented by two downsampling and cross visual state spaces of the corresponding feature encoding stages; The cross-vision state space includes: Two layers of standardization, the input of each layer before standardization is divided into two branches, one of which is fused with subsequent features through skip chaining, and the other is processed through layer standardization. The results of each layer standardization are divided into two branches, one of which is processed by the linear layer to form dynamic weights, and the other of which is processed by the linear layer and depth-separable convolution to obtain feature maps; the two feature maps obtained by the two paths are set as feature maps ; Feature map Through two-dimensional selective scanning processing: two-dimensional selective scanning first performs scanning expansion, respectively, Scan forward from the upper left to the lower right in the horizontal and vertical directions, and then scan backward from the lower right to the upper left in the horizontal and vertical directions to obtain the feature map. Expand into four sets of flat sequences and ; Take two sets of four flattened sequences as input and pass them through the linear layer to obtain and : , ; , ; but, , , , , in, , is a matrix whose parameters can be learned; is the identity matrix; Swapping matrices in the state matrix , ; Four flat sequences The state equation is: ; ; in, is a matrix with learnable parameters, and the corresponding four sets of flat sequences are obtained through the state equation The output sequences are sorted and merged in the same order; Four flat sequences The state equation is: ; ; in, is a matrix with learnable parameters, and the corresponding four sets of flat sequences are obtained through the state equation The output sequences are sorted and merged in the same order; The two fusion results are normalized by two layers and weighted with their respective dynamic weights. The weighted results are processed by two linear layers and fused with their respective skip chains, and finally processed by linear layers respectively.
6. The coke oven equipment intelligent monitoring method according to claim 1, characterized in that: The visual state space feature fusion module includes: four levels of fusion layers based on the visual state space: fusion layer 1, fusion layer 2, fusion layer 3 and fusion layer 4; the structure of each fusion layer is consistent, including: Two linear layers that process different received features respectively, the input features before each linear layer are divided into two branches, one branch is processed by a linear layer branch to form a dynamic weight, and the other branch is input into the linear layer and the depth-separable convolution layer; in the linear layer and the depth-separable convolution layer structure, the output of the linear layer is divided into two branches, one branch is input into the depth-separable convolution layer, and the other branch is processed by a convolution layer to obtain the weight, the output of the depth-separable convolution layer is processed by feature splicing, and the feature splicing splices the two received features in the channel dimension, and the result of the feature splicing is processed by the fused visual state space, and the fused visual state space expands the spliced features by forward scanning and reverse scanning to obtain two sets of spliced scan sequences, then the fused visual state space of each fusion layer takes the two spliced scan sequences as input respectively, and obtains them respectively through the linear layer : , , but, , , in, is a matrix whose parameters can be learned; is the identity matrix; The state equation of the two spliced scan sequences is: ; ; in, is a matrix with learnable parameters. The output sequence corresponding to the two concatenated scan sequences is obtained through the state equation. The two output sequences are sorted and fused in the same order. The fusion results are split and normalized by two layers respectively, and then weighted with their respective weights. The weighted results are processed by two linear layers and weighted with their respective dynamic weights. The weighted results are concatenated in dimension and finally mapped by linear layer to reduce the channel dimension.
7. The coke oven equipment intelligent monitoring method according to claim 1, characterized in that: Constructing training data: collecting RGB images and thermal images by imaging equipment arranged at the coke oven exhaust gas exchange switch, exchange plug and addition and subtraction plug; using the mask of the coke oven exhaust gas exchange switch, exchange plug and addition and subtraction plug as the first label, using the type of the posture of the transmission mechanism and the air port cover in the exhaust gas exchange switch, and the type of the exchange plug and addition and subtraction plug as the second label, annotating the collected RGB images and thermal images to obtain training data; using the training data to train the coke oven equipment recognition model.
8. The intelligent monitoring method for coke oven equipment according to claim 7, characterized in that: Using training data to train the coke oven equipment identification model includes: The IOU loss between the first label and the mask predicted by the coke oven equipment identification model is used as the first loss function; the parameters of the fully connected layer of the coke oven equipment identification model are frozen, and based on the first loss function, the parameters of the coke oven equipment identification model except the fully connected layer are updated using the gradient descent algorithm; until the first loss function is less than the set threshold; the parameters of the coke oven equipment identification model except the fully connected layer are frozen; the cross entropy loss between the second label and the types of exchange valves, addition and subtraction valves, and the types of postures of the transmission mechanism and the air port cover in the exhaust gas exchange switch predicted by the coke oven equipment identification model is used as the second loss function, and based on the second loss function, only the parameters of the fully connected layer of the coke oven equipment identification model are updated using the gradient descent algorithm until the second loss function is less than the set threshold.
9. The coke oven equipment intelligent monitoring method according to claim 1, characterized in that: The first three feature decoding stages of the visual state space decoder are realized by decoding the visual state space and upsampling, and the last feature decoding stage is realized by decoding the visual state space, a fully connected layer and a mask mapping layer, wherein the decoding visual state space includes: a convolution layer, a visual state space, and the output of the visual state space is fused with subsequent features through skip chaining on the one hand, and is weighted with the channel attention after layer normalization on the other hand, and is fused with the output of the visual state space after weighting; the output of the visual state space is processed by a convolution layer after normalization, and the output of the convolution layer is divided into two branches, one is processed by a maximum pooling layer, and the other is processed by an average pooling layer, and the outputs of the maximum pooling layer and the average pooling layer are added and processed by a sigmoid activation function to obtain the channel attention.
