An abnormal state identification method, system, storage medium and electronic device
By building a neural network model for violations and using edge computing technology, abnormal states in the gas pipeline valve chamber can be identified, solving the problem of high costs associated with manual monitoring and achieving intelligent identification and management of abnormal states.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- PIPECHINA SOUTH CHINA CO
- Filing Date
- 2023-02-22
- Publication Date
- 2026-08-04
AI Technical Summary
In existing technologies, the judgment of abnormal conditions in areas such as gas pipeline valve chambers mainly relies on manual monitoring, resulting in high labor costs.
By employing a neural network model for violations and edge computing technology, images are acquired through cameras, and the neural network model is used to identify violations such as smoking, not wearing protective clothing, making phone calls, and not wearing safety belts when working at heights. Combined with fire smoke and perimeter intrusion recognition, intelligent monitoring is achieved.
It enables intelligent monitoring of areas such as gas pipeline valve chambers, reduces labor costs, improves the accuracy of abnormal state identification, and ensures the level of safety management.
Smart Images
Figure CN116206366B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of abnormal state recognition technology, and in particular to an abnormal state recognition method, system, storage medium and electronic device. Background Technology
[0002] Currently, manual monitoring is often used to determine whether there are any abnormalities in areas such as gas pipeline valve chambers and gas pipeline distribution networks, which results in high labor costs. Summary of the Invention
[0003] The technical problem to be solved by the present invention is to provide an abnormal state identification method and system to address the shortcomings of the prior art.
[0004] The technical solution of the abnormal state identification method of the present invention is as follows:
[0005] A neural network model for violations of rules and regulations is constructed. The neural network model for violations of rules and regulations includes an ARN network and an MLP structure arranged in sequence. The ARN network includes five convolutional blocks in series. Each convolutional block includes a convolutional layer, a ReLU activation layer, a pooling layer and a normalization layer arranged in sequence. The MLP structure includes three fully connected blocks in series. Each fully connected block includes a fully connected layer, a ReLU activation layer and a drop layer arranged in sequence.
[0006] Based on the first image set, the violation and illegality neural network model is trained to obtain a violation and illegality recognition model;
[0007] Acquire the first image within the preset area;
[0008] Using the aforementioned violation identification model, the first image is used to identify at least one of the following behaviors: smoking, not wearing protective clothing, making a phone call, and not wearing a safety belt while working at height.
[0009] The technical solution of the abnormal state identification system of the present invention is as follows:
[0010] It includes a model building module, a model training module, a first acquisition module, and a recognition module;
[0011] The model building module is used to: build a violation and violation neural network model, which includes an ARN network and an MLP structure arranged in sequence. The ARN network includes five convolutional blocks in series. Each convolutional block includes a convolutional layer, a ReLU activation layer, a pooling layer and a normalization layer arranged in sequence. The MLP structure includes three fully connected blocks in series. Each fully connected block includes a fully connected layer, a ReLU activation layer and a drop layer arranged in sequence.
[0012] The model training module is used to: train the violation and illegality neural network model based on the first image set to obtain a violation and illegality recognition model;
[0013] The first acquisition module is used to: acquire a first image within a preset area;
[0014] The identification module is used to: use the violation identification model to identify whether the first image contains at least one of the following behaviors: smoking, not wearing protective clothing, making a phone call, and not wearing a safety belt while working at height.
[0015] The present invention provides a storage medium storing instructions that, when read by a computer, cause the computer to execute an abnormal state identification method as described in any of the preceding claims.
[0016] An electronic device according to the present invention includes a processor and the aforementioned storage medium, wherein the processor executes instructions in the storage medium.
[0017] The technical effects of this invention are as follows:
[0018] It can intelligently monitor whether there are any abnormalities in a preset area, effectively reducing labor costs. Attached Figure Description
[0019] Figure 1 This is a flowchart illustrating an abnormal state identification method according to an embodiment of the present invention;
[0020] Figure 2 This is a schematic diagram of an abnormal state identification system according to an embodiment of the present invention. Detailed Implementation
[0021] like Figure 1 As shown, an abnormal state identification method according to an embodiment of the present invention includes the following steps:
[0022] S1. Construct a neural network model for violations. The neural network model for violations includes an ARN network and an MLP structure arranged in sequence. The ARN network includes five convolutional blocks in series. Each convolutional block includes a convolutional layer, a ReLU activation layer, a pooling layer and a normalization layer arranged in sequence. The MLP structure includes three fully connected blocks in series. Each fully connected block includes a fully connected layer, a ReLU activation layer and a drop layer arranged in sequence.
[0023] The violation-of-violations neural network model can be called the Identification-Of-Violations neural network model (IV model for short). The shallow ARN network of the IV model is designed based on CaffeNet and consists of five concatenated convolutional blocks. Each convolutional block includes a convolutional layer, a ReLU activation layer, a pooling layer, and a normalization layer arranged in sequence. That is, each convolutional block is composed of a convolutional layer, a ReLU activation layer, a pooling layer, and a normalization layer in sequence. The ARN network is used to extract internal attribute features of the image. In the five trained convolutional blocks, because the receptive field of the shallow convolutional blocks, namely the first three convolutional blocks, is small, it can often extract pixel-level information, such as color, texture, edges, and corners. The deeper convolutional blocks, namely the fourth and fifth convolutional blocks, have a larger receptive field, and the IV model compresses the image information, obtaining some information about the overall image, that is, semantic information, such as internal attribute features of the image, such as behavior, clothing, and state.
[0024] The extracted semantic information is then fed into the MLP structure of the IV model for recognition. The MLP network of the IV model consists of three fully connected blocks in series. Each fully connected block is composed of a fully connected layer, a ReLU activation layer, and a drop random deactivation layer in sequence. The three fully connected blocks are responsible for converging and reasoning about attribute information from shallow to deep, and providing inference results based on the recognition target. The random deactivation layer controls the model complexity by randomly deactivating a portion of neurons during model training, thereby greatly reducing overfitting.
[0025] S2. Based on the first image set, the violation and illegality neural network model is trained to obtain the violation and illegality recognition model. The specific process of model training is well known to those skilled in the art and will not be described in detail here.
[0026] S3. Acquire the first image within a preset area; whereby the preset area is the gas pipeline valve chamber, but can also be set according to actual conditions. Specifically, the first image can be acquired through a camera or other means.
[0027] S4. Using a violation identification model, identify whether the first image contains at least one of the following behaviors: smoking, not wearing protective clothing, making a phone call, or not wearing a safety belt while working at height.
[0028] Specifically, image data processing and enhancement methods can be used to transform and crop the real-time first image acquired by the camera. The processed image is then fed into a violation identification model to identify at least one of the following behaviors: smoking, not wearing protective clothing, making a phone call, and not wearing a safety belt while working at height. When at least one of these behaviors is detected, an audible and visual alarm is triggered.
[0029] Optionally, the above technical solution also includes:
[0030] S10. Construct a YoLoX-based neural network model. The YoLoX-based neural network model includes: an input terminal, a backbone network, a Neck network with an FPN structure, and a prediction network arranged in sequence. The input terminal is used for image data augmentation. The backbone network includes a Focus network, a Dark2 network, a Dark3 network, a Dark4 network, and a Dark5 network arranged in sequence. The prediction network includes a decoupling head network and an Anchor Free structure arranged in sequence. The decoupling head network includes three decoupling heads arranged in parallel.
[0031] The YoLoX-based neural network model can be called the Anomaly-State-YoLoX Neural Network Model (ASY model for short). The feature extraction structure in the ASY model is mainly designed and optimized based on the YoLoX backbone framework. The ASY model includes: an input network, a backbone network, a Neck network with an FPN structure, and a prediction network arranged in sequence. Specifically:
[0032] 1) On the input side, two image data augmentation methods similar to those used in YoloX were employed during training: Mosaic and Mixup. Mosaic uses random scaling, cropping, and arrangement to stitch images together, improving the model's ability to recognize small and complex targets. MixUp randomly shrinks an image and embeds it into another image to create a new, complex image, which can improve the model's performance in recognizing small targets on limited data. Both data augmentation methods were disabled in the last 15 epochs of training.
[0033] 2) The backbone network of the ASY model includes the Focus network, Dark2 network, Dark3 network, Dark4 network and Dark5 network arranged in sequence. The Focus network selects pixels at different positions and reassembles them before passing them through a standard convolutional layer. Specifically, it includes a padding operation, a standard convolution, batch normalization and activation layer arranged in sequence, which focuses the width and height information of the image into the channel space.
[0034] The Dark2, Dark3, and Dark4 networks share similar network structures, differing only in size. Each consists of one convolutional layer and one CSP layer, used for progressively deeper feature extraction. The CSP layer comprises two standard convolutional layers, a bottleneck layer, a channel stacking operation, and another standard convolutional layer in sequence, reducing computation while extracting features at different scales. The Dark5 network differs from the Dark structures in its size setting, primarily by adding an SPP bottleneck layer between the convolutional and CSP layers. This bottleneck layer performs spatial pyramid-based pooling operations, reducing model complexity and enabling the fusion and extraction of effective information across multiple scales.
[0035] 3) The ASY model's Neck network uses an FPN structure, meaning it employs an FPN for fusion. Specifically, the Neck structure utilizes feature information from different stages and scales within the ASY model's backbone network, fusing and converging information based on a feature pyramid architecture. This involves upsampling high-level feature information to obtain the feature map for prediction. The addition of the Neck network allows the model to selectively acquire feature information at different scales, similar to residual connections, enabling the ASY model to better perform target recognition tasks at various scales.
[0036] 4) The ASY model's prediction network comprises a decoupling head network and an anchor-free structure arranged sequentially. The decoupling head network consists of three parallel decoupling heads, which further subdivide and extract different feature information at different scales across the three neck stages, and then stack and fuse them along the channel dimension (corresponding to the target box category, foreground or background, and the target box coordinates), ultimately obtaining feature maps at three different scales. The addition of the decoupling heads allows the model to extract more precise features and significantly improves the model's convergence speed. The anchor-free structure fuses and stacks the feature maps output by the decoupling heads to obtain a feature vector containing all prediction information across the three scales. Compared to the traditional anchor-based structure, the anchor-free structure reduces the number of parameters by two-thirds.
[0037] S11. Based on the second image set, the YoLoX-based neural network model is trained to obtain a fire smoke recognition model. The specific process of model training is well known to those skilled in the art and will not be described in detail here.
[0038] S12. Obtain a second image within a preset area, which can be obtained through a camera or other means.
[0039] S13. Using a fire and smoke recognition model, identify whether there is fire and smoke in the second image. Specifically, use image data processing and enhancement methods to transform and crop the real-time second image acquired by the camera, and then input it into the fire and smoke recognition model for recognition. When fire and smoke are detected, issue an audible and visual alarm.
[0040] Optionally, the above technical solution also includes:
[0041] S20. Based on the third image set, train the YoLoV5 model to obtain the perimeter intrusion recognition model;
[0042] The YoLoV5 model has a similar structure to the ASY model, both consisting of an input network, a backbone network, a neck network, and a prediction network. The main difference lies in the prediction network's use of an anchor-based structure instead of an anchor-free structure, allowing for better face recognition performance through a larger number of parameters. Furthermore, the YoLoV5 model's neck network employs a complete feature pyramid structure (FPN+PAN) instead of an FPN structure. This is because faces exhibit significant scale variations in images, and a complete feature pyramid structure with both upsampling and downsampling extracts more multi-scale feature information, thus improving face recognition performance. Functionally, the YoLoV5 model differs from the ASY model. It is trained using images containing faces and their labels to identify unauthorized intruders (other than staff members) and trigger an alarm.
[0043] S21. Obtain real-time video within a preset area, specifically through a camera or other means;
[0044] S22. Determine the target frame from the real-time video and use the perimeter intrusion identification model to identify whether there is perimeter intrusion behavior in the real-time video. Specifically, a target frame can be determined every 5 frames in the real-time video, or a target frame can be determined every second, or a key frame can be determined as the target frame. Specifically, video processing and image data processing and enhancement methods are used to transform the target frame in the real-time video acquired by the camera and input it into the perimeter intrusion identification model. When perimeter intrusion behavior is identified, an audible and visual alarm is issued.
[0045] The above steps are completed based on edge computing, specifically based on an integrated edge computing system with local data acquisition, processing, calculation and alarm capabilities. It also uses Docker Swarm technology and fiber optic access to the industrial television monitoring system to transmit the identification and alarm information results to the industrial television monitoring system in real time, thereby realizing the feature labeling and alarm prompts for abnormal events.
[0046] This invention identifies violations such as smoking, not wearing protective clothing, making phone calls, and not wearing safety belts while working at heights within the valve chamber area. It also identifies abnormal conditions such as smoke and fire in the equipment within the valve chamber and monitors perimeter intrusion behaviors such as personnel climbing over walls and unauthorized personnel entering the valve chamber. Alarm information is pushed to the industrial television monitoring system via an edge computing terminal, enabling feature labeling and alarm notifications for abnormal events. This replaces the existing laser beam and fiber optic vibration early warning systems in the valve chamber, applying an upgraded and optimized intelligent video algorithm to the valve chamber scenario, thereby improving the level of valve chamber safety management.
[0047] This invention achieves the following effects in terms of intelligent recognition accuracy:
[0048] 1) The accuracy rate of identifying personnel violations is greater than 90%;
[0049] 2) The accuracy rate of abnormal state identification is greater than 90%;
[0050] 3) Perimeter intrusion detection accuracy is greater than 90%;
[0051] 4) Recognition accuracy is greater than 80% under low light conditions;
[0052] 5) The underreporting rate is less than 10%.
[0053] In the above embodiments, although the steps are numbered S1, S2, etc., they are only specific embodiments given in this application. Those skilled in the art can adjust the execution order of S1, S2, etc. according to the actual situation, which is also within the protection scope of this invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.
[0054] like Figure 2As shown, an abnormal state recognition system 200 according to an embodiment of the present invention includes a model building module 210, a model training module 220, a first acquisition module 230 and a recognition module 240;
[0055] The model building module 210 is used to build a violation neural network model, which includes an ARN network and an MLP structure arranged in sequence. The ARN network includes five convolutional blocks in series. Each convolutional block includes a convolutional layer, a ReLU activation layer, a pooling layer and a normalization layer arranged in sequence. The MLP structure includes three fully connected blocks in series. Each fully connected block includes a fully connected layer, a ReLU activation layer and a drop layer arranged in sequence.
[0056] The model training module 220 is used to: train the violation and illegality neural network model based on the first image set to obtain the violation and illegality recognition model;
[0057] The first acquisition module 230 is used to: acquire a first image within a preset area;
[0058] The identification module 240 is used to: use a violation identification model to identify whether the first image contains at least one of the following behaviors: smoking, not wearing protective clothing, making a phone call, and not wearing a safety belt while working at height.
[0059] Optionally, the above technical solution also includes a second acquisition module;
[0060] Model building module 210 is also used for:
[0061] A YoLoX-based neural network model is constructed. The YoLoX-based neural network model includes: an input terminal, a backbone network, a Neck network with an FPN structure, and a prediction network arranged in sequence. The input terminal is used for image data augmentation. The backbone network includes a Focus network, a Dark2 network, a Dark3 network, a Dark4 network, and a Dark5 network arranged in sequence. The prediction network includes a decoupling head network and an Anchor Free structure arranged in sequence. The decoupling head network includes three decoupling heads arranged in parallel.
[0062] The model training module 220 is also used to: train the YoLoX-based neural network model based on the second image set to obtain a fire smoke recognition model;
[0063] The second acquisition module is used to: acquire a second image within a preset area;
[0064] The recognition module 240 is also used to: use a fire smoke recognition model to identify whether there is fire and smoke in the second image.
[0065] Optionally, the above technical solution also includes a third acquisition module;
[0066] The model training module 220 is also used to: train the YoLoV5 model based on the third image set to obtain a perimeter intrusion recognition model;
[0067] The third acquisition module is used to: acquire real-time video within a preset area;
[0068] The identification module 240 is also used to: determine the target frame from the real-time video and use the perimeter intrusion identification model to identify whether there is perimeter intrusion behavior in the real-time video.
[0069] Optionally, in the above technical solution, the preset area is the gas pipeline valve chamber.
[0070] The parameters and steps for implementing the corresponding functions of each unit module in the abnormal state identification system 200 of the present invention described above can be referred to the parameters and steps in the embodiment of the abnormal state identification method above, and will not be repeated here.
[0071] An embodiment of the present invention provides a storage medium storing instructions, which, when read by a computer, cause the computer to execute any of the above-mentioned abnormal state identification methods.
[0072] An electronic device according to an embodiment of the present invention is characterized in that it includes a processor and a storage medium, wherein the processor executes instructions in the storage medium, and the electronic device may be a computer, a tablet computer, etc.
[0073] Those skilled in the art will know that this invention can be implemented as a system, method, or computer program product.
[0074] Therefore, this disclosure can be implemented in the following forms: it can be entirely hardware, entirely software (including firmware, resident software, microcode, etc.), or a combination of hardware and software, generally referred to herein as a "circuit," "module," or "system." Furthermore, in some embodiments, the invention can also be implemented as a computer program product in one or more computer-readable media, the computer-readable medium containing computer-readable program code.
[0075] Any combination of one or more computer-readable media may be used. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus, or device.
[0076] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.
Claims
1. An abnormal state identification method, characterized in that, include: A neural network model for violations of rules and regulations is constructed. The neural network model for violations of rules and regulations includes an ARN network and an MLP structure arranged in sequence. The ARN network includes five convolutional blocks in series. Each convolutional block includes a convolutional layer, a ReLU activation layer, a pooling layer and a normalization layer arranged in sequence. The MLP structure includes three fully connected blocks in series. Each fully connected block includes a fully connected layer, a ReLU activation layer and a drop layer arranged in sequence. Based on the first image set, the violation and illegality neural network model is trained to obtain a violation and illegality recognition model; Acquire the first image within the preset area; Using the aforementioned violation identification model, the first image is identified to determine whether it contains at least one of the following behaviors: smoking, not wearing protective clothing, making a phone call, and not wearing a safety belt while working at height. Also includes: A YoLoX-based neural network model is constructed, comprising: an input terminal, a backbone network, a Neck network with an FPN structure, and a prediction network arranged in sequence. The input terminal is used for image data augmentation. The backbone network includes a Focus network, a Dark2 network, a Dark3 network, a Dark4 network, and a Dark5 network arranged in sequence. The prediction network includes a decoupling head network and an Anchor Free structure arranged in sequence. The decoupling head network includes three decoupling heads arranged in parallel. The Focus network selects pixels at different locations and reassembles them before passing them through a standard convolutional layer. Specifically, it includes a padding operation, a standard convolution, batch normalization, and an activation layer arranged in sequence, which focuses the width and height information of the image into the channel space. The Dark2, Dark3, and Dark4 networks all consist of one convolutional layer and one CSP layer, used to extract feature information layer by layer. The CSP layer is composed of two standard convolutional layers, one bottleneck layer, one channel stacking operation, and one standard convolutional layer in sequence, which is used to reduce the amount of computation while extracting feature information at different scales. The Dark5 network adds an SPP bottleneck layer between the convolutional layer and the CSP layer to complete the pooling operation based on the spatial pyramid. Based on the second image set, the YoLoX-based neural network model is trained to obtain a fire smoke recognition model; Acquire a second image within the preset area; The presence of fire and smoke in the second image is identified using a fire and smoke recognition model.
2. The abnormal state identification method according to claim 1, characterized in that, Also includes: Based on the third image set, the YoLoV5 model is trained to obtain a perimeter intrusion recognition model; Acquire real-time video within the preset area; The target frame is determined from the real-time video, and the perimeter intrusion identification model is used to identify whether there is perimeter intrusion behavior in the real-time video.
3. An abnormal state identification method according to any one of claims 1 to 2, characterized in that, The preset area is the gas pipeline valve chamber.
4. An abnormal state identification system, characterized in that, It includes a model building module, a model training module, a first acquisition module, and a recognition module; The model building module is used to: build a violation and violation neural network model, which includes an ARN network and an MLP structure arranged in sequence. The ARN network includes five convolutional blocks in series. Each convolutional block includes a convolutional layer, a ReLU activation layer, a pooling layer and a normalization layer arranged in sequence. The MLP structure includes three fully connected blocks in series. Each fully connected block includes a fully connected layer, a ReLU activation layer and a drop layer arranged in sequence. The model training module is used to: train the violation and illegality neural network model based on the first image set to obtain a violation and illegality recognition model; The first acquisition module is used to: acquire a first image within a preset area; The identification module is used to: use the violation identification model to identify whether the first image contains at least one of the following behaviors: smoking, not wearing protective clothing, making a phone call, and not wearing a safety belt while working at height; It also includes a second acquisition module, and the model building module is further used for: A YoLoX-based neural network model is constructed, comprising: an input terminal, a backbone network, a Neck network with an FPN structure, and a prediction network arranged in sequence. The input terminal is used for image data augmentation. The backbone network includes a Focus network, a Dark2 network, a Dark3 network, a Dark4 network, and a Dark5 network arranged in sequence. The prediction network includes a decoupling head network and an Anchor Free structure arranged in sequence. The decoupling head network includes three decoupling heads arranged in parallel. The Focus network selects pixels at different locations and reassembles them before passing them through a standard convolutional layer. Specifically, it includes a padding operation, a standard convolution, batch normalization, and an activation layer arranged in sequence, which focuses the width and height information of the image into the channel space. The Dark2, Dark3, and Dark4 networks all consist of one convolutional layer and one CSP layer, used to extract feature information layer by layer. The CSP layer is composed of two standard convolutional layers, one bottleneck layer, one channel stacking operation, and one standard convolutional layer in sequence, which is used to reduce the amount of computation while extracting feature information at different scales. The Dark5 network adds an SPP bottleneck layer between the convolutional layer and the CSP layer to complete the pooling operation based on the spatial pyramid. The model training module is also used to: train the YoLoX-based neural network model based on the second image set to obtain a fire smoke recognition model; The second acquisition module is used to: acquire a second image within the preset area; The recognition module is also used to: use a fire and smoke recognition model to identify whether there is fire and smoke in the second image.
5. An abnormal state identification system according to claim 4, characterized in that, It also includes a third acquisition module; The model training module is also used to: train the YoLoV5 model based on the third image set to obtain a perimeter intrusion recognition model; The third acquisition module is used to: acquire real-time video within the preset area; The identification module is also used to: determine the target frame from the real-time video, and use the perimeter intrusion identification model to identify whether there is perimeter intrusion behavior in the real-time video.
6. An abnormal state identification system according to any one of claims 4 to 5, characterized in that, The preset area is the gas pipeline valve chamber.
7. A storage medium, characterized in that, The storage medium stores instructions that, when read by a computer, cause the computer to execute an abnormal state identification method as described in any one of claims 1 to 3.
8. An electronic device, characterized in that, It includes a processor and the storage medium of claim 7, wherein the processor executes instructions in the storage medium.