AI fire monitoring system deployed on programmable logic controller and control method

Abstract

The invention discloses an AI fire monitoring system deployed on a programmable logic controller, comprising: a programmable logic controller having at least a dual-core processor and configured with an image acquisition module, the programmable logic controller being configured to deploy an industrial control operating system supporting a virtualization function, processor resources in a programmable logic controller are divided through a virtualization function of an industrial control operating system, and two independent operating system examples are virtualized, wherein one example is a real-time operating system, and the other example is a real-time operating system; one example is a Linux operating system which is used for executing a runtime program of a programmable logic controller to ensure the real-time performance and high reliability of an industrial automation control task, and the other example is a Linux operating system which is used for running an AI reasoning program to execute a fire monitoring task based on image recognition; the image collection module collects image data in real time and sends the image data to the AI reasoning program, and when the reasoning result of the AI reasoning program is a preset type, the real-time operation system executes a preset action program.

Description

Technical Field

[0001] The present invention belongs to the field of industrial control technology, and in particular relates to an AI fire monitoring system and a control method deployed on a programmable logic controller. Background Art

[0002] The AI-based fire monitoring systems currently on the market usually adopt a distributed architecture, deploying AI models on separate inference devices or cloud platforms, and relying on communication between multiple systems to complete fire monitoring, data processing, and response control. The specific process is that the fire monitoring sensor first transmits the data to the AI ​​inference device. After the AI ​​device executes the fire detection algorithm, it passes the inference results to the cloud for data processing or storage. Finally, the cloud sends the instructions to the local PLC device to control the alarm system or automatic fire extinguishing device. There are several significant defects in the multi-level communication system: 1. The cost of multiple sets of equipment is high: the deployment of AI inference devices and cloud platforms requires independent hardware support, which increases the overall cost and complexity of the system. 2. Slow response speed: data needs to be transmitted, processed, and returned multiple times, which increases communication and processing delays, resulting in longer fire response time.

[0003] Invention patent CN202410539887.5 discloses a fire safety monitoring method based on the Internet of Things, wherein the fire safety monitoring system based on the alarm area model is composed of an image recognition module, a sound detection module, a sensor analysis module, a data preprocessing module, a risk assessment module, an early warning generation module, and a monitoring output module; in the present invention, the sub-module design of the image recognition module enables the system to more accurately identify flames, smoke and fire spread, increases the recognition range and depth of fire scenes, and the multi-dimensional analysis of sound detection helps the system capture tiny abnormal sounds, greatly advances the early warning time, and through efficient data preprocessing and risk assessment modules, accurate classification of fire risks is achieved. The real-time monitoring output module based on the MQTT protocol ensures the rapid transmission and accurate feedback of information, which greatly shortens the response time. Summary of the invention

[0004] To solve the above problems, the purpose of the present invention is to provide an AI fire monitoring system and control method deployed on a programmable logic controller. The AI ​​fire monitoring module includes video data input function, data preprocessing function, and AI model reasoning function, and sends the AI ​​reasoning results to the PLC through shared memory to execute alarm or fire extinguishing control during runtime, thereby realizing intelligent fire detection and automatic fire extinguishing.

[0005] The technical solution provided by the present invention is: an AI fire monitoring system deployed on a programmable logic controller, comprising: a programmable logic controller having at least a dual-core processor and configured with an image acquisition module, and the programmable logic controller is configured as follows:

[0006] Deploy an industrial control operating system that supports virtualization functions, divide the processor resources in the programmable logic controller through the virtualization function of the industrial control operating system, and virtualize two independent operating system instances: one instance is a real-time operating system, which is used to execute the runtime program of the programmable logic controller to ensure the real-time performance and high reliability of industrial automation control tasks, and the other instance is a Linux operating system, which is used to run AI reasoning programs to perform fire monitoring tasks based on image recognition;

[0007] The image acquisition module acquires image data in real time and sends it to the AI ​​reasoning program. When the reasoning result of the AI ​​reasoning program is of a preset type, the real-time operating system executes a predetermined action program.

[0008] Preferably, the programmable logic controller is further configured to: divide the multi-core processor resources of the programmable logic controller through a virtualization layer, allocate at least one CPU core to the runtime program, and allocate at least one CPU core to the AI ​​reasoning program.

[0009] Preferably, the programmable logic controller is configured with a specific shared memory, and the runtime program in the real-time operating system can pass the data or status information collected by the image acquisition module to the AI ​​reasoning program in the Linux operating system through the specific shared memory.

[0010] Preferably, the AI ​​reasoning program performs fire detection on the image data and stores the result in the specific shared memory. When the monitoring result indicates that a fire has occurred, the runtime program triggers the alarm system of the programmable logic controller and executes a preset emergency handling procedure.

[0011] Preferably, the AI ​​reasoning program is a lightweight convolutional neural network based on the YOLOv8 model. The YOLOv8 model is trained based on a large amount of fire image data and then deployed to the Linux operating system. When the monitoring system is running, the Linux operating system obtains the image data collected by the image acquisition module, processes it, and passes it to the YOLOv8 model.

[0012] Based on the same concept, the present invention also provides an AI fire monitoring system control method deployed on a programmable logic controller, comprising the following steps:

[0013] The image acquisition module collects image data in real time and reads the system clock cycle. After reaching the specified interval cycle, the program calls the camera data reading function block to read a frame of RGB image and save it to a temporary variable.

[0014] After preprocessing, it is stored in the memory of the programmable logic controller;

[0015] The AI ​​reasoning program determines whether a fire has occurred based on the preprocessed image data;

[0016] When the judgment result is that a fire has occurred, the runtime program in the programmable logic controller triggers the alarm system of the programmable logic controller and executes a preset emergency processing procedure.

[0017] Preferably, the method further comprises:

[0018] The runtime program in the real-time operating system transmits the data or status information collected by the image acquisition module to the AI ​​reasoning program in the Linux operating system through a specific shared memory;

[0019] The AI ​​reasoning program performs fire detection on the image data and stores the result in the specific shared memory;

[0020] Wherein, a specific shared memory is configured in the programmable logic controller.

[0021] Preferably, the method further comprises:

[0022] When the monitoring result indicates that a fire has occurred, the runtime program triggers the alarm system of the programmable logic controller and executes a preset emergency processing procedure.

[0023] Preferably, the method further comprises training the AI ​​reasoning program:

[0024] Before the system is run, a large amount of fire image data is divided into a training set and a test set. The training set is used to train the AI ​​reasoning program, and the test set is used to verify whether the training results meet the predetermined accuracy requirements;

[0025] After the system is running, the image data acquired in real time is used to further optimize and fine-tune the AI ​​reasoning program.

[0026] Based on the same concept, the present invention also provides a readable storage medium, on which a processing program is stored. When the processing program is executed by a processor, the AI ​​fire monitoring system control method deployed on a programmable logic controller as described above is implemented.

[0027] Due to the adoption of the above technical solution, the present invention has the following advantages and positive effects compared with the prior art:

[0028] The present invention deploys a real-time operating system for executing a programmable logic controller runtime program and a Linux operating system for running an AI reasoning program to perform a fire monitoring task based on image recognition on a single programmable logic controller device. The runtime software is responsible for implementing the execution of the logic control program and the interface of the PLCopen standard motion control function block, and runs in a loop according to input sampling, IEC61131-3 program execution, output refresh, and execution response callback functions. The AI ​​reasoning software is responsible for video data input, video data preprocessing, AI model creation, and looping AI model reasoning and reasoning result output. The shared memory is responsible for realizing the information interaction between the PLC runtime software and the AI ​​reasoning software. The integrated deployment of the programmable logic controller runtime and the AI ​​model reasoning is realized without the need for additional external communication or hardware. Moreover, compared with the prior art, the technical solution of this embodiment deploys the AI ​​model on a programmable logic controller closer to the site. After receiving an alarm signal during operation, it immediately triggers a predefined emergency response measure, such as starting a fire protection system or issuing an alarm notification, and responds more quickly. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The specific embodiments of the present invention are further described in detail below with reference to the accompanying drawings, wherein:

[0030] Figure 1 The present invention shows the working process of the AI ​​fire monitoring system deployed on the programmable logic controller during actual operation. DETAILED DESCRIPTION

[0031] The present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. The advantages and features of the present invention will become more apparent from the following description and claims. It should be noted that the drawings are all in a very simplified form and use non-precise ratios, which are only used to conveniently and clearly assist in explaining the purpose of the embodiments of the present invention.

[0032] It should be noted that all directional indications in the embodiments of the present invention (such as up, down, left, right, front, back, etc.) are only used to explain the relative position relationship, movement status, etc. between the components under a certain specific posture (as shown in the accompanying drawings). If the specific posture changes, the directional indication will also change accordingly.

[0033] Embodiment 1

[0034] like Figure 1 As shown, this embodiment provides an AI fire monitoring system deployed on a programmable logic controller, including: a programmable logic controller, the programmable logic controller having at least a dual-core processor and configured with an image acquisition module, the programmable logic controller being configured as:

[0035] Deploy an industrial control operating system that supports virtualization functions, divide the processor resources in the programmable logic controller through the virtualization function of the industrial control operating system, and virtualize two independent operating system instances: one instance is a real-time operating system, which is used to execute the runtime program of the programmable logic controller to ensure the real-time performance and high reliability of industrial automation control tasks, and the other instance is a Linux operating system, which is used to run AI reasoning programs to perform fire monitoring tasks based on image recognition;

[0036] The image acquisition module acquires image data in real time and sends it to the AI ​​reasoning program. When the reasoning result of the AI ​​reasoning program is of a preset type, the real-time operating system executes a predetermined action program.

[0037] The technical solution of this embodiment is to deploy a real-time operating system for executing the programmable logic controller runtime program and a Linux operating system for running the AI ​​reasoning program to perform the image recognition-based fire monitoring task on a single programmable logic controller device. The runtime software is responsible for implementing the execution of the logic control program and the interface of the PLCopen standard motion control function block, and runs in a loop according to input sampling, IEC61131-3 program execution, output refresh and execution response callback function. The AI ​​reasoning software is responsible for video data input, video data preprocessing, AI model creation, and looping AI model reasoning and reasoning result output. The shared memory is responsible for realizing the information interaction between the PLC runtime software and the AI ​​reasoning software. The integrated deployment of the programmable logic controller runtime and the AI ​​model reasoning is realized without the need for additional external communication or hardware. And compared with the prior art, the technical solution of this embodiment is that the AI ​​model is deployed on a programmable logic controller closer to the site. After receiving the alarm signal during operation, the predefined emergency response measures are immediately triggered, such as starting the fire protection system or issuing an alarm notification, and the response is faster.

[0038] The present invention is further described in detail below to make the technical solution of the present invention clearer and more specific.

[0039] First, the hardware deployment architecture. The fire monitoring system of the present invention is deployed on a PLC device with at least a dual-core processor, which is equipped with a camera for image acquisition. The dual-core or multi-core processor provides basic hardware support for the system's multi-tasking processing, ensuring real-time and parallel execution of reasoning tasks.

[0040] Furthermore, virtualized operating system deployment. In PLC devices, industrial control operating systems (such as Intewell) that support virtualization functions are deployed. Through the virtualization function of the operating system, the processor resources in the PLC device are divided and two independent operating system instances are virtualized: one instance is a real-time operating system (RTOS), which is used to execute the runtime program of the PLC to ensure the real-time and high reliability of industrial automation control tasks. The other instance is a Linux operating system that runs an AI reasoning program to perform fire monitoring tasks based on image recognition.

[0041] Furthermore, task allocation and resource isolation. The multi-core processor resources of the PLC device are divided into different operating system instances through the virtualization layer. Specifically: a CPU core is allocated to the PLC runtime program so that it can independently and efficiently handle real-time control tasks. The remaining processor cores are allocated to the AI ​​reasoning program running on Linux to perform complex AI model reasoning.

[0042] Furthermore, the PLC runtime communicates with the AI ​​reasoning program. The two operating system instances communicate through shared memory. The PLC runtime program in the real-time operating system can pass the data or status information of the image acquisition device to the AI ​​reasoning program in the Linux system through shared memory. The AI ​​reasoning program performs fire detection on the image data through the YOLOv8 model and returns the detection results to the PLC runtime program through shared memory. If the AI ​​reasoning program detects a fire, it will trigger the PLC's alarm system and execute the corresponding emergency handling procedures.

[0043] Preferably, the programmable logic controller is further configured to: divide the multi-core processor resources of the programmable logic controller through a virtualization layer, allocate at least one CPU core to the runtime program, and allocate at least one CPU core to the AI ​​reasoning program.

[0044] Preferably, the programmable logic controller is configured with a specific shared memory, and the runtime program in the real-time operating system can pass the data or status information collected by the image acquisition module to the AI ​​reasoning program in the Linux operating system through the specific shared memory.

[0045] Preferably, the AI ​​reasoning program performs fire detection on the image data and stores the result in the specific shared memory. When the monitoring result indicates that a fire has occurred, the runtime program triggers the alarm system of the programmable logic controller and executes a preset emergency handling procedure.

[0046] See also Figure 1 , the actual operation process of the system is as follows:

[0047] 1. The camera continuously collects images of the surrounding environment and transmits the image data to the PLC.

[0048] 2. When the PLC is running, the image data is transferred to the AI ​​reasoning program in the Linux system through shared memory.

[0049] 3. The AI ​​reasoning program calls the YOLOv8 model to perform fire detection on the image.

[0050] 4. If the YOLOv8 model detects a fire, the AI ​​reasoning program transmits the alarm signal back to the PLC runtime via shared memory.

[0051] Preferably, the AI ​​reasoning program is a lightweight convolutional neural network based on the YOLOv8 model. The YOLOv8 model is trained based on a large amount of fire image data and then deployed to the Linux operating system. When the monitoring system is running, the Linux operating system obtains the image data collected by the image acquisition module, processes it, and passes it to the YOLOv8 model.

[0052] During the PLC runtime software cycle, the program is executed as follows:

[0053] Step 1: Input sampling. Read the current PLC device status, such as external sensor information, and synchronize it to zone I.

[0054] Step 2: State synchronization, copy the shared memory input area data to the local I area.

[0055] Step 3: IEC61131-3 program execution. Execute the IEC61131-3 program written by the PLC user, including the logic control program. For function blocks that need to execute corresponding actions such as fire alarm or water gun opening, if the instructions are read from the shared memory instruction area, the control function block will be in the enabled / triggered state; for function blocks that do not require AI reasoning software to provide instructions, they are directly read from the local state data to change the state of the function block.

[0056] In the AI ​​inference software cycle, the following steps are followed to execute the program and operate the shared memory:

[0057] Step 1: Read the camera data and read the system clock cycle. After reaching the specified interval cycle, the program calls the camera data reading function block, reads a frame of RGB image, and saves it into a temporary variable.

[0058] Step 2: Data preprocessing: The data read by the camera is processed into 640*640 resolution data, and image preprocessing such as normalization is performed.

[0059] Step 3: AI model reasoning. Execute the AI ​​model reasoning program and send the processed data to the AI ​​model. The AI ​​model will reason based on the data and give the reasoning results, including whether there is a fire event. If there is a fire event, the location of the fire event will be output. Send the reasoning results to the output queue.

[0060] Step 4: Synchronize AI inference results. Send the output queue that stores AI inference results to the shared memory area. If a fire occurs, send the location of the fire to the shared memory area. If no fire occurs, write a safety report to the shared memory area.

[0061] The AI ​​reasoning program in the present invention is implemented based on the YOLOv8 deep learning algorithm. YOLOv8 is a lightweight convolutional neural network (CNN) with efficient target detection capabilities. In order to meet the needs of fire monitoring, YOLOv8 is further optimized and fine-tuned on a pre-trained fire data set so that it can accurately identify fire signals, such as flames, smoke and other features. The image data acquired by the camera is first processed by the Linux operating system and passed to the YOLOv8 model. The model recognizes fire-related targets in the image in real time and outputs fire detection results, including information such as the position of the detection frame and the probability of fire. Once the fire detection result exceeds the preset alarm threshold, the system will immediately feedback to the PLC runtime to trigger the alarm mechanism.

[0062] Embodiment 2

[0063] Based on the same concept, this embodiment provides an AI fire monitoring system control method deployed on a programmable logic controller, which is applied to any of the monitoring systems described above, and includes the following steps:

[0064] The image acquisition module collects image data in real time and reads the system clock cycle. After reaching the specified interval cycle, the program calls the camera data reading function block to read a frame of RGB image and save it to a temporary variable.

[0065] After preprocessing, it is stored in the memory of the programmable logic controller;

[0066] The AI ​​reasoning program determines whether a fire has occurred based on the preprocessed image data;

[0067] When the judgment result is that a fire has occurred, the runtime program in the programmable logic controller triggers the alarm system of the programmable logic controller and executes a preset emergency processing procedure.

[0068] The technical solution of this embodiment is to deploy a real-time operating system for executing the programmable logic controller runtime program and a Linux operating system for running the AI ​​reasoning program to perform the image recognition-based fire monitoring task on a single programmable logic controller device. The runtime software is responsible for implementing the execution of the logic control program and the interface of the PLCopen standard motion control function block, and runs in a loop according to input sampling, IEC61131-3 program execution, output refresh and execution response callback function. The AI ​​reasoning software is responsible for video data input, video data preprocessing, AI model creation, and looping AI model reasoning and reasoning result output. The shared memory is responsible for realizing the information interaction between the PLC runtime software and the AI ​​reasoning software. The integrated deployment of the programmable logic controller runtime and the AI ​​model reasoning is realized without the need for additional external communication or hardware. And compared with the prior art, the technical solution of this embodiment is that the AI ​​model is deployed on a programmable logic controller closer to the site. After receiving the alarm signal during operation, the predefined emergency response measures are immediately triggered, such as starting the fire protection system or issuing an alarm notification, and the response is faster.

[0069] Preferably, the method further comprises:

[0070] The runtime program in the real-time operating system transmits the data or status information collected by the image acquisition module to the AI ​​reasoning program in the Linux operating system through a specific shared memory;

[0071] The AI ​​reasoning program performs fire detection on the image data and stores the result in the specific shared memory;

[0072] Wherein, a specific shared memory is configured in the programmable logic controller.

[0073] Preferably, the method further comprises:

[0074] When the monitoring result indicates that a fire has occurred, the runtime program triggers the alarm system of the programmable logic controller and executes a preset emergency processing procedure.

[0075] Preferably, the method further comprises training the AI ​​reasoning program:

[0076] Before the system is run, a large amount of fire image data is divided into a training set and a test set. The training set is used to train the AI ​​reasoning program, and the test set is used to verify whether the training results meet the predetermined accuracy requirements;

[0077] After the system is running, the image data acquired in real time is used to further optimize and fine-tune the AI ​​reasoning program.

[0078] Based on the same concept, the present invention also provides a readable storage medium, on which a processing program is stored. When the processing program is executed by a processor, the AI ​​fire monitoring system control method deployed on a programmable logic controller as described above is implemented.

[0079] Based on the same concept, the present invention also provides an electronic device, comprising: a memory, the memory being used to store a processing program; a processor, the processor implementing any one of the above-mentioned AI fire monitoring system control methods deployed on a programmable logic controller when executing the processing program.

[0080] The control method of the AI ​​fire monitoring system deployed on the programmable logic controller can be stored in a computer-readable storage medium if it is implemented in the form of program instructions and sold or used as an independent product. Based on this understanding, the technical solution of this embodiment is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of software, and the computer software is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc., and other media that can store program codes.

[0081] Those skilled in the art can clearly understand that, for the sake of convenience and brevity in description, the identification content specifically executed by the above-described system and device can refer to the corresponding process in the aforementioned method embodiment.

[0082] The embodiments of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above embodiments. Even if various changes are made to the present invention, if these changes fall within the scope of the claims of the present invention and their equivalents, they still fall within the protection scope of the present invention.

Claims

1. An AI fire monitoring system deployed on a programmable logic controller, characterized in that: include: A programmable logic controller, wherein the programmable logic controller has at least a dual-core processor and is configured with an image acquisition module, wherein the programmable logic controller is configured to: Deploy an industrial control operating system that supports virtualization functions, divide the processor resources in the programmable logic controller through the virtualization function of the industrial control operating system, and virtualize two independent operating system instances: one instance is a real-time operating system, which is used to execute the runtime program of the programmable logic controller to ensure the real-time performance and high reliability of industrial automation control tasks, and the other instance is a Linux operating system, which is used to run AI reasoning programs to perform fire monitoring tasks based on image recognition; The image acquisition module acquires image data in real time and sends it to the AI ​​reasoning program. When the reasoning result of the AI ​​reasoning program is of a preset type, the real-time operating system executes a predetermined action program.

2. The AI ​​fire monitoring system deployed on a programmable logic controller according to claim 1 is characterized in that: The programmable logic controller is further configured to: divide the multi-core processor resources of the programmable logic controller through a virtualization layer, allocate at least one CPU core to the runtime program, and allocate at least one CPU core to the AI ​​reasoning program.

3. The AI ​​fire monitoring system deployed on a programmable logic controller according to claim 2 is characterized in that: The programmable logic controller is configured with a specific shared memory, and the runtime program in the real-time operating system can pass the data or status information collected by the image acquisition module to the AI ​​reasoning program in the Linux operating system through the specific shared memory.

4. The AI ​​fire monitoring system deployed on a programmable logic controller according to claim 3 is characterized in that: The AI ​​reasoning program performs fire detection on the image data and stores the result in the specific shared memory. When the monitoring result is a fire, the runtime program triggers the alarm system of the programmable logic controller and executes a preset emergency processing procedure.

5. The AI ​​fire monitoring system deployed on a programmable logic controller according to claim 4 is characterized in that: The AI ​​reasoning program is a lightweight convolutional neural network based on the YOLOv8 model. The YOLOv8 model is trained based on a large amount of fire image data and then deployed to the Linux operating system. When the monitoring system is running, the Linux operating system obtains the image data collected by the image acquisition module, processes it, and passes it to the YOLOv8 model.

6. A control method for an AI fire monitoring system deployed on a programmable logic controller, applied to the monitoring system according to any one of claims 1 to 5, characterized in that: The steps include: The image acquisition module collects image data in real time and reads the system clock cycle. After reaching the specified interval cycle, the program calls the camera data reading function block to read a frame of RGB image and save it to a temporary variable. After preprocessing, it is stored in the memory of the programmable logic controller; The AI ​​reasoning program determines whether a fire has occurred based on the preprocessed image data; When the judgment result is that a fire has occurred, the runtime program in the programmable logic controller triggers the alarm system of the programmable logic controller and executes a preset emergency processing procedure.

7. The control method according to claim 6, characterized in that: The method further comprises: The runtime program in the real-time operating system transmits the data or status information collected by the image acquisition module to the AI ​​reasoning program in the Linux operating system through a specific shared memory; The AI ​​reasoning program performs fire detection on the image data and stores the result in the specific shared memory; Wherein, a specific shared memory is configured in the programmable logic controller.

8. The control method according to claim 6, characterized in that: The method further comprises: When the monitoring result indicates that a fire has occurred, the runtime program triggers the alarm system of the programmable logic controller and executes a preset emergency processing procedure.

9. The control method according to claim 6, characterized in that: The method further includes training the AI ​​reasoning program: Before the system is run, a large amount of fire image data is divided into a training set and a test set. The training set is used to train the AI ​​reasoning program, and the test set is used to verify whether the training results meet the predetermined accuracy requirements; After the system is running, the image data acquired in real time is used to further optimize and fine-tune the AI ​​reasoning program.

10. A readable storage medium, characterized in that: The readable storage medium stores a processing program, and when the processing program is executed by the processor, the AI ​​fire monitoring system control method deployed on the programmable logic controller as described in any one of claims 6 to 9 is implemented.

Images