AI-based pipe gallery fire operation hidden danger automatic identification method and device

Abstract

The invention provides an AI-based pipe gallery fire operation hidden danger automatic identification method and device. The method comprises the steps of collecting data of a pipe gallery fire operation site in real time and preprocessing the data; identifying and comparing the check points by adopting an image detection classification algorithm based on a diffusion model and a large model distillation technology to obtain an identification and comparison result; and calling a pre-constructed LLM large language model, carrying out potential safety hazard identification on the preprocessed multi-modal data and the identification comparison result through a multi-modal input algorithm, and outputting a multi-modal fire operation comprehensive supervision report. A diffusion model and a large model distillation technology are utilized, data augmentation and data preprocessing are provided, image detection and classification of a pipe gallery fire operation site are realized, and real-time monitoring, automatic hidden danger identification and intelligent early warning of the pipe gallery fire operation site can be realized through LLM construction of an overall system framework and multi-modal input algorithm analysis. And the accuracy and efficiency of potential safety hazard identification are improved.

Description

Technical Field

[0001] The present application relates to the field of industrial safety technology, and in particular to an artificial intelligence (AI)-based method and device for automatically identifying hidden dangers of hot work in pipe corridors. Background Art

[0002] Hot work in the pipe corridor refers to welding, cutting, grinding and other operations that may generate sparks or high temperatures inside or around the pipe corridor. Hot work in the pipe corridor is highly dangerous and is one of the special operations. Due to the large number of points and long lines in the pipe corridor, this type of operation has many safety hazards, such as leakage of flammable and explosive gases, lack of fire-fighting facilities, and improper protection of operators. Traditional methods for identifying safety hazards mainly rely on manual inspections and regular inspections, which have problems such as waste of resources, low identification efficiency, limited coverage, and susceptibility to subjective factors. Abnormal behavior can only be identified after the fact by checking monitoring and other methods, which has the risk of accidents and can no longer meet current application needs. Intelligent means are needed to assist in improving the level of supervision and reducing safety risks.

[0003] The current target detection technology for hot work operations can achieve good results in a single specific environment, but it has certain limitations when facing complex scenes. It does not meet the requirements of accuracy and real-time performance in the hot work process, especially for the detection of safety hazards, and cannot guarantee the safety of hot work workers to the greatest extent.

[0004] Therefore, how to improve the level and accuracy of automatic identification of hidden dangers of hot work in pipe corridors is a technical problem that technical personnel in this field need to urgently solve. Summary of the invention

[0005] In view of this, it is necessary to provide an AI-based automatic identification method and device for pipeline corridor fire operations. By integrating the diffusion model and large model distillation technology, image detection and classification of pipeline corridor fire operations are realized. At the same time, the powerful capabilities of the large language model (LLM) are used to build a comprehensive automatic identification device system for pipeline corridor fire operations. By integrating the text understanding, image recognition and decision support functions of LLM, the device system can realize real-time monitoring of pipeline corridor fire operations, automatic identification of hidden dangers and intelligent early warning, and improve the accuracy and efficiency of safety hazard identification.

[0006] In a first aspect, an embodiment of the present application provides an AI-based method for automatically identifying hidden dangers of hot work in a pipe gallery, the method comprising the following steps: S1: Obtain data from explosion-proof cameras and environmental sensors to collect real-time data from the fire operation site in the corridor; S2: preprocessing the collected multimodal data; S3: It uses an image detection and classification algorithm based on a diffusion model and large model distillation technology to provide data augmentation and data preprocessing, and performs identification and comparison of checkpoints on the preprocessed image data to obtain identification and comparison results; S4: calling the pre-built LLM large language model, identifying potential safety hazards on the pre-processed multimodal data and the recognition and comparison results through a multimodal input algorithm, and outputting a multimodal hot work comprehensive supervision report; S5: Display reports through a visualization platform based on the comprehensive supervision report, issue hidden danger warnings and response suggestions when preset conditions are met, record hidden danger handling status, and optimize the LLM large language model.

[0007] Optionally, in another implementation of the first aspect of the present invention, the step S1: acquiring data collected by explosion-proof cameras and environmental sensors at the hot work site in the pipe gallery in real time includes: According to the principle of all-round coverage, cameras that support high-definition video acquisition, night vision and real-time transmission capabilities are installed at the fire operation site in the tunnel to collect real-time video streams; Deploy a variety of sensors to monitor environmental parameters, including: Gas sensor, used to detect the concentration of combustible gas; Temperature sensor, used to monitor ambient temperature; Humidity sensor, used to monitor humidity changes; Smoke sensors to detect fire risks; Sound sensor, used to monitor sound changes; Deploy edge computing devices locally to pre-process camera and sensor data, reduce data transmission volume and improve response speed.

[0008] Optionally, in another implementation of the first aspect of the present invention, the S2: preprocessing the collected multimodal data includes: Classify the collected data according to data format, including numerical data, video or image data, text data, and audio data; The collected multimodal data are cleaned separately, including: Use Kalman filter algorithm to remove noise from numerical data; For image / video data: use Gaussian denoising algorithm to reduce image noise; Remove special characters, stop words and irrelevant symbols from text data; Resample the audio to a uniform sampling rate to ensure consistency, use energy or zero-crossing rate detection and remove silence; The missing values ​​of the collected multimodal data are processed separately, including: For numerical data, linear interpolation is used to fill missing values; for text data, prediction models are used to fill missing values; for image / video data, image restoration technology is used to fill missing areas; Use statistical methods on audio to detect and handle outliers; Perform data standardization and normalization on multimodal data, including: Perform Min-Max normalization on the numerical data in the sensor data to make the data in the same dimension; Normalize the image pixel values ​​and scale the pixel values ​​to the range of [0, 1]; Normalize the audio signal to the range of [-1, 1], enhance the high frequency part through a high-pass filter, and balance the spectrum.

[0009] Optionally, in another implementation of the first aspect of the present invention, S3: using an image detection and classification algorithm based on a diffusion model and a large model distillation technology, providing data augmentation and data preprocessing, performing recognition and comparison of checkpoints on the preprocessed image data, and obtaining recognition and comparison results, including: Obtain preprocessed image data; call the distilled target classification model; use the distilled target classification model to identify and compare the checkpoints to obtain the identification and comparison results; Among them, the classification model after distillation is obtained through the following steps: An initial diffusion model is constructed by mapping a fixed Markov chain to a latent space; noise is gradually added to the initial diffusion model to obtain an approximate posterior value; a reverse Markov transform is found by learning an inverse process to train the diffusion model; Collecting historical image data as sample image data; Using the training diffusion model to input sample image data to generate training data for a classification image model, and annotating the data with labels; Using the training data and the annotations to construct an image classification model and train it, Use a large diffusion model as the teacher model and a small reverse process model as the student model. Perform progressive distillation at a ratio of two teacher models to one student model; To obtain the target classification model after distillation; The identification and comparison results are the identification and comparison results of the detection points, including: personnel, location, type of operation, fire prevention measures, and finished product protection measures.

[0010] Optionally, in another implementation of the first aspect of the present invention, the objective loss function of the diffusion model is for: , in , Both represent positive conditional probabilities, , Both represent positive conditional probabilities, express The error function from 1 to T, represents KL divergence; Objective loss function for target classification model for: , Where N is the number of samples, K is the number of checkpoints, is the true label of the i-th sample under the k-th label, is the probability of the i-th sample under the k-th label.

[0011] Optionally, in another implementation of the first aspect of the present invention, the S4: calling a pre-built LLM large language model, identifying potential safety hazards on the pre-processed multimodal data and the recognition and comparison results through a multimodal input algorithm, and outputting a multimodal hot work comprehensive supervision report, including: Get the pre-built multimodal LLM large language model; Input preprocessed multimodal data; Extract the input multimodal fusion data feature matrix through the multimodal input algorithm , , This includes text extraction feature vector , video extraction feature vector , image extraction feature vector , audio extraction feature vector ; Compare the input multimodal fusion data feature matrix with the standard multimodal fusion data feature matrix in the knowledge base to obtain similarity comparison results and output the hidden danger probability; Output a comprehensive supervision report on multi-modal hot work operations based on the probability of hidden dangers.

[0012] Optionally, in another implementation of the first aspect of the present invention, in obtaining the similarity comparison result, the similarity expression is calculated as: , in, is the standard multimodal fusion data feature matrix, To adjust the amplitude.

[0013] In a second aspect, an embodiment of the present application provides an AI-based automatic identification device for potential hazards of hot work in a pipe gallery, which is applied to the AI-based automatic identification method for potential hazards of hot work in a pipe gallery described in the first aspect, and is characterized in that the device comprises: The data acquisition module is used to obtain data collected by explosion-proof cameras and environmental sensors in real time at the fire operation site in the pipeline corridor; A data processing module, used for preprocessing the collected multimodal data; The checkpoint recognition module is used to use the image detection and classification algorithm based on the diffusion model and large model distillation technology to identify and compare the checkpoints on the preprocessed image data to obtain the identification and comparison results; The hidden danger identification module is used to call the pre-built LLM large language model, identify potential safety hazards on the pre-processed multimodal data and the identification and comparison results through a multimodal input algorithm, and output a comprehensive supervision report on multimodal hot work; The early warning and decision-making module is used to display reports through a visualization platform based on the comprehensive supervision report, issue hidden danger warnings and response suggestions when the preset conditions are met, record the hidden danger handling situation, and optimize the LLM large language model.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including: processor; a memory for storing processor-executable instructions; Among them, the processor is configured to implement the AI-based automatic identification method for hidden dangers of fire operations in the pipeline corridor as described in the first aspect when executing the instructions.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a program, wherein the program instructs a device to execute the AI-based automatic identification method for hidden dangers of fire operations in a pipeline corridor as described in the first aspect.

[0016] In the technical solution provided by the present invention, data of the hot work site in the pipeline corridor is collected in real time by acquiring explosion-proof cameras and environmental sensors; the collected multimodal data is preprocessed; an image detection and classification algorithm based on a diffusion model and a large model distillation technology is used to identify and compare checkpoints on the preprocessed image data to obtain identification and comparison results; a pre-constructed LLM large language model is called, and potential safety hazards are identified for the preprocessed multimodal data and the identification and comparison results through a multimodal input algorithm, and a comprehensive supervision report on the multimodal hot work is output; a report is displayed through a visualization platform according to the comprehensive supervision report, and when the preset conditions are met, hidden danger warnings and response suggestions are issued, the hidden danger handling situation is recorded, and the LLM large language model is optimized.

[0017] The beneficial effects of the system specifically include: (1) Using diffusion models and knowledge distillation technology, we can detect and classify images of hot work sites in pipe corridors. This algorithm can identify and compare the following checkpoints, thus realizing image detection and classification of hot work sites in pipe corridors.

[0018] (2) The overall system framework is constructed through LLM, and multi-modal input (operation approval form, on-site video, pictures, spatial location information, etc.) is used for algorithm analysis, and then multi-modal output is used for hot work supervision report, abnormal alarm, and platform display. It can realize real-time monitoring of the hot work site in the corridor, automatic identification of hidden dangers, and intelligent early warning, thereby improving the accuracy and efficiency of identifying safety hazards.

[0019] (3) Through LLM's fault prediction technology, by analyzing equipment status data, potential hidden dangers can be predicted and maintenance can be carried out in advance.

[0020] (4) Conduct research on intelligent algorithms and control strategies to achieve automatic control and optimization of LLM. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A flowchart of an AI-based automatic identification method for potential hazards of fire operations in a pipe gallery provided in one embodiment of the present application.

[0022] Figure 2 A schematic diagram of the distillation technology using a diffusion model and a large model provided in one embodiment of the present application.

[0023] Figure 3 A schematic diagram of an LLM model provided for another embodiment of the present application.

[0024] Figure 4 A schematic diagram of a module of an AI-based automatic identification device for hidden dangers of fire operations in a pipe gallery provided in one embodiment of the present application.

[0025] Figure 5 A schematic diagram of an electronic terminal device provided in one embodiment of the present application. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0027] It should be noted that, in the embodiments of the present application, "at least one" refers to one or more, and "more" refers to two or more. Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art in the present application. The terms used in the specification of the present application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0028] It should be noted that, in the embodiments of the present application, words such as "first" and "second" are only used for the purpose of distinguishing descriptions, and cannot be understood as indicating or implying relative importance, nor can they be understood as indicating or implying order. Features defined as "first" and "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or explanations. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a concrete way.

[0029] Based on the implementations in this application, all other implementations obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0030] The core of this application is to provide an AI-based automatic identification method for hidden dangers of fire operations in pipeline corridors, which can improve the automation and intelligent management level of anode protection equipment to achieve remote monitoring, intelligent control and safety protection, and ensure that the anode protection equipment can operate safely and efficiently even without supervision.

[0031] Figure 1 A flowchart of an AI-based automatic identification method for fire hazards in a pipe gallery provided in an embodiment of the present application. Figure 1 As shown, an AI-based automatic identification method for hidden dangers of hot work in a pipe gallery comprises the following steps: S1: Obtain data from explosion-proof cameras and environmental sensors to collect real-time data from the fire operation site in the corridor.

[0032] It can be understood that a sensor is a component or device that has the function of sensing and detecting certain information of the object being measured and converts it into a corresponding useful signal according to a certain rule. It is usually composed of a sensitive element and a conversion element.

[0033] For the automatic identification equipment of hot work hazards, a variety of sensors are used to collect operating parameters or scene data during the hot work process. The main factors affecting its hazard identification are scene parameters (for example, personnel verification, location confirmation, personnel work behavior, personnel protection behavior, etc.) and production process parameters (for example, sound, smoke, etc.); these types of parameters need to be monitored at all times.

[0034] The parameters collected by the above sensors are transmitted and saved in the form of text, images, videos, etc. In addition to the above-mentioned influencing factors, fault detection methods based on sound recognition have gradually become a new research hotspot. Specific sound signals are generated during the operation process, which can reflect the health status of the equipment or the progress of the business and provide important clues for fault diagnosis.

[0035] The sound during the operation is collected, and the audio signal is preprocessed, feature extracted and machine learning model trained to finally achieve accurate fault identification. Experiments have shown that this method has a very high accuracy in the detection of various types of faults. This method provides an effective solution for intelligent fault detection in hot work in pipe corridors.

[0036] Specifically, in the embodiment of the present application, the step S1: obtaining data collected by explosion-proof cameras and environmental sensors in real time at the fire operation site in the pipe gallery includes: According to the principle of all-round coverage, cameras that support high-definition video acquisition, night vision and real-time transmission capabilities are installed at the fire operation site in the tunnel to collect real-time video streams; Deploy a variety of sensors to monitor environmental parameters, including: Gas sensor, used to detect the concentration of combustible gas; Temperature sensor, used to monitor ambient temperature; Humidity sensor, used to monitor humidity changes; Smoke sensors to detect fire risks; Sound sensor, used to monitor sound changes; Deploy edge computing devices locally to pre-process camera and sensor data, reduce data transmission volume and improve response speed.

[0037] It is understandable that appropriate cameras and sensors are selected and deployed at key locations in the tunnel. Integrate cameras and sensors with data transmission equipment and monitoring platforms. Explosion-proof cameras: such as Hikvision explosion-proof series. Environmental sensors: such as Honeywell gas sensors, Bosch temperature and humidity sensors, etc. The collected data is transmitted to the monitoring platform via wireless transmission: LoRa or 5G module.

[0038] It can be understood that after completing the preprocessing of the original anode protection equipment audio signal, a variety of technologies are used to extract audio features related to equipment failure from the time domain, frequency domain and time-frequency domain, mainly STFT and MFCC. Through STFT, the audio signal can be effectively converted from the time domain to the frequency domain, the spectrum can be obtained, and the frequency distribution of the signal in different time periods can be analyzed. In order to more accurately simulate the sensitivity of the human ear to different frequencies, the spectrum is filtered using a Mel filter. This is because the center frequency of the Mel filter is logarithmically distributed, which is consistent with the auditory characteristics of the human ear, and can extract frequency features that are closely related to the working status of the equipment.

[0039] S2: Preprocess the collected multimodal data.

[0040] It is understandable that preprocessing the collected multimodal data (such as videos, images, sensor data, text, sound, etc.) is a key step to ensure data quality and improve the effectiveness of subsequent analysis and model training. The goal of multimodal data preprocessing is to convert data from different sources and formats into a unified format suitable for analysis, while removing noise, filling missing values, and extracting useful features. The following are the detailed steps and methods for multimodal data preprocessing.

[0041] Specifically, in the embodiment of the present application, the step S2: preprocessing the collected multimodal data includes: Classify the collected data according to data format, including numerical data, video or image data, text data, and audio data; The collected multimodal data are cleaned separately, including: Use Kalman filter algorithm to remove noise from numerical data; For image / video data: use Gaussian denoising algorithm to reduce image noise; Remove special characters, stop words and irrelevant symbols from text data; Resample the audio to a uniform sampling rate to ensure consistency, use energy or zero-crossing rate detection and remove silence; The missing values ​​of the collected multimodal data are processed separately, including: For numerical data, linear interpolation is used to fill missing values; for text data, prediction models are used to fill missing values; for image / video data, image restoration technology is used to fill missing areas; Use statistical methods on audio to detect and handle outliers; Perform data standardization and normalization on multimodal data, including: Perform Min-Max normalization on the numerical data in the sensor data to make the data in the same dimension; Normalize the image pixel values ​​and scale the pixel values ​​to the range of [0, 1]; Normalize the audio signal to the range of [-1, 1], enhance the high frequency part through a high-pass filter, and balance the spectrum.

[0042] Specifically, in the embodiments of the present application, through these preprocessing steps, the signal-to-noise ratio of the audio signal is significantly improved, external interference and unnecessary components in the signal are removed, and a clearer and more reliable data basis is provided for subsequent feature extraction, thereby ensuring the efficiency and accuracy of fault detection.

[0043] S3: Adopt the image detection and classification algorithm based on diffusion model and large model distillation technology, provide data augmentation and data preprocessing, perform identification and comparison of checkpoints on the preprocessed image data, and obtain the identification and comparison results.

[0044] It can be understood that diffusion models are generative models, meaning that they are used to generate data that is similar to the training data. Fundamentally, diffusion models work by corrupting the training data by continuously adding Gaussian noise, and then recovering the data by learning to invert this noise process. After training, we can use the diffusion model to generate data by simply passing randomly sampled noise through the learned denoising process.

[0045] Figure 2 This is a schematic diagram of the distillation technology using a diffusion model and a large model provided in one embodiment of the present application. Figure 2 As shown in Figure 1, the diffusion model is a generative model that provides data augmentation and data preprocessing by simulating the diffusion process of data distribution to generate high-quality data (such as images, audio, etc.). The core idea is to transform the data distribution into a simple distribution (such as a Gaussian distribution) by gradually adding noise, and then learn the inverse process to recover the data from the noise. After training, the diffusion model can be used to generate data by simply passing randomly sampled noise through the learned denoising process.

[0046] It can be understood that large model distillation is a model compression technique that aims to transfer the knowledge of a large model (teacher model) to a small model (student model), thereby reducing the computational cost and resource requirements of the model while maintaining performance. The key steps of distillation include: Step 1, training the teacher model. Use the complete dataset to train a high-performance large model. Step 2, generate soft labels. Use the teacher model to infer the training data and generate soft labels (softlabels), that is, probability distributions. Step 3, train the student model. Use soft labels and true labels to jointly train the student model to mimic the behavior of the teacher model.

[0047] It can be understood that in this embodiment, combining the diffusion model with the large model distillation can significantly reduce the computational cost and resource requirements of the diffusion model while maintaining the generation quality. The following is the idea and method of the combination.

[0048] Specifically, in the embodiment of the present application, S3: adopts an image detection and classification algorithm based on a diffusion model and a large model distillation technology, provides data augmentation and data preprocessing, performs identification and comparison of checkpoints on the preprocessed image data, and obtains identification and comparison results, including: Obtain preprocessed image data; call the distilled target classification model; use the distilled target classification model to identify and compare the checkpoints to obtain the identification and comparison results; Among them, the classification model after distillation is obtained through the following steps: An initial diffusion model is constructed by mapping a fixed Markov chain to a latent space; noise is gradually added to the initial diffusion model to obtain an approximate posterior value; a reverse Markov transform is found by learning an inverse process to train the diffusion model; Collecting historical image data as sample image data; Using the training diffusion model to input sample image data to generate training data for a classification image model, and annotating the data with labels; Using the training data and the annotations to construct an image classification model and train it, Use a large diffusion model as the teacher model and a small reverse process model as the student model. Perform progressive distillation at a ratio of two teacher models to one student model; To obtain the target classification model after distillation; The identification and comparison results are the identification and comparison results of the detection points, including: personnel, location, type of operation, fire prevention measures, and finished product protection measures.

[0049] Specifically, in the embodiment of the present application, the objective loss function of the diffusion model is for: , in , Both represent positive conditional probabilities, , Both represent positive conditional probabilities, express The error function from 1 to T, represents KL divergence; Objective loss function for target classification model for: , Where N is the number of samples, K is the number of checkpoints, is the true label of the i-th sample under the k-th label, is the probability of the i-th sample under the k-th label.

[0050] The main inspection contents during the hot work are the comparison and verification between the hot work permit and the actual personnel, location, type of operation, fire prevention measures and finished product protection measures at the tunnel site. The image detection and classification algorithm based on the diffusion model and large model distillation technology can identify and compare the following checkpoints.

[0051] In the embodiment of the present application, it can be, for example: Personnel: Personnel identification, whether the declaration is consistent with the site, quantity and face. Location: No identification is required, the camera position is consistent with the cabin partition position filled in the fire permit. Type of operation: electric welding, gas welding, cutting welding, electric drill drilling, identification of whether special machines and gas cylinders are used. At present, there is mainly one welding method in the corridor, two-protection welding in arc welding in electric welding, and the special machine is the carbon dioxide cylinder, and the others are conventional. The identifiable anomalies are: whether the gas cylinder is a carbon dioxide cylinder, whether a cutting machine is used, and whether an electric drill (regardless of the size) is used. Firefighting measures: fire buckets, fire extinguishers. Finished product protection measures: The pipe corridor is relatively narrow, and basically all sections are covered if they need to be covered, so the problem can be converted into identifying whether the cross-sectional space below the fire point and 2 meters on the left and right is covered. Further focus can be placed on two types of flame retardants (such as fire blankets) to identify whether the covering is compliant.

[0052] S4: Call the pre-built LLM large language model, identify potential safety hazards on the pre-processed multimodal data and the recognition and comparison results through the multimodal input algorithm, and output a comprehensive supervision report on multimodal hot work.

[0053] Figure 3 This is a schematic diagram of an LLM model provided by another embodiment of the present application. Figure 3 As shown, the pre-built LLM large language model has multi-modal input data, and the corresponding retrieval results can be obtained based on the comparison of the knowledge base.

[0054] It is understandable that the Large Language Model (LLM) is a natural language processing (NLP) model based on deep learning that can understand and generate human language. The Large Language Model is a deep learning model based on the Transformer architecture that learns the statistical laws and semantic representation of language through training on large-scale text data. It can complete a variety of tasks, such as text generation, text translation, text understanding, question-answering systems, retrieval systems, etc.

[0055] The multimodal large language model is a deep learning model that can simultaneously process and understand multiple modal data (such as text, images, audio, video, etc.). Unlike traditional single-modal language models (such as GPT, BERT), the multimodal large language model can more comprehensively understand and generate content by integrating information from multiple modalities, thereby supporting more complex tasks and applications. It can process multiple types of data input at the same time, such as text + image, text + audio, text + video, etc. The application in this embodiment includes three modal data, which are text + audio + image.

[0056] Specifically, in the embodiment of the present application, the S4: calling the pre-built LLM large language model, identifying potential safety hazards on the pre-processed multimodal data and the recognition and comparison results through a multimodal input algorithm, and outputting a multimodal hot work comprehensive supervision report, including: Get the pre-built multimodal LLM large language model; Input preprocessed multimodal data; Extract the input multimodal fusion data feature matrix through the multimodal input algorithm , , This includes text extraction feature vector , video extraction feature vector , image extraction feature vector , audio extraction feature vector ; Compare the input multimodal fusion data feature matrix with the standard multimodal fusion data feature matrix in the knowledge base to obtain similarity comparison results and output the hidden danger probability; Output a comprehensive supervision report on multi-modal hot work operations based on the probability of hidden dangers.

[0057] Specifically, in the embodiment of the present application, in obtaining the similarity comparison result, the similarity expression is calculated as: , in, is the standard multimodal fusion data feature matrix, To adjust the amplitude.

[0058] S5: Display reports through a visualization platform based on the comprehensive supervision report, issue hidden danger warnings and response suggestions when preset conditions are met, record hidden danger handling status, and optimize the LLM large language model.

[0059] Specifically, through feedback data, a variety of evaluation indicators (such as accuracy, F1 score, BLEU, etc.) can be used to comprehensively evaluate model performance and perform optimization based on the evaluation results.

[0060] It is understandable that in addition to carrying out equipment operation and maintenance work, it is also possible to set online monitoring levels, safety warnings, and maintenance work allocation, make relevant inspection or fault handling plans, arrange maintenance personnel to handle them, help maintenance personnel respond quickly, and reduce the risk of major failures caused by ignoring small defects.

[0061] Figure 4 1 is a functional module diagram of an AI-based automatic identification device for fire hazards in a pipe gallery provided in an embodiment of the present application. Figure 4 The device for automatically identifying hidden dangers of hot work in a pipe gallery based on AI is applied to the method for automatically identifying hidden dangers of hot work in a pipe gallery based on AI as described in Example 1, and includes a data acquisition module 11, a data processing module 12, a checkpoint identification module 13, a hidden danger identification module 14, and an early warning and decision-making module 15 connected in sequence, wherein: The data acquisition module 11 is used to obtain data collected by explosion-proof cameras and environmental sensors in real time at the fire operation site in the pipeline corridor.

[0062] The data processing module 12 is used to pre-process the collected multimodal data.

[0063] The checkpoint identification module 13 is used to use an image detection and classification algorithm based on a diffusion model and a large model distillation technology to perform identification and comparison of checkpoints on the preprocessed image data to obtain identification and comparison results.

[0064] The hidden danger identification module 14 is used to call the pre-built LLM large language model, identify potential safety hazards on the pre-processed multimodal data and the identification and comparison results through a multimodal input algorithm, and output a comprehensive supervision report on multimodal hot work.

[0065] The early warning and decision-making module 15 is used to display reports through a visualization platform based on the comprehensive supervision report, issue hidden danger warnings and response suggestions when preset conditions are met, record hidden danger handling situations, and optimize the LLM large language model.

[0066] It can be understood that the embodiment of the present application provides an AI-based automatic identification device for hidden dangers of hot work in pipe corridors, which collects and preprocesses data from the hot work site in the pipe corridor in real time; uses an image detection and classification algorithm based on a diffusion model and large model distillation technology to identify and compare checkpoints to obtain identification and comparison results; calls a pre-built LLM large language model, and uses a multimodal input algorithm to identify potential safety hazards for the pre-processed multimodal data and the identification and comparison results, and outputs a comprehensive supervision report on multimodal hot work. Using diffusion models and large model distillation technology, data augmentation and data preprocessing are provided to achieve image detection and classification of the hot work site in the pipe corridor. By constructing an overall system framework and multimodal input algorithm analysis through LLM, real-time monitoring of the hot work site in the pipe corridor, automatic identification of hidden dangers, and intelligent early warning can be achieved, thereby improving the accuracy and efficiency of identifying safety hazards.

[0067] See also Figure 5 , Figure 5 This is an electronic terminal device 500 provided in an embodiment of the present application. Figure 5 The electronic terminal device 500 shown includes at least the following parts: one or more processors 501, one or more input devices 502, one or more output devices 503 and one or more memories 504. The processors 501, input devices 502, output devices 503 and memories 504 communicate with each other via a communication bus 505. The memory 504 is used to store computer programs, which include program instructions. The processor 501 is used to execute the program instructions stored in the memory 504. The processor 501 is configured to call the program instructions to perform the following operations to perform the functions of the modules / units in the above-mentioned device embodiments, such as Figure 4 Functionality of the modules shown.

[0068] In an embodiment of the present application, a computer-readable storage medium includes program instructions, wherein the program instructions instruct a device to execute a system as in the first aspect. For example, the instructions instruct the device to execute Figure 1 An AI-based automatic identification method for hidden dangers of hot work in a pipe gallery is shown in the steps.

[0069] It should be understood that in the embodiment of the present invention, the processor 501 may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or the processor may be any conventional processor, etc. It should be noted that a part of the electronic device 500 of the above-mentioned embodiment may also be implemented by a computer. In this case, the program for implementing the control function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer and executed.

[0070] The input device 502 may include a touch panel, a fingerprint sensor (for collecting the user's fingerprint information and fingerprint direction information), a microphone, etc., and the output device 503 may include a display (LCD, etc.), a speaker, etc.

[0071] It should be noted that the "computer" mentioned here refers to a computer built into the electronic device 500, and uses a computer including hardware such as an OS and peripheral devices. In addition, "computer-readable recording medium" refers to removable media such as a floppy disk, a magneto-optical disk, a ROM, a CD-ROM, and a storage device such as a hard disk built into the computer.

[0072] Furthermore, "computer-readable recording media" may include: media that dynamically store programs for a short period of time, such as communication lines when sending programs via networks such as the Internet or communication lines such as telephone lines; and media that store programs for a fixed period of time, such as volatile memories inside computers that serve as servers or clients in this case. In addition, the above-mentioned program may be a program for realizing a part of the above-mentioned functions, or a program that can realize the above-mentioned functions by combining with a program already recorded in a computer.

[0073] In addition, the electronic device 500 in the above embodiment can also be implemented as a collection (device group) composed of multiple devices. Each device constituting the device group can have a part or all of the functions or functional blocks of the electronic device 500 in the above embodiment. As a device group, it is sufficient to have all the functions or functional blocks of the electronic device 500.

[0074] It can be understood that the AI-based automatic identification method and device for fire-related hazards in pipe corridors provided in the embodiments of the present application can collect and preprocess the data of the fire-related hazards in the pipe corridors in real time; use the image detection and classification algorithm based on the diffusion model and large model distillation technology to identify and compare the checkpoints to obtain the identification and comparison results; call the pre-built LLM large language model, and use the multimodal input algorithm to identify potential safety hazards for the pre-processed multimodal data and the identification and comparison results, and output a comprehensive supervision report on multimodal fire-related hazards. Using the diffusion model and large model distillation technology, data augmentation and data preprocessing are provided to achieve image detection and classification of the fire-related hazards in the pipe corridors. By constructing the overall system framework and multimodal input algorithm analysis through LLM, real-time monitoring, automatic identification of hazards and intelligent early warning of the fire-related hazards in the pipe corridors can be achieved, thereby improving the accuracy and efficiency of safety hazard identification.

[0075] Those skilled in the art should recognize that the above embodiments are only used to illustrate the present application and are not intended to be limiting of the present application. As long as they are within the spirit and scope of the present application, appropriate changes and modifications to the above embodiments are within the scope of protection claimed in the present application.

Claims

1. An AI-based automatic identification method for hidden dangers of hot work in pipe gallery, characterized in that: The method comprises the following steps: S1: Obtain data from explosion-proof cameras and environmental sensors to collect real-time data from the fire operation site in the corridor; S2: preprocessing the collected multimodal data; S3: Adopts image detection and classification algorithms based on diffusion model and large model distillation technology, provides data augmentation and data preprocessing, performs identification and comparison of checkpoints on preprocessed image data, and obtains identification and comparison results; S4: calling the pre-built LLM large language model, identifying potential safety hazards on the pre-processed multimodal data and the recognition and comparison results through a multimodal input algorithm, and outputting a multimodal hot work comprehensive supervision report; S5: Display reports through a visualization platform based on the comprehensive supervision report, issue hidden danger warnings and response suggestions when preset conditions are met, record hidden danger handling status, and optimize the LLM large language model.

2. According to the AI-based automatic identification method for hidden dangers of hot work in pipe gallery according to claim 1, it is characterized in that: S1: Acquire explosion-proof cameras and environmental sensors to collect data from the hot work site in the pipe gallery in real time, including: According to the principle of all-round coverage, cameras that support high-definition video acquisition, night vision and real-time transmission capabilities are installed at the fire operation site in the tunnel to collect real-time video streams; Deploy a variety of sensors to monitor environmental parameters, including: Gas sensor, used to detect the concentration of combustible gas; Temperature sensor, used to monitor ambient temperature; Humidity sensor, used to monitor humidity changes; Smoke sensors to detect fire risks; Sound sensor, used to monitor sound changes; Deploy edge computing devices locally to pre-process camera and sensor data, reduce data transmission volume and improve response speed.

3. According to the AI-based automatic identification method for hidden dangers of hot work in pipe gallery according to claim 2, it is characterized in that: S2: preprocessing the collected multimodal data, including: Classify the collected data according to data format, including numerical data, video or image data, text data, and audio data; The collected multimodal data are cleaned separately, including: Use Kalman filter algorithm to remove noise from numerical data; For image / video data: use Gaussian denoising algorithm to reduce image noise; Remove special characters, stop words and irrelevant symbols from text data; Resample the audio to a uniform sampling rate to ensure consistency, use energy or zero-crossing rate detection and remove silence; The missing values ​​of the collected multimodal data are processed separately, including: For numerical data, linear interpolation is used to fill missing values; for text data, prediction models are used to fill missing values; for image / video data, image restoration technology is used to fill missing areas; Use statistical methods on audio to detect and handle outliers; Perform data standardization and normalization on multimodal data, including: Perform Min-Max normalization on the numerical data in the sensor data to make the data in the same dimension; Normalize the image pixel values ​​and scale the pixel values ​​to the range of [0, 1]; Normalize the audio signal to the range of [-1, 1], enhance the high frequency part through a high-pass filter, and balance the spectrum.

4. According to the AI-based automatic identification method for hidden dangers of hot work in pipe gallery according to claim 2, it is characterized in that: S3: adopts an image detection and classification algorithm based on a diffusion model and a large model distillation technology, provides data augmentation and data preprocessing, performs identification and comparison of checkpoints on the preprocessed image data, and obtains identification and comparison results, including: Obtain preprocessed image data; call the distilled target classification model; use the distilled target classification model to identify and compare the checkpoints to obtain the identification and comparison results; Among them, the classification model after distillation is obtained through the following steps: An initial diffusion model is constructed by mapping a fixed Markov chain to a latent space; noise is gradually added to the initial diffusion model to obtain an approximate posterior value; a reverse Markov transform is found by learning an inverse process to train the diffusion model; Collecting historical image data as sample image data; Using the training diffusion model to input sample image data to generate training data for a classification image model, and annotating the data with labels; Using the training data and the annotations to construct an image classification model and train it, Use a large diffusion model as the teacher model and a small reverse process model as the student model. Perform progressive distillation at a ratio of two teacher models to one student model; To obtain the target classification model after distillation; The identification and comparison results are the identification and comparison results of the detection points, including: personnel, location, type of operation, fire prevention measures, and finished product protection measures.

5. According to the AI-based automatic identification method for hidden dangers of hot work in pipe gallery according to claim 4, it is characterized in that: The objective loss function of the diffusion model for: ， in , Both represent positive conditional probabilities, , Both represent positive conditional probabilities, express The error function from 1 to T, represents KL divergence; Objective loss function for target classification model for: ， Where N is the number of samples, K is the number of checkpoints, is the true label of the i-th sample under the k-th label, is the probability of the i-th sample under the k-th label.

6. According to the AI-based automatic identification method for hidden dangers of hot work in pipe gallery according to claim 4, it is characterized in that: S4: calling the pre-built LLM large language model, identifying potential safety hazards on the pre-processed multimodal data and the recognition and comparison results through a multimodal input algorithm, and outputting a comprehensive supervision report on multimodal hot work, including: Get the pre-built multimodal LLM large language model; Input preprocessed multimodal data; Extract the input multimodal fusion data feature matrix through the multimodal input algorithm , ， This includes text extraction feature vector , video extraction feature vector , image extraction feature vector , audio extraction feature vector ; Compare the input multimodal fusion data feature matrix with the standard multimodal fusion data feature matrix in the knowledge base to obtain similarity comparison results and output the hidden danger probability; Output a comprehensive supervision report on multi-modal hot work operations based on the probability of hidden dangers.

7. The method for automatically identifying hidden dangers of hot work in pipe gallery based on AI according to claim 6 is characterized in that: In the obtaining of the similarity comparison result, the similarity expression is calculated as: ， in, is the standard multimodal fusion data feature matrix, To adjust the amplitude.

8. An AI-based automatic identification device for hot work hazards in pipe gallery, applied to the AI-based automatic identification method for hot work hazards in pipe gallery as claimed in any one of claims 1 to 7, characterized in that: The device comprises: The data acquisition module is used to obtain data collected by explosion-proof cameras and environmental sensors in real time at the fire operation site in the pipeline corridor; A data processing module, used for preprocessing the collected multimodal data; The checkpoint recognition module is used to use the image detection and classification algorithm based on the diffusion model and large model distillation technology to identify and compare the checkpoints on the preprocessed image data to obtain the identification and comparison results; The hidden danger identification module is used to call the pre-built LLM large language model, identify potential safety hazards on the pre-processed multimodal data and the identification and comparison results through a multimodal input algorithm, and output a comprehensive supervision report on multimodal hot work; The early warning and decision-making module is used to display reports through a visualization platform based on the comprehensive supervision report, issue hidden danger warnings and response suggestions when the preset conditions are met, record the hidden danger handling situation, and optimize the LLM large language model.

9. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the AI-based automatic identification method for hidden dangers of fire operations in the pipeline corridor as described in any one of claims 1-7 when executing the instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and the program instructs the device to execute the AI-based automatic identification method for hidden dangers of hot work in a pipe gallery as described in any one of claims 1-7.

Images