Inspection robot safety warning method and system based on multi-modal data fusion

Abstract

The application provides a kind of based on multimodal data fusion's inspection robot safety early warning method and system, it is related to inspection early warning technical field, the method includes: the position information of robot is obtained;Through being set in multimodal sensor combination in inspection robot, inspection data is obtained, wherein, inspection data includes: harmful gas concentration data, real-time inspection video, environmental temperature data, environmental humidity data, environmental gas data, wind force data, inspection audio data and thermal imaging data;According to harmful gas concentration data, environmental temperature data, environmental humidity data and wind force data, determine harmful gas early warning coefficient;According to real-time inspection video, environmental gas data and thermal imaging data, determine fire source early warning coefficient;According to thermal imaging data and inspection audio data, determine equipment early warning coefficient;Generate early warning information.According to the application, the timeliness and accuracy of early warning are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of inspection early warning, and in particular to a safety early warning method and system for an inspection robot based on multi-modal data fusion. BACKGROUND

[0002] In related technologies, the environment status of inspection can be warned based on data collected by sensors arranged in an inspection robot, but the related technologies cannot distinguish cross interference (for example, hydrogen sulfide and sulfur dioxide exist at the same time) and have poor prediction ability, that is, the accuracy of the warning result cannot be guaranteed and the diffusion trend of pollution cannot be predicted according to environmental factors.

[0003] The information disclosed in the background section of this application is only intended to deepen the understanding of the general background of the present application, and should not be regarded as acknowledging or implying in any form that this information constitutes prior art known to those skilled in the art. SUMMARY

[0004] The present application provides a safety early warning method and system for an inspection robot based on multi-modal data fusion, which can solve the technical problems that the accuracy of the warning result cannot be guaranteed and the diffusion trend of pollution cannot be predicted according to environmental factors in related technologies.

[0005] According to a first aspect of the present application, a safety early warning method for an inspection robot based on multi-modal data fusion is provided, comprising:

[0006] obtaining position information of the robot;

[0007] at multiple times in an inspection cycle, obtaining inspection data through a multi-modal sensor combination arranged in the inspection robot, wherein the inspection data includes harmful gas concentration data, real-time inspection video, environmental temperature data, environmental humidity data, environmental gas data, wind power data, inspection audio data, and thermal imaging data;

[0008] determining a harmful gas warning coefficient according to the harmful gas concentration data, the environmental temperature data, the environmental humidity data, and the wind power data;

[0009] determining a fire source warning coefficient according to the real-time inspection video, the environmental gas data, and the thermal imaging data;

[0010] determining a device warning coefficient according to the thermal imaging data and the inspection audio data;

[0011] generating warning information according to the position information, the harmful gas warning coefficient, the fire source warning coefficient, and the device warning coefficient.

[0012] According to the present application, the harmful gas concentration data, the environmental temperature data, the environmental humidity data and the wind force data are used to determine a harmful gas early warning coefficient, which comprises:

[0013] The historical harmful gas concentration data is obtained.

[0014] The harmful gas concentration data and the historical harmful gas concentration data are used to determine a harmful gas concentration danger coefficient.

[0015] The harmful gas concentration data, the environmental temperature data, the environmental humidity data and the wind force data are used to determine a harmful gas diffusion danger coefficient.

[0016] The harmful gas concentration danger coefficient and the harmful gas diffusion danger coefficient are used to determine a harmful gas early warning coefficient.

[0017] According to the present application, the harmful gas concentration data and the historical harmful gas concentration data are used to determine a harmful gas concentration danger coefficient, which comprises:

[0018] The harmful gas concentration data and the historical harmful gas concentration data are used to determine a harmful gas concentration change amplitude.

[0019] The harmful gas concentration change amplitude and a preset harmful gas concentration change amplitude threshold value are used to determine a harmful gas concentration change danger identification result.

[0020] The harmful gas concentration data and a preset harmful gas concentration threshold value are used to determine a harmful gas concentration danger identification result.

[0021] The harmful gas concentration change danger identification result and the harmful gas concentration danger identification result are used to determine a harmful gas concentration danger coefficient.

[0022] According to the present application, the harmful gas concentration data, the environmental temperature data, the environmental humidity data and the wind force data are used to determine a harmful gas diffusion danger coefficient, which comprises:

[0023] The test harmful gas concentration data, the test environmental temperature data, the test environmental humidity data, the test wind force data and the test diffusion speed in a plurality of historical test periods are obtained.

[0024] The test harmful gas concentration data, the test environmental temperature data, the test environmental humidity data, the test wind force data and the test diffusion speed are used to determine a diffusion speed relationship function.

[0025] The diffusion speed relationship function, the harmful gas concentration data, the environmental temperature data, the environmental humidity data and the wind force data are used to determine a predicted diffusion speed.

[0026] According to the predicted diffusion speed, a harmful gas diffusion risk coefficient is determined.

[0027] According to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind force data and the test diffusion speed, a diffusion speed relationship function is determined, comprising:

[0028] According to the formula

[0029]

[0030] A diffusion speed relationship function is determined by a coefficient equation, wherein if is a conditional function, is the test diffusion speed of the ith harmful gas in the kth historical test period, is a preset diffusion speed threshold value, is the test wind force data in the kth historical test period, is a preset wind force data threshold value, is the test environment humidity data in the kth historical test period, is a preset environment humidity threshold value, is the test harmful gas concentration data of the ith harmful gas in the kth historical test period, is a preset harmful gas concentration data threshold value, is the test environment temperature data in the kth historical test period, is a preset environment temperature threshold value, 、 、 、 、 、 、 、 、 、 and is a coefficient to be determined;

[0031] According to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind force data and the test diffusion speed, the coefficient to be determined is solved to obtain a solution value of the coefficient to be determined;

[0032] According to the solution value of the coefficient to be determined and the coefficient equation, a diffusion speed relationship function is obtained.

[0033] According to the real-time inspection video, the environment gas data and the thermal imaging data, a fire source early warning coefficient is determined, comprising:

[0034] In the real-time inspection video, whether a fire source and smoke exist in an inspection site is identified by an image detection model, a fire source identification result and a smoke identification result are determined;

[0035] When the fire source identification result is 1, a fire source early warning coefficient is determined as 3;

[0036] When the fire source identification result is 0, a fire source early warning coefficient is determined according to the smoke identification result, the environmental gas data and the thermal imaging data.

[0037] According to the present application, when the fire source identification result is 0, a fire source early warning coefficient is determined according to the smoke identification result, the environmental gas data and the thermal imaging data, comprising:

[0038] According to the environmental gas data, carbon monoxide concentration and oxygen concentration are determined;

[0039] According to the thermal imaging data, regional temperature is determined;

[0040] Historical carbon monoxide concentration, historical oxygen concentration and historical regional temperature are acquired;

[0041] According to the carbon monoxide concentration, the oxygen concentration, the historical carbon monoxide concentration and the historical oxygen concentration, a regional gas abnormality result is determined;

[0042] According to the regional temperature and the historical regional temperature, a regional temperature abnormality result is determined;

[0043] According to the smoke identification result, the regional gas abnormality result and the regional temperature abnormality result, a fire source early warning coefficient is determined.

[0044] According to the present application, a device early warning coefficient is determined according to the thermal imaging data and the inspection audio data, comprising:

[0045] According to the thermal imaging data, device temperature data is determined;

[0046] According to the device temperature data and a set device temperature threshold value, a device temperature abnormality identification result is determined;

[0047] In a plurality of second historical test periods, device historical test running condition data of a test device is acquired, wherein the device historical test running condition data comprises historical device running efficiency, historical device vibration value and historical device working audio data;

[0048] Reference running efficiency, reference vibration value and normal working audio data of a test device are acquired;

[0049] determine a historical equipment normal operation coefficient according to the historical equipment operation efficiency, the historical equipment vibration value, the reference operation efficiency and the reference vibration value;

[0050] process the historical equipment working audio data according to the equipment operation condition prediction model, and obtain a training prediction equipment normal operation coefficient;

[0051] obtain a training loss function of the equipment operation condition prediction model according to the training prediction equipment normal operation coefficient, the historical equipment normal operation coefficient, the historical equipment working audio data and the normal working audio data;

[0052] train the equipment operation condition prediction model according to the training loss function of the equipment operation condition prediction model, and obtain a trained equipment operation condition prediction model;

[0053] process the inspection audio data according to the trained equipment operation condition prediction model, and determine a real-time equipment normal operation coefficient;

[0054] determine an equipment operation abnormality recognition result according to the real-time equipment normal operation coefficient;

[0055] determine an equipment early warning coefficient according to the equipment temperature abnormality recognition result and the equipment operation abnormality recognition result.

[0056] According to the present application, the training loss function of the equipment operation condition prediction model is obtained according to the training prediction equipment normal operation coefficient, the historical equipment normal operation coefficient, the historical equipment working audio data and the normal working audio data, which comprises:

[0057] determine a historical working root mean square value and a historical working peak factor according to the historical equipment working audio data;

[0058] determine a normal working root mean square value and a normal working peak factor according to the normal working audio data;

[0059] According to the formula

[0060]

[0061] determine the training loss function of the equipment operation condition prediction model wherein, is a training prediction equipment normal operation coefficient of a test equipment at a bth moment of a yth second historical test period, is a historical equipment normal operation coefficient of the test equipment at the bth moment of the yth second historical test period, is a historical working root mean square value of the test equipment at the bth moment of the yth second historical test period, a root mean square value of normal work of the testing device, a historical work peak factor of the testing device at the bth moment of the yth second historical test period, a normal work peak factor of the testing device, Y is the number of the second historical test periods, y≤Y, B is the number of the moments of the second historical test periods, b≤B, y, Y, b and B are all positive integers.

[0062] According to a second aspect of the present application, a safety warning system for a patrol robot based on multi-modal data fusion is provided, comprising:

[0063] a position information module, configured to acquire position information of the robot;

[0064] a patrol data module, configured to acquire patrol data through a combination of multi-modal sensors arranged in the patrol robot at multiple moments in a patrol period, wherein the patrol data comprises harmful gas concentration data, real-time patrol video, environmental temperature data, environmental humidity data, environmental gas data, wind force data, patrol audio data and thermal imaging data;

[0065] a gas warning module, configured to determine a harmful gas warning coefficient according to the harmful gas concentration data, the environmental temperature data, the environmental humidity data and the wind force data;

[0066] a fire source warning module, configured to determine a fire source warning coefficient according to the real-time patrol video, the environmental gas data and the thermal imaging data;

[0067] a device warning module, configured to determine a device warning coefficient according to the thermal imaging data and the patrol audio data;

[0068] a warning information module, configured to generate warning information according to the position information, the harmful gas warning coefficient, the fire source warning coefficient and the device warning coefficient.

[0069] Technical effects: According to the present application, multi-modal data fusion can be realized by the inspection robot, combined with real-time environment perception and abnormal pattern recognition, harmful gas hazard conditions, fire abnormal conditions and equipment abnormal working conditions can be effectively found, harmful gas warning coefficients, fire warning coefficients and equipment warning coefficients can be determined, and rapid accident warning and positioning can be realized, video information of safety hidden danger points can be uploaded to the management personnel, corresponding warning information can be generated, the timeliness and accuracy of the warning can be improved. When determining the diffusion velocity relationship function, the diffusion velocity relationship function can be determined according to the test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind data and test diffusion velocity, the influence of gas concentration, temperature, humidity and wind on diffusion velocity is accurately described, and the accuracy and objectivity of the diffusion velocity relationship function are improved. When determining the training loss function, the training loss function of the equipment operation condition prediction model can be obtained according to the training predicted equipment normal operation coefficient, historical equipment normal operation coefficient, historical equipment working audio data and normal working audio data. In the calculation process, the influence of the above data on the error of the training predicted equipment normal operation coefficient can be determined according to the root mean square value and the peak factor of the possible influence of the equipment working condition, and the influence and the relative error of the training predicted equipment normal operation coefficient are used to set the training loss function, so that the equipment operation condition prediction model in the training process makes the training loss function decrease, and the accuracy of the equipment operation condition prediction model is improved more targeted.

[0070] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. Other features and aspects of the present application will become more apparent from the following detailed description of exemplary embodiments, with reference to the accompanying drawings. BRIEF DESCRIPTION OF DRAWINGS

[0071] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed in the embodiments or the prior art description will be briefly introduced as follows. Obviously, the drawings in the following description are only some embodiments of the present application, and those skilled in the art can also obtain other embodiments from these drawings without creative labor;

[0072] Figure 1 Exemplarily, a flowchart of a safety warning method of an inspection robot based on multi-modal data fusion according to an embodiment of the present application is shown;

[0073] Figure 2 Exemplarily, a flowchart of determining a harmful gas warning coefficient according to an embodiment of the present application is shown;

[0074] Figure 3 Exemplarily, a flowchart of determining a fire warning coefficient according to an embodiment of the present application is shown;

[0075] Figure 4 A flow chart for determining a device early warning coefficient according to an embodiment of the present application is shown exemplarily.

[0076] Figure 5 A block diagram of a safety early warning system for a patrol robot based on multi-modal data fusion according to an embodiment of the present application is shown exemplarily. DETAILED DESCRIPTION

[0077] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0078] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and some embodiments can not be described again for the same or similar concepts or processes.

[0079] Figure 1 A flow chart of a safety early warning method for a patrol robot based on multi-modal data fusion according to an embodiment of the present application is shown exemplarily, and the method comprises:

[0080] Step S1, acquiring position information of a robot;

[0081] Step S2, acquiring patrol data through a multi-modal sensor combination arranged in the patrol robot at multiple time points in a patrol cycle, wherein the patrol data comprises harmful gas concentration data, real-time patrol video, environmental temperature data, environmental humidity data, environmental gas data, wind force data, patrol audio data and thermal imaging data;

[0082] Step S3, determining a harmful gas early warning coefficient according to the harmful gas concentration data, the environmental temperature data, the environmental humidity data and the wind force data;

[0083] Step S4, determining a fire source early warning coefficient according to the real-time patrol video, the environmental gas data and the thermal imaging data;

[0084] Step S5, determining a device early warning coefficient according to the thermal imaging data and the patrol audio data;

[0085] Step S6, generating early warning information according to the position information, the harmful gas early warning coefficient, the fire source early warning coefficient and the device early warning coefficient.

[0086] The safety warning method based on multi-modal data fusion of the inspection robot according to the embodiment of the present application can realize multi-modal data fusion through the inspection robot, combine real-time environment perception and abnormal pattern recognition, effectively find harmful gas hazard conditions, fire abnormal conditions and equipment working abnormal conditions, determine harmful gas warning coefficients, fire warning coefficients and equipment warning coefficients, and realize rapid accident warning and positioning, upload safety hazard point video information to managers, generate corresponding warning information, and improve the timeliness and accuracy of the warning.

[0087] According to an embodiment of the present application, in step S1, the position information of the robot is acquired.

[0088] For example, the environment map of the underground sewage treatment plant is constructed by using a laser radar or visual information, and the position information of the robot in the underground sewage treatment plant is determined.

[0089] According to an embodiment of the present application, in step S2, at multiple times in the inspection cycle, the inspection data is acquired through the multi-modal sensor combination arranged in the inspection robot, wherein the inspection data includes harmful gas concentration data, real-time inspection video, environment temperature data, environment humidity data, environment gas data, wind power data, inspection audio data and thermal imaging data.

[0090] For example, the multi-modal sensor combination includes a multi-channel gas sensor (which can be used to detect oxygen, carbon monoxide, oxygen, hydrogen sulfide, methane and ammonia), an infrared thermal imager, a thermal imaging dual-light camera, an acoustic sensor, a temperature and humidity sensor and an anemometer, and the inspection data is collected in real time during the inspection process through the multi-modal sensor combination.

[0091] According to an embodiment of the present application, in step S3, the harmful gas warning coefficient is determined according to the harmful gas concentration data, the environment temperature data, the environment humidity data and the wind power data.

[0092] Figure 2 An exemplary flowchart for determining the harmful gas warning coefficient according to an embodiment of the present application is shown.

[0093] According to an embodiment of the present application, step S3 includes:

[0094] Step S31, acquiring historical harmful gas concentration data;

[0095] Step S32, determining a harmful gas concentration danger coefficient according to the historical harmful gas concentration data and the harmful gas concentration data;

[0096] Step S33, determining a harmful gas diffusion risk coefficient according to the harmful gas concentration data, the environment temperature data, the environment humidity data and the wind force data;

[0097] Step S34, determining a harmful gas early warning coefficient according to the harmful gas concentration risk coefficient and the harmful gas diffusion risk coefficient.

[0098] For example, the harmful gas concentration data (e.g. methane concentration, hydrogen sulfide concentration and ammonia concentration, etc.) of each position in the last historical inspection period of the current inspection period is acquired, i.e. historical harmful gas concentration data; the harmful gas concentration condition of the current position is evaluated according to the harmful gas concentration data of various harmful gases at the current position of the inspection robot and the historical harmful gas concentration data of the current position, and the harmful gas concentration risk coefficient of various harmful gases at the current position is determined; the diffusion condition of the harmful gas is evaluated according to the harmful gas concentration data, the environment temperature data, the environment humidity data and the wind force data, and the harmful gas diffusion risk coefficient of various harmful gases at the current position is determined; and the harmful gas early warning coefficient is determined by adding the harmful gas concentration risk coefficient and the harmful gas diffusion risk coefficient.

[0099] According to one embodiment of the present application, step S32 comprises:

[0100] Step S321, determining a harmful gas concentration change amplitude according to the historical harmful gas concentration data and the harmful gas concentration data;

[0101] Step S322, determining a harmful gas concentration change risk identification result according to the harmful gas concentration change amplitude and a preset harmful gas concentration change amplitude threshold value;

[0102] Step S323, determining a harmful gas concentration risk identification result according to the harmful gas concentration data and a preset harmful gas concentration threshold value;

[0103] Step S324, determining a harmful gas concentration risk coefficient according to the harmful gas concentration change risk identification result and the harmful gas concentration risk identification result.

[0104] For example, according to the ratio of the difference between the harmful gas concentration data of the i-th harmful gas at the current position of the inspection robot and the historical harmful gas concentration data and the historical harmful gas concentration data of the i-th harmful gas, the harmful gas concentration change range of the i-th harmful gas is determined; if the harmful gas concentration change range of the i-th harmful gas is greater than or equal to a preset harmful gas concentration change range threshold (which can be set to 5%), it indicates that the harmful gas concentration of the i-th harmful gas at this position is significantly higher than that at the last inspection, and there may be a safety hazard, which needs to be warned, and the harmful gas concentration change risk identification result of the i-th harmful gas is 1, otherwise, the harmful gas concentration change risk identification result of the i-th harmful gas is 0; if the harmful gas concentration data of the i-th harmful gas is greater than or equal to a preset harmful gas concentration threshold (determined according to the type of harmful gas, for example, the preset harmful gas concentration threshold of hydrogen sulfide is 5ppm, and the preset harmful gas concentration threshold of methane is 1% LEL), it indicates that the i-th harmful gas will bring a safety hazard, which needs to be warned, and the harmful gas concentration risk identification result of the i-th harmful gas is 1, otherwise, the harmful gas concentration risk identification result of the i-th harmful gas is 0; the harmful gas concentration risk coefficient is determined by adding the harmful gas concentration change risk identification result and the harmful gas concentration risk identification result of the i-th harmful gas.

[0105] According to an embodiment of the present application, step S33 comprises:

[0106] Step S331, obtaining test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind speed data and test diffusion speed in a plurality of historical test periods;

[0107] Step S332, determining a diffusion speed relationship function according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind speed data and the test diffusion speed;

[0108] Step S333, determining a predicted diffusion speed according to the diffusion speed relationship function, the harmful gas concentration data, the environment temperature data, the environment humidity data and the wind speed data;

[0109] Step S334, determining a harmful gas diffusion risk coefficient according to the predicted diffusion speed.

[0110] For example, in a historical test period, the influence of environmental factors and harmful gas concentration on the diffusion speed of harmful gas is tested in a similar experimental environment of an underground sewage treatment plant to obtain test harmful gas concentration data, test environmental temperature data, test environmental humidity data, test wind data and test diffusion speed; harmful gas concentration data, environmental temperature data, environmental humidity data and wind data will all affect the diffusion speed to some extent, for example, when the wind data is large, the diffusion speed is fast, and based on the correlation of the above data, the diffusion speed relationship function between the test harmful gas concentration data, the test environmental temperature data, the test environmental humidity data, the test wind data and the test diffusion speed can be determined; in the conventional technology, the diffusion speed of harmful gas can be detected in real time by setting a gas sensor array, but due to the large range of the underground sewage treatment plant, the cost of setting the gas sensor array is too high, and due to the poor environmental conditions of the underground sewage treatment plant, the gas sensor needs to be maintained, therefore, the predicted diffusion speed can be determined by fitting to improve the efficiency of harmful gas diffusion speed detection, harmful gas concentration data, environmental temperature data, environmental humidity data and wind data in the current inspection period are substituted into the diffusion speed relationship function to determine the predicted diffusion speed of various harmful gases at the current position; according to the type and predicted diffusion speed of the harmful gas, the harmful gas diffusion danger coefficient is determined, for example, when the harmful gas is hydrogen sulfide, when the predicted diffusion speed is greater than or equal to 0.5 ppm / min, the harmful gas diffusion danger coefficient corresponding to the harmful gas is 1, indicating that the harmful gas diffuses too fast, otherwise, the harmful gas diffusion danger coefficient corresponding to the harmful gas is 0, when the harmful gas is methane, when the predicted diffusion speed is greater than or equal to 200 ppm / min, the harmful gas diffusion danger coefficient corresponding to the harmful gas is 1, indicating that the harmful gas diffuses too fast, otherwise, the harmful gas diffusion danger coefficient corresponding to the harmful gas is 0.

[0111] According to one embodiment of the present application, step S332 comprises determining the undetermined coefficient equation of the diffusion speed relationship function according to formula (1),

[0112] (1)

[0113] wherein if is a conditional function, is the test diffusion speed of the i-th harmful gas in the k-th historical test period, is a preset diffusion speed threshold, is the test wind data in the k-th historical test period, is a preset wind data threshold, is the test environmental humidity data in the k-th historical test period, is a preset environmental humidity threshold, test harmful gas concentration data of the ith harmful gas in the kth historical test cycle, preset harmful gas concentration data threshold, test ambient temperature data in the kth historical test cycle, preset ambient temperature threshold, , , , , , , , , , and are undetermined coefficients;

[0114] According to the test harmful gas concentration data, the test ambient temperature data, the test ambient humidity data, the test wind force data and the test diffusion speed, the undetermined coefficients are solved to obtain a solution value of the undetermined coefficients;

[0115] According to the solution value of the undetermined coefficients and the undetermined coefficient equation, a diffusion speed relationship function is obtained.

[0116] According to one embodiment of the present application, test diffusion speed of the ith harmful gas in the kth historical test cycle and preset diffusion speed threshold, which can be set to 1 ppm / min, a ratio of test wind force data in the kth historical test cycle to preset wind force data threshold, which can be set to 1 m / s, a ratio of test ambient temperature data in the kth historical test cycle to preset ambient temperature threshold, which can be set to 25 degrees Celsius, a ratio of test ambient humidity data in the kth historical test cycle to preset ambient humidity threshold, which can be set to 75%, a ratio of test harmful gas concentration data of the ith harmful gas in the kth historical test cycle to preset harmful gas concentration data threshold, which is set according to the type of harmful gas, for example, the value corresponding to hydrogen sulfide is 100 ppm, , , , and are all dimensionless values.

[0117] According to one embodiment of the present application, represents that the test wind data in the kth historical test period has a positive correlation with the test diffusion speed of the ith harmful gas in the kth historical test period, and the greater the wind, the faster the air flow, and the greater the diffusion speed of the harmful gas, represents that the test ambient temperature data in the kth historical test period has a positive correlation with the test diffusion speed of the ith harmful gas in the kth historical test period, and the higher the temperature, the faster the intermolecular motion, and the greater the diffusion speed of the harmful gas, represents that the test ambient humidity data in the kth historical test period has a negative correlation with the test diffusion speed of the ith harmful gas in the kth historical test period, and the higher the humidity, the faster the gas dissolution speed, thereby limiting the diffusion of the gas, and the smaller the diffusion speed of the harmful gas.

[0118] According to one embodiment of the present application, in formula (1), the value of the conditional function includes the following two cases: when the condition of is met, the value of the conditional function is , which indicates that the harmful gas concentration is too high to form a plume effect, and the higher the gas concentration, the slower the diffusion speed, which indicates that under the condition of a higher harmful concentration gas, the test harmful gas concentration data of the ith harmful gas in the kth historical test period has a negative correlation with the test diffusion speed, and when the condition of is not met, the value of the conditional function is , which indicates that under the condition of a lower harmful concentration gas, the diffusion speed has a positive correlation with the concentration, and the harmful gas will diffuse from a position with a higher concentration to a position with a lower concentration. Based on the above relationship, the undetermined coefficient equation of the diffusion speed relationship function can be obtained.

[0119] According to one embodiment of the present application, the above undetermined coefficient equation involves multiple parameters, that is, the test harmful gas concentration data, the test ambient temperature data, the test ambient humidity data, the test wind data and the test diffusion speed in at least 11 historical test periods are fitted to solve the above multiple undetermined coefficients. There are 11 undetermined coefficients, that is, , , , , , , , , , and Solving the 11 undetermined coefficients according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind force data and the test diffusion speed, obtaining the solution values of the 11 undetermined coefficients, and substituting the solution values of the 11 undetermined coefficients into the undetermined coefficient equation to obtain the diffusion speed relationship function.

[0120] In this way, the diffusion speed relationship function can be determined according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind force data and the test diffusion speed, the influence of the gas concentration, the temperature, the humidity and the wind force on the diffusion speed is accurately described, and the accuracy and objectivity of the diffusion speed relationship function are improved.

[0121] According to an embodiment of the present application, in step S4, a fire source early warning coefficient is determined according to the real-time inspection video, the environment gas data and the thermal imaging data.

[0122] Figure 3 An exemplary flow chart for determining a fire source early warning coefficient according to an embodiment of the present application is shown.

[0123] According to an embodiment of the present application, step S4 includes:

[0124] In step S41, whether there is a fire source and smoke in the inspection site is identified by an image detection model in the real-time inspection video, a fire source identification result and a smoke identification result are determined;

[0125] In step S42, when the fire source identification result is 1, the fire source early warning coefficient is determined as 3;

[0126] In step S43, when the fire source identification result is 0, the fire source early warning coefficient is determined according to the smoke identification result, the environment gas data and the thermal imaging data.

[0127] For example, the image detection model is a deep learning neural network model such as a convolutional neural network model, which can be trained according to historical data to enable the image detection model to identify whether there is a flame or smoke in an image, determine the inspection images at each time of the inspection period according to the real-time inspection video, identify whether there is a fire source and smoke in the inspection site according to the image detection model to determine a fire source identification result and a smoke identification result, when there is a fire source, the fire source identification result is 1, otherwise, the fire source identification result is 0, when there is smoke, the smoke identification result is 1, otherwise, the smoke identification result is 0; when the fire source identification result is 1, it indicates that there is an open flame, and the fire source early warning coefficient is determined as 3; when the fire source identification result is 0, it indicates that there is no open flame, and the fire source early warning coefficient is determined according to the smoke identification result, the environment gas data and the thermal imaging data.

[0128] According to one embodiment of the present application, step S43 comprises:

[0129] Step S431, according to the ambient gas data, determine carbon monoxide concentration and oxygen concentration;

[0130] Step S432, according to the thermal imaging data, determine area temperature;

[0131] Step S433, obtain historical carbon monoxide concentration, historical oxygen concentration and historical area temperature;

[0132] Step S434, according to the carbon monoxide concentration, the oxygen concentration, the historical carbon monoxide concentration, the historical oxygen concentration, determine area gas anomaly result;

[0133] Step S435, according to the area temperature and the historical area temperature, determine area temperature anomaly result;

[0134] Step S436, according to the smoke identification result, the area gas anomaly result and the area temperature anomaly result, determine fire source early warning coefficient.

[0135] For example, according to the multi-channel gas sensor to detect carbon monoxide concentration and oxygen concentration; through the infrared thermal imager to detect the area temperature; obtain the carbon monoxide concentration, oxygen concentration and area temperature of each position in multiple historical inspection periods, and according to the carbon monoxide concentration, oxygen concentration and area temperature in multiple historical inspection periods to determine the historical carbon monoxide concentration, historical oxygen concentration and historical area temperature, which represent the normal carbon monoxide concentration, normal oxygen concentration and normal area temperature of each position; if the carbon monoxide concentration is greater than the historical carbon monoxide concentration, and the oxygen concentration is less than the historical oxygen concentration, it indicates that there may be a burning phenomenon, and the area gas anomaly result is 1, otherwise the area gas anomaly result is 0; if the area temperature is much greater than the historical area temperature (more than 40% of the historical area temperature), such as the area temperature is 70 degrees Celsius and the historical area temperature is 50 degrees Celsius, it indicates that there may be a burning phenomenon, and the area temperature anomaly result is 1, otherwise the area temperature anomaly result is 0; according to the sum of the smoke identification result, the area gas anomaly result and the area temperature anomaly result, determine the fire source early warning coefficient, such as when the smoke identification result, the area gas anomaly result and the area temperature anomaly result are all 1, it indicates that there is a certain phenomenon of burning of hidden fire, and the fire source early warning coefficient is 3, the greater the sum of the smoke identification result, the area gas anomaly result and the area temperature anomaly result, the greater the possibility of burning of hidden fire, and the greater the fire source early warning coefficient.

[0136] According to one embodiment of the present application, in step S5, according to the thermal imaging data and the inspection audio data, determine the equipment early warning coefficient.

[0137] Figure 4 An exemplary flowchart of determining a device early warning coefficient according to an embodiment of the present application is shown.

[0138] According to an embodiment of the present application, step S5 comprises:

[0139] Step S51, determining device temperature data according to the thermal imaging data;

[0140] Step S52, determining a device temperature anomaly recognition result according to the device temperature data and a set device temperature threshold value;

[0141] Step S53, obtaining device historical test running condition data of the test device in a plurality of second historical test periods, wherein the device historical test running condition data comprises: historical device running efficiency, historical device vibration value and historical device working audio data;

[0142] Step S54, obtaining reference running efficiency, reference vibration value and normal working audio data of the test device;

[0143] Step S55, determining a historical device normal running coefficient according to the historical device running efficiency, the historical device vibration value, the reference running efficiency and the reference vibration value;

[0144] Step S56, processing the historical device working audio data according to a device running condition prediction model to obtain a training predicted device normal running coefficient;

[0145] Step S57, obtaining a training loss function of the device running condition prediction model according to the training predicted device normal running coefficient, the historical device normal running coefficient, the historical device working audio data and the normal working audio data;

[0146] Step S58, training the device running condition prediction model according to the training loss function of the device running condition prediction model to obtain a trained device running condition prediction model;

[0147] Step S59, processing the inspection audio data according to the trained device running condition prediction model to determine a real-time device normal running coefficient;

[0148] Step S510, determining a device running anomaly recognition result according to the real-time device normal running coefficient;

[0149] Step S511, determining a device early warning coefficient according to the device temperature anomaly recognition result and the device running anomaly recognition result.

[0150] For example, the device temperature data of the working equipment in the underground sewage treatment plant is detected by an infrared thermal imager. If the device temperature data is greater than or equal to a set device temperature threshold (which can be set to 70 degrees Celsius), it indicates that there is an electrical fire safety hazard, and the device temperature anomaly recognition result is 1, otherwise, the device temperature anomaly recognition result is 0; in the second historical test period, the same type of equipment as the equipment in the underground sewage treatment plant is selected as the test equipment, and the historical equipment running efficiency (the ratio of effective output power to total input power), the historical equipment vibration value (for example, the equipment vibration value is 2mm / s) and the historical equipment working audio data of the test equipment in the second historical test period are collected; the reference running efficiency, the reference vibration value when the equipment is just out of the factory state are collected, and the normal working audio data of the test equipment when it is working normally is collected; the running efficiency difference value is determined according to the historical equipment running efficiency minus the reference running efficiency, the vibration value difference value is determined according to the reference vibration value minus the historical equipment vibration value, and the historical equipment normal running coefficient is determined according to the sum of the ratio of the running efficiency difference value to the reference running efficiency and the ratio of the vibration value difference value to the reference vibration value. The larger the historical equipment normal running coefficient is, the higher the running efficiency of the test equipment in the second historical test period is, the lower the vibration value is, and the better the running condition is; the historical equipment working audio data is processed according to the equipment running condition prediction model to obtain the training prediction equipment normal running coefficient; the training loss function of the equipment running condition prediction model is obtained according to the training prediction equipment normal running coefficient, the historical equipment normal running coefficient, the historical equipment working audio data and the normal working audio data; the equipment running condition prediction model is trained according to the training loss function of the equipment running condition prediction model, so that the prediction accuracy of the equipment running condition prediction model is improved, and the trained equipment running condition prediction model is obtained; the real-time equipment normal running coefficient is determined by processing the inspection audio data according to the trained equipment running condition prediction model; if the real-time equipment normal running coefficient is less than 0, it indicates that the running efficiency or vibration of the equipment is abnormal, and the equipment running anomaly recognition result is determined to be 1, otherwise, the equipment running anomaly recognition result is 0; the device warning coefficient is determined by summing the device temperature anomaly recognition result and the device running anomaly recognition result.

[0151] According to one embodiment of the present application, step S67 comprises:

[0152] According to the historical equipment working audio data, the historical working root mean square value and the historical working peak factor are determined;

[0153] According to the normal working audio data, the normal working root mean square value and the normal working peak factor are determined;

[0154] The training loss function of the equipment running condition prediction model is determined according to formula (2) ,

[0155] (2)

[0156] wherein, is a training prediction device normal operation coefficient of the test device at the bth moment of the yth second historical test period, is a historical device normal operation coefficient of the test device at the bth moment of the yth second historical test period, is a historical working root mean square value of the test device at the bth moment of the yth second historical test period, is a normal working root mean square value of the test device, is a historical working peak factor of the test device at the bth moment of the yth second historical test period, is a normal working peak factor of the test device, Y is the number of second historical test periods, y≤Y, B is the number of moments of the second historical test period, b≤B, y, Y, b and B are all positive integers.

[0157] According to one embodiment of the present application, time-frequency domain feature extraction is performed on historical device working audio data and normal working audio data, and historical working root mean square value and historical working peak factor, and normal working root mean square value and normal working peak factor are determined respectively.

[0158] According to one embodiment of the present application, is a ratio of the historical working root mean square value to the normal working root mean square value of the test device at the bth moment of the yth second historical test period, the larger the ratio, the larger the historical working root mean square value, is a ratio of the historical working peak factor to the normal working peak factor of the test device at the bth moment of the yth second historical test period, the larger the ratio, the larger the historical working peak factor, indicates that the root mean square value and the peak factor are negatively correlated with the training prediction device normal operation coefficient, for example, the larger the root mean square value, the larger the overall sound energy intensity, indicating that there is serious friction or looseness, resulting in a decrease in device working efficiency and the generation of abnormal vibration values, the smaller the training prediction device normal operation coefficient, the larger the peak factor, indicating that there is a greater possibility of impact failure (such as bearing ball damage, blade fracture impact), resulting in a decrease in device working efficiency and the generation of abnormal vibration values, and the smaller the training prediction device normal operation coefficient.

[0159] According to one embodiment of the present application, is a relative error of the training prediction device normal operation coefficient to the historical device normal operation coefficient of the test device at the bth moment of the yth second historical test period, and the relative error is calculated by The relative errors of the training prediction equipment normal operation coefficients and the historical equipment normal operation coefficients of the test equipment at the bth moment of the yth second historical test period are weighted and averaged to obtain a training loss function, and in the training process, the training loss function is reduced, so that the relative errors of the training prediction equipment normal operation coefficients and the historical equipment normal operation coefficients are reduced, the accuracy of the equipment operation condition prediction model for equipment operation condition prediction is improved, and the accuracy of the equipment operation condition prediction model is improved.

[0160] In this way, the training loss function of the equipment operation condition prediction model can be obtained according to the training prediction equipment normal operation coefficients, the historical equipment normal operation coefficients, the historical equipment working audio data and the normal working audio data. In the calculation process, the influence of the above data on the error of the training prediction equipment normal operation coefficients can be determined according to the root mean square value and the peak factor, and the training loss function is set based on the influence and the relative error of the training prediction equipment normal operation coefficients, so that the equipment operation condition prediction model reduces the training loss function in the training process, and more accurately improves the accuracy of the equipment operation condition prediction model.

[0161] According to one embodiment of the present application, in step S6, the warning information is generated according to the position information, the harmful gas warning coefficient, the fire source warning coefficient and the equipment warning coefficient.

[0162] For example, when the harmful gas warning coefficient is equal to 3, it indicates that the harmful gas hazard condition at the position is serious, and the highest warning information is generated, when the harmful gas warning coefficient is equal to 2 and 1, it indicates that there is a certain harmful gas safety hazard at the position, when the harmful gas warning coefficient is equal to 0, it indicates that there is no harmful gas safety hazard, and according to the position information of the inspection robot, the corresponding warning information is generated, such as A position exists the highest safety warning of the i-th harmful gas; when the fire source warning coefficient is equal to 3, it indicates that there is a hidden fire or an open fire, when the fire source warning coefficient is less than 3 and greater than 0, it indicates that there is a possibility of hidden fire burning, when the fire source warning coefficient is equal to 0, it indicates that there is no fire condition, and according to the position information of the inspection robot, the corresponding warning information is generated, such as A position exists an open fire; when the equipment warning coefficient is greater than 0, it indicates that the temperature of the equipment is abnormal or the equipment works abnormally, and according to the position information of the inspection robot, the corresponding warning information is generated, such as A position of the working equipment has abnormal temperature.

[0163] The safety warning method of the inspection robot based on multi-modal data fusion according to the embodiment of the present application can realize multi-modal data fusion through the inspection robot, effectively find harmful gas hazard conditions, fire abnormal conditions and equipment working abnormal conditions by combining real-time environment perception and abnormal pattern recognition, determine harmful gas warning coefficients, fire warning coefficients and equipment warning coefficients, and realize rapid accident warning and positioning, upload safety hazard point video information to managers, generate corresponding warning information, and improve the timeliness and accuracy of the warning. When determining the diffusion speed relationship function, the diffusion speed relationship function can be determined according to the test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind power data and test diffusion speed, which accurately describes the influence of gas concentration, temperature, humidity and wind power on the diffusion speed, and improves the accuracy and objectivity of the diffusion speed relationship function. When determining the training loss function, the training loss function of the equipment operation condition prediction model can be obtained according to the training predicted equipment normal operation coefficient, historical equipment normal operation coefficient, historical equipment working audio data and normal working audio data. In the calculation process, the influence of the above data on the error of the training predicted equipment normal operation coefficient can be determined according to the root mean square value and the peak factor on the possible influence of the equipment working condition, and the influence and the relative error of the training predicted equipment normal operation coefficient are used to set the training loss function, so that the equipment operation condition prediction model reduces the training loss function in the training process, and more accurately improves the accuracy of the equipment operation condition prediction model.

[0164] Figure 5 An example of a block diagram of a safety warning system of an inspection robot based on multi-modal data fusion according to an embodiment of the present application is shown, the system comprising:

[0165] A position information module for obtaining position information of the robot;

[0166] An inspection data module for obtaining inspection data at multiple times in an inspection period through a multi-modal sensor combination arranged in the inspection robot, wherein the inspection data includes harmful gas concentration data, real-time inspection video, environment temperature data, environment humidity data, environment gas data, wind power data, inspection audio data and thermal imaging data;

[0167] A gas warning module for determining a harmful gas warning coefficient according to the harmful gas concentration data, the environment temperature data, the environment humidity data and the wind power data;

[0168] A fire warning module for determining a fire warning coefficient according to the real-time inspection video, the environment gas data and the thermal imaging data;

[0169] The device early warning module is configured to determine a device early warning coefficient according to the thermal imaging data and the patrol audio data.

[0170] The early warning information module is configured to generate early warning information according to the position information, the harmful gas early warning coefficient, the fire source early warning coefficient, and the device early warning coefficient.

[0171] The present application can be a method, an apparatus, a system, and / or a computer program product. The computer program product can include a computer readable storage medium having computer readable program instructions stored therein for executing various aspects of the present application.

[0172] Those skilled in the art will understand that the above description and the embodiments of the present application shown in the drawings are only examples and do not limit the present application. The purpose of the present application has been fully and effectively achieved. The functional and structural principles of the present application have been demonstrated and described in the embodiments, and the embodiments of the present application can be modified or changed in any way without departing from the principles.

Claims

1. A safety warning method for inspection robots based on multimodal data fusion, characterized in that: include: Get the robot's location information; At multiple moments during the inspection cycle, inspection data is acquired through a combination of multimodal sensors installed in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind speed data, inspection audio data, and thermal imaging data; Determining a harmful gas warning coefficient according to the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data includes: Obtain historical harmful gas concentration data; determining a hazardous gas concentration risk factor based on the historical hazardous gas concentration data and the hazardous gas concentration data; Determining a hazardous gas diffusion risk coefficient based on the hazardous gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data includes: Obtain test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind speed data and test diffusion speed data in multiple historical test cycles; Determining a diffusion velocity relationship function according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind speed data, and the test diffusion velocity includes: According to the formula Determine the unknown coefficient equation of the diffusion velocity relationship function, where if is the conditional function, is the test diffusion rate of the i-th harmful gas in the k-th historical test cycle, is the preset diffusion speed threshold, is the test wind data in the kth historical test cycle, To preset the wind data threshold, is the test environment humidity data in the kth historical test cycle, To preset the ambient humidity threshold, is the test harmful gas concentration data of the i-th harmful gas in the k-th historical test cycle, To preset the harmful gas concentration data threshold, is the test environment temperature data in the kth historical test cycle, is the preset ambient temperature threshold, 、 、 、 、 、 、 、 、 、 and is the coefficient to be determined; Solving the undetermined coefficients according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind force data, and the test diffusion velocity to obtain solution values ​​of the undetermined coefficients; Obtaining a diffusion velocity relationship function according to the solved value of the undetermined coefficient and the undetermined coefficient equation; Determining a predicted diffusion speed based on the diffusion speed relationship function, the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; Determining a hazardous gas diffusion risk factor based on the predicted diffusion rate; Determining a harmful gas warning coefficient based on the harmful gas concentration hazard coefficient and the harmful gas diffusion hazard coefficient; Determining a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data; Determining an equipment warning coefficient based on the thermal imaging data and the inspection audio data; Warning information is generated based on the location information, the harmful gas warning coefficient, the fire source warning coefficient and the equipment warning coefficient.

2. The inspection robot safety early warning method based on multimodal data fusion according to claim 1 is characterized in that: Determining a hazardous gas concentration risk factor based on the historical hazardous gas concentration data and the hazardous gas concentration data includes: determining a change range of harmful gas concentration based on the historical harmful gas concentration data and the harmful gas concentration data; Determining a hazardous gas concentration change hazard identification result based on the hazardous gas concentration change amplitude and a preset hazardous gas concentration change amplitude threshold; Determining a hazardous gas concentration hazard identification result based on the hazardous gas concentration data and a preset hazardous gas concentration threshold; A harmful gas concentration hazard coefficient is determined according to the harmful gas concentration change hazard identification result and the harmful gas concentration hazard identification result.

3. The inspection robot safety early warning method based on multimodal data fusion according to claim 1 is characterized in that: Determining a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data includes: In the real-time inspection video, an image detection model is used to identify whether there is a fire source and smoke at the inspection site, and a fire source identification result and a smoke identification result are determined; When the fire source identification result is 1, the fire source warning coefficient is determined to be 3; When the fire source identification result is 0, a fire source warning coefficient is determined according to the smoke identification result, the ambient gas data and the thermal imaging data.

4. The inspection robot safety early warning method based on multimodal data fusion according to claim 3 is characterized in that: When the fire source identification result is 0, determining a fire source warning coefficient according to the smoke identification result, the ambient gas data, and the thermal imaging data includes: determining carbon monoxide concentration and oxygen concentration according to the ambient gas data; determining a regional temperature based on the thermal imaging data; Get historical carbon monoxide concentration, historical oxygen concentration and historical regional temperature; Determining regional gas anomaly results based on the carbon monoxide concentration, the oxygen concentration, the historical carbon monoxide concentration, and the historical oxygen concentration; Determining a regional temperature anomaly result based on the regional temperature and the historical regional temperature; A fire source warning coefficient is determined based on the smoke identification result, the regional gas anomaly result, and the regional temperature anomaly result.

5. The inspection robot safety early warning method based on multimodal data fusion according to claim 1 is characterized in that: Determining a device warning coefficient based on the thermal imaging data and the inspection audio data includes: Determining device temperature data based on the thermal imaging data; Determining a device temperature anomaly identification result based on the device temperature data and a set device temperature threshold; Acquiring historical test operation status data of the test equipment during a plurality of second historical test cycles, wherein the historical test operation status data includes: historical equipment operation efficiency, historical equipment vibration value, and historical equipment working audio data; Obtain the benchmark operating efficiency, benchmark vibration value and normal operating audio data of the test equipment; Determining a historical equipment normal operation coefficient based on the historical equipment operating efficiency, the historical equipment vibration value, the benchmark operating efficiency, and the benchmark vibration value; Processing the historical equipment working audio data according to the equipment operating status prediction model to obtain a training prediction equipment normal operating coefficient; Obtaining a training loss function of a device operation status prediction model based on the training prediction device normal operation coefficient, the historical device normal operation coefficient, the historical device operation audio data, and the normal operation audio data; Training the device operation status prediction model according to the training loss function of the device operation status prediction model to obtain a trained device operation status prediction model; Processing the inspection audio data according to the trained equipment operation status prediction model to determine the real-time equipment normal operation coefficient; Determining an equipment operation abnormality identification result based on the real-time equipment normal operation coefficient; An equipment warning coefficient is determined based on the equipment temperature anomaly identification result and the equipment operation anomaly identification result.

6. The inspection robot safety early warning method based on multimodal data fusion according to claim 5 is characterized in that: Obtaining a training loss function of a device operation status prediction model based on the training prediction device normal operation coefficient, the historical device normal operation coefficient, the historical device operation audio data, and the normal operation audio data, including: Determining a historical operating root mean square value and a historical operating peak factor based on the historical device operating audio data; Determining a normal working root mean square value and a normal working peak factor based on the normal working audio data; According to the formula Determine the training loss function for the equipment health prediction model ,in, The normal operation coefficient of the training prediction equipment at the bth moment of the yth second historical test cycle is, is the historical equipment normal operation coefficient of the test equipment at the bth moment in the yth second historical test cycle, is the historical operating RMS value of the test equipment at the bth moment of the yth second historical test cycle, is the RMS value of the normal operation of the test equipment, is the historical operating peak factor of the test equipment at the bth moment in the yth second historical test cycle, is the normal working peak factor of the test equipment, Y is the number of the second historical test cycle, y≤Y, B is the number of moments in the second historical test cycle, b≤B, y, Y, b and B are all positive integers.

7. A patrol robot safety warning system based on multimodal data fusion for executing the method according to any one of claims 1 to 6, characterized in that: include: Position information module, used to obtain the robot's position information; An inspection data module is used to obtain inspection data at multiple moments during the inspection cycle through a combination of multimodal sensors installed in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind speed data, inspection audio data, and thermal imaging data; a gas warning module, configured to determine a harmful gas warning coefficient based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; A fire source warning module, configured to determine a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data; An equipment warning module, configured to determine an equipment warning coefficient based on the thermal imaging data and the inspection audio data; The early warning information module is used to generate early warning information based on the location information, the harmful gas early warning coefficient, the fire source early warning coefficient and the equipment early warning coefficient.

Images