Inspection robot safety early warning method and system based on multi-modal data fusion

Abstract

The invention provides an inspection robot safety early warning method and system based on multi-modal data fusion, and relates to the technical field of inspection early warning, and the method comprises the steps: obtaining the position information of a robot; the method comprises the steps that inspection data are obtained through a multi-mode sensor combination arranged in an inspection robot, and the inspection data comprise harmful gas concentration data, real-time inspection videos, environment temperature data, environment humidity data, environment gas data, wind power data, inspection audio data and thermal imaging data; determining a harmful gas early warning coefficient according to the harmful gas concentration data, the environment temperature data, the environment humidity data and the wind power data; determining a fire source early warning coefficient according to the real-time inspection video, the environmental gas data and the thermal imaging data; determining an equipment early warning coefficient according to the thermal imaging data and the inspection audio data; and generating early warning information. According to the invention, the timeliness and accuracy of early warning are improved.

Description

Technical Field

[0001] The present invention relates to the field of patrol inspection and early warning technology, and in particular to a patrol robot safety early warning method and system based on multimodal data fusion. Background Art

[0002] In related technologies, early warnings can be issued for environmental conditions during inspections based on data collected by sensors installed in inspection robots. However, related technologies cannot distinguish cross-interference (for example, the simultaneous presence of hydrogen sulfide and sulfur dioxide) and have poor predictive capabilities. That is, the accuracy of the early warning results cannot be guaranteed, and the diffusion trend of pollution cannot be predicted based on environmental factors.

[0003] The information disclosed in the background technology section of this application is only intended to deepen the understanding of the general background technology of this application, and should not be regarded as an admission or any form of suggestion that the information constitutes the prior art already known to those skilled in the art. Summary of the Invention

[0004] The present invention provides a patrol robot safety early warning method and system based on multimodal data fusion, which can solve the technical problems that related technologies cannot guarantee the accuracy of early warning results and cannot predict the diffusion trend of pollution based on environmental factors.

[0005] According to a first aspect of the present invention, a safety early warning method for an inspection robot based on multimodal data fusion is provided, comprising: Get the robot's location information; At multiple moments during the inspection cycle, inspection data is acquired through a combination of multimodal sensors installed in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind speed data, inspection audio data, and thermal imaging data; determining a harmful gas warning coefficient according to the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; Determining a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data; Determining an equipment warning coefficient based on the thermal imaging data and the inspection audio data; Warning information is generated based on the location information, the harmful gas warning coefficient, the fire source warning coefficient and the equipment warning coefficient.

[0006] According to the present invention, determining a harmful gas warning coefficient based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data includes: Obtain historical harmful gas concentration data; determining a hazardous gas concentration risk factor based on the historical hazardous gas concentration data and the hazardous gas concentration data; determining a hazardous gas diffusion risk coefficient based on the hazardous gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; A harmful gas warning coefficient is determined based on the harmful gas concentration hazard coefficient and the harmful gas diffusion hazard coefficient.

[0007] According to the present invention, determining the hazardous gas concentration risk factor based on the historical hazardous gas concentration data and the hazardous gas concentration data includes: determining a change range of harmful gas concentration based on the historical harmful gas concentration data and the harmful gas concentration data; Determining a hazardous gas concentration change hazard identification result based on the hazardous gas concentration change amplitude and a preset hazardous gas concentration change amplitude threshold; Determining a hazardous gas concentration hazard identification result based on the hazardous gas concentration data and a preset hazardous gas concentration threshold; A harmful gas concentration hazard coefficient is determined according to the harmful gas concentration change hazard identification result and the harmful gas concentration hazard identification result.

[0008] According to the present invention, determining the harmful gas diffusion risk coefficient based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data includes: Obtain test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind speed data and test diffusion speed data in multiple historical test cycles; Determining a diffusion velocity relationship function according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind speed data, and the test diffusion velocity; Determining a predicted diffusion speed based on the diffusion speed relationship function, the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; The hazardous gas diffusion risk coefficient is determined based on the predicted diffusion speed.

[0009] According to the present invention, determining a diffusion velocity relationship function based on the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind speed data, and the test diffusion velocity includes: According to the formula

[0010] Determine the unknown coefficient equation of the diffusion velocity relationship function, where if is the conditional function, is the test diffusion rate of the i-th harmful gas in the k-th historical test cycle, is the preset diffusion speed threshold, is the test wind data in the kth historical test cycle, To preset the wind data threshold, is the test environment humidity data in the kth historical test cycle, To preset the ambient humidity threshold, is the test harmful gas concentration data of the i-th harmful gas in the k-th historical test cycle, To preset the harmful gas concentration data threshold, is the test environment temperature data in the kth historical test cycle, is the preset ambient temperature threshold, 、 、 、 、 、 、 、 、 、 and is the coefficient to be determined; Solving the undetermined coefficients according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind force data, and the test diffusion velocity to obtain solution values ​​of the undetermined coefficients; A diffusion velocity relationship function is obtained according to the solved value of the undetermined coefficient and the undetermined coefficient equation.

[0011] According to the present invention, determining a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data includes: In the real-time inspection video, an image detection model is used to identify whether there is a fire source and smoke at the inspection site, and a fire source identification result and a smoke identification result are determined; When the fire source identification result is 1, the fire source warning coefficient is determined to be 3; When the fire source identification result is 0, a fire source warning coefficient is determined according to the smoke identification result, the ambient gas data and the thermal imaging data.

[0012] According to the present invention, when the fire source identification result is 0, determining the fire source warning coefficient based on the smoke identification result, the ambient gas data, and the thermal imaging data includes: determining carbon monoxide concentration and oxygen concentration based on the ambient gas data; determining a regional temperature based on the thermal imaging data; Get historical carbon monoxide concentration, historical oxygen concentration and historical regional temperature; Determining regional gas anomaly results based on the carbon monoxide concentration, the oxygen concentration, the historical carbon monoxide concentration, and the historical oxygen concentration; Determining a regional temperature anomaly result based on the regional temperature and the historical regional temperature; A fire source warning coefficient is determined based on the smoke identification result, the regional gas anomaly result, and the regional temperature anomaly result.

[0013] According to the present invention, determining the equipment warning coefficient based on the thermal imaging data and the inspection audio data includes: Determining device temperature data based on the thermal imaging data; Determining a device temperature anomaly identification result based on the device temperature data and a set device temperature threshold; Acquiring historical test operation status data of the test equipment during a plurality of second historical test cycles, wherein the historical test operation status data includes: historical equipment operation efficiency, historical equipment vibration value, and historical equipment working audio data; Obtain the benchmark operating efficiency, benchmark vibration value and normal operating audio data of the test equipment; Determining a historical equipment normal operation coefficient based on the historical equipment operating efficiency, the historical equipment vibration value, the benchmark operating efficiency, and the benchmark vibration value; Processing the historical equipment working audio data according to the equipment operating status prediction model to obtain a training prediction equipment normal operating coefficient; Obtaining a training loss function of a device operation status prediction model based on the training prediction device normal operation coefficient, the historical device normal operation coefficient, the historical device operation audio data, and the normal operation audio data; Training the device operation status prediction model according to the training loss function of the device operation status prediction model to obtain a trained device operation status prediction model; Processing the inspection audio data according to the trained equipment operation status prediction model to determine the real-time equipment normal operation coefficient; Determining an equipment operation abnormality identification result based on the real-time equipment normal operation coefficient; An equipment warning coefficient is determined based on the equipment temperature anomaly identification result and the equipment operation anomaly identification result.

[0014] According to the present invention, a training loss function of a device operation status prediction model is obtained based on the training prediction device normal operation coefficient, the historical device normal operation coefficient, the historical device operation audio data, and the normal operation audio data, including: Determining a historical operating root mean square value and a historical operating peak factor based on the historical device operating audio data; Determining a normal working root mean square value and a normal working peak factor based on the normal working audio data; According to the formula

[0015] Determine the training loss function for the equipment health prediction model ,in, The normal operation coefficient of the training prediction equipment at the bth moment of the yth second historical test cycle is is the historical equipment normal operation coefficient of the test equipment at the bth moment in the yth second historical test cycle, is the historical operating RMS value of the test equipment at the bth moment of the yth second historical test cycle, is the RMS value of the normal operation of the test equipment, is the historical operating peak factor of the test equipment at the bth moment in the yth second historical test cycle, is the normal working peak factor of the test equipment, Y is the number of the second historical test cycle, y≤Y, B is the number of moments in the second historical test cycle, b≤B, y, Y, b and B are all positive integers.

[0016] According to a second aspect of the present invention, a patrol robot safety warning system based on multimodal data fusion is provided, comprising: Position information module, used to obtain the robot's position information; An inspection data module is used to obtain inspection data at multiple moments during the inspection cycle through a combination of multimodal sensors installed in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind speed data, inspection audio data, and thermal imaging data; a gas warning module, configured to determine a harmful gas warning coefficient based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; A fire source warning module, configured to determine a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data; An equipment warning module, configured to determine an equipment warning coefficient based on the thermal imaging data and the inspection audio data; The early warning information module is used to generate early warning information based on the location information, the harmful gas early warning coefficient, the fire source early warning coefficient and the equipment early warning coefficient.

[0017] Technical Effect: According to the present invention, multimodal data fusion can be achieved through inspection robots, combined with real-time environmental perception and abnormal pattern recognition, to effectively discover harmful gas hazards, abnormal fire source conditions and abnormal equipment operation conditions, determine the harmful gas warning coefficient, fire source warning coefficient and equipment warning coefficient, and achieve rapid accident warning and positioning, upload safety hazard point video information to management personnel, generate corresponding warning information, and improve the timeliness and accuracy of warnings. When determining the diffusion rate relationship function, the diffusion rate relationship function can be determined based on the test harmful gas concentration data, test environmental temperature data, test environmental humidity data, test wind speed data and test diffusion speed, accurately describing the influence of gas concentration, temperature, humidity and wind speed on the diffusion rate, and improving the accuracy and objectivity of the diffusion rate relationship function. When determining the training loss function, the training loss function of the equipment operation status prediction model can be obtained based on the training prediction equipment normal operation coefficient, the historical equipment normal operation coefficient, the historical equipment working audio data and the normal working audio data. During the calculation process, the influence of the above data on the error of the training prediction equipment normal operation coefficient can be determined based on the possible influence of the root mean square value and the peak factor on the equipment working condition, and the training loss function is set based on the influence and the relative error of the training prediction equipment normal operation coefficient, so that the training loss function of the equipment operation status prediction model is reduced during the training process, and the accuracy of the equipment operation status prediction model is improved in a more targeted manner.

[0018] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and not limiting of the present invention. Other features and aspects of the present invention will become more apparent from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. Those skilled in the art can derive other embodiments based on these drawings without inventive efforts. Figure 1 A schematic diagram exemplarily illustrates a flow chart of a safety early warning method for an inspection robot based on multimodal data fusion according to an embodiment of the present invention; Figure 2 A flowchart of determining a harmful gas warning coefficient according to an embodiment of the present invention is exemplarily shown; Figure 3 A flowchart for determining a fire source warning coefficient according to an embodiment of the present invention is exemplarily shown; Figure 4A flowchart for determining a device warning coefficient according to an embodiment of the present invention is exemplarily shown; Figure 5 A block diagram of a patrol robot safety warning system based on multimodal data fusion according to an embodiment of the present invention is exemplarily shown. DETAILED DESCRIPTION

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts shall fall within the scope of protection of the present invention.

[0021] The following specific embodiments are used to describe the technical solution of the present invention in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0022] Figure 1 The following is a flow chart of a safety warning method for an inspection robot based on multimodal data fusion according to an embodiment of the present invention, wherein the method includes: Step S1, obtaining the position information of the robot; Step S2, at multiple moments in the inspection cycle, obtaining inspection data through a multimodal sensor combination provided in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind speed data, inspection audio data, and thermal imaging data; Step S3, determining a harmful gas warning coefficient based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; Step S4, determining a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data; Step S5, determining the equipment warning coefficient based on the thermal imaging data and the inspection audio data; Step S6: Generate warning information based on the location information, the harmful gas warning coefficient, the fire source warning coefficient, and the equipment warning coefficient.

[0023] According to the inspection robot safety warning method based on multimodal data fusion according to the embodiment of the present invention, multimodal data fusion can be realized through the inspection robot, and real-time environmental perception and abnormal pattern recognition can be combined to effectively discover harmful gas hazard conditions, abnormal fire source conditions and abnormal equipment operation conditions, determine the harmful gas warning coefficient, fire source warning coefficient and equipment warning coefficient, and realize rapid accident warning and positioning, upload video information of safety hazard points to management personnel, generate corresponding warning information, and improve the timeliness and accuracy of warnings.

[0024] According to one embodiment of the present invention, in step S1 , the position information of the robot is obtained.

[0025] For example, lidar or visual information is used to construct an environmental map of an underground sewage treatment plant and determine the location information of the robot in the underground sewage treatment plant.

[0026] According to one embodiment of the present invention, in step S2, at multiple moments in the inspection cycle, inspection data is acquired by a multimodal sensor combination provided in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind data, inspection audio data and thermal imaging data.

[0027] For example, the multimodal sensor combination includes: multi-channel gas sensors (which can be used to detect oxygen, carbon monoxide, oxygen, hydrogen sulfide, methane and ammonia), infrared thermal imagers, thermal imaging dual-light cameras, acoustic sensors, temperature and humidity sensors and anemometers. Through the multimodal sensor combination, inspection data can be collected in real time during the inspection process.

[0028] According to one embodiment of the present invention, in step S3, a harmful gas warning coefficient is determined based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data.

[0029] Figure 2 A flow chart for determining a harmful gas warning coefficient according to an embodiment of the present invention is exemplarily shown.

[0030] According to one embodiment of the present invention, step S3 includes: Step S31, obtaining historical harmful gas concentration data; Step S32, determining a hazardous gas concentration risk factor based on the historical hazardous gas concentration data and the hazardous gas concentration data; Step S33, determining a harmful gas diffusion risk coefficient based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; Step S34: determining a harmful gas warning coefficient based on the harmful gas concentration risk coefficient and the harmful gas diffusion risk coefficient.

[0031] For example, obtain the harmful gas concentration data (such as methane concentration, hydrogen sulfide concentration and ammonia concentration, etc.) at each location in the previous historical inspection cycle of the current inspection cycle, that is, the historical harmful gas concentration data; based on the harmful gas concentration data of various harmful gases at the current location of the inspection robot and the historical harmful gas concentration data of the current location, evaluate the harmful gas concentration status at the current location, and determine the harmful gas concentration hazard coefficient of various harmful gases at the current location; based on the harmful gas concentration data, ambient temperature data, ambient humidity data and wind speed data, evaluate the diffusion status of harmful gases, and determine the harmful gas diffusion hazard coefficient of various harmful gases at the current location; determine the harmful gas warning coefficient by adding the harmful gas concentration hazard coefficient and the harmful gas diffusion hazard coefficient.

[0032] According to one embodiment of the present invention, step S32 includes: Step S321, determining a change range of the harmful gas concentration based on the historical harmful gas concentration data and the harmful gas concentration data; Step S322, determining a hazardous gas concentration change hazard identification result based on the hazardous gas concentration change amplitude and a preset hazardous gas concentration change amplitude threshold; Step S323, determining a hazardous gas concentration hazard identification result based on the hazardous gas concentration data and a preset hazardous gas concentration threshold; Step S324 : determining a hazardous gas concentration hazard coefficient according to the hazardous gas concentration change hazard identification result and the hazardous gas concentration hazard identification result.

[0033] For example, the change amplitude of the harmful gas concentration of the i-th harmful gas is determined based on the ratio of the difference between the harmful gas concentration data of the i-th harmful gas at the current position of the inspection robot and the historical harmful gas concentration data to the historical harmful gas concentration data of the i-th harmful gas; if the change amplitude of the harmful gas concentration of the i-th harmful gas is greater than or equal to the preset harmful gas concentration change amplitude threshold (which can be set to 5%), it means that the harmful gas concentration of the i-th harmful gas at this position has increased significantly compared with the last inspection, and there may be safety hazards, and an early warning is required. The hazardous gas concentration change identification result of the i-th harmful gas is 1, otherwise, the harmful gas concentration of the i-th harmful gas is 1. The change hazard identification result is 0; if the harmful gas concentration data of the i-th harmful gas is greater than or equal to the preset harmful gas concentration threshold (determined according to the type of harmful gas, for example, the preset harmful gas concentration threshold corresponding to hydrogen sulfide is 5ppm, and the preset harmful gas concentration threshold corresponding to methane is 1%LEL), it means that the i-th harmful gas will bring safety hazards and an early warning is required. The harmful gas concentration hazard identification result of the i-th harmful gas is 1; otherwise, the harmful gas concentration hazard identification result of the i-th harmful gas is 0; the harmful gas concentration hazard coefficient is determined by adding the harmful gas concentration change hazard identification result and the harmful gas concentration hazard identification result of the i-th harmful gas.

[0034] According to one embodiment of the present invention, step S33 includes: Step S331, obtaining test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind speed data, and test diffusion speed data in multiple historical test cycles; Step S332, determining a diffusion velocity relationship function based on the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind speed data, and the test diffusion velocity; Step S333, determining a predicted diffusion velocity based on the diffusion velocity relationship function, the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; Step S334: determining the hazardous gas diffusion risk coefficient based on the predicted diffusion speed.

[0035] For example, in the historical test cycle, in a similar experimental environment of an underground sewage treatment plant, the effects of environmental factors and harmful gas concentrations on the diffusion rate of harmful gases were tested, and test harmful gas concentration data, test environmental temperature data, test environmental humidity data, test wind data and test diffusion rate were obtained; harmful gas concentration data, environmental temperature data, environmental humidity data and wind data will affect the diffusion rate to a certain extent. For example, when the wind data is larger, the diffusion rate is faster. Based on the correlation of the above data, the diffusion rate relationship function between the test harmful gas concentration data, the test environmental temperature data, the test environmental humidity data, the test wind data and the test diffusion rate can be determined; in traditional technology, the diffusion rate of harmful gases can be detected in real time by setting up a gas sensor array, but due to the large scope of the underground sewage treatment plant, the cost of setting up a gas sensor array is too high, and due to the harsh environmental conditions of the underground sewage treatment plant, the gas sensor needs to be Maintenance is carried out, therefore, the predicted diffusion speed can be determined by fitting, so as to improve the efficiency of the detection of the diffusion speed of harmful gases, and the harmful gas concentration data, ambient temperature data, ambient humidity data and wind speed data in the current inspection cycle are substituted into the diffusion speed relationship function to determine the predicted diffusion speed of various harmful gases at the current position; according to the type of harmful gas and the predicted diffusion speed, the harmful gas diffusion risk coefficient is determined, such as, when the harmful gas is hydrogen sulfide, when the predicted diffusion speed is greater than or equal to 0.5ppm / min, the harmful gas diffusion risk coefficient corresponding to the harmful gas is 1, indicating that the harmful gas diffuses too fast, otherwise, the harmful gas diffusion risk coefficient corresponding to the harmful gas is 0, when the harmful gas is methane, when the predicted diffusion speed is greater than or equal to 200ppm / min, the harmful gas diffusion risk coefficient corresponding to the harmful gas is 1, indicating that the harmful gas diffuses too fast, otherwise, the harmful gas diffusion risk coefficient corresponding to the harmful gas is 0.

[0036] According to one embodiment of the present invention, step S332 includes: determining the undetermined coefficient equation of the diffusion velocity relationship function according to formula (1), (1) Among them, if is a conditional function, is the test diffusion rate of the i-th harmful gas in the k-th historical test cycle, is the preset diffusion speed threshold, is the test wind data in the kth historical test cycle, To preset the wind data threshold, is the test environment humidity data in the kth historical test cycle, To preset the ambient humidity threshold, is the test harmful gas concentration data of the i-th harmful gas in the k-th historical test cycle, To preset the harmful gas concentration data threshold, is the test environment temperature data in the kth historical test cycle, is the preset ambient temperature threshold, 、 、 、 、 、 、 、 、 、 and is the coefficient to be determined; Solving the undetermined coefficients according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind force data, and the test diffusion velocity to obtain solution values ​​of the undetermined coefficients; A diffusion velocity relationship function is obtained according to the solved value of the undetermined coefficient and the undetermined coefficient equation.

[0037] According to one embodiment of the present invention, is the test diffusion rate of the i-th harmful gas in the k-th historical test cycle and the preset diffusion rate threshold, Can be set to 1ppm / min, The ratio of the test wind data in the kth historical test period to the preset wind data threshold, Can be set to 1m / s, is the ratio of the test environment temperature data in the kth historical test cycle to the preset environment temperature threshold, Can be set to 25 degrees Celsius, is the ratio of the test environment humidity data in the kth historical test cycle to the preset environment humidity threshold, Can be set to 75%, is the ratio of the test harmful gas concentration data of the i-th harmful gas in the k-th historical test cycle to the preset harmful gas concentration data threshold, Set according to the type of harmful gas, for example, the value corresponding to hydrogen sulfide is 100ppm, 、 、 、 and All are dimensionless values.

[0038] According to one embodiment of the present invention, It means that the test wind speed data in the k-th historical test cycle is positively correlated with the test diffusion speed of the i-th harmful gas in the k-th historical test cycle. The greater the wind speed, the faster the air flow and the greater the diffusion speed of the harmful gas. It indicates that the test environment temperature data in the kth historical test cycle has a positive correlation with the test diffusion rate of the i-th harmful gas in the kth historical test cycle. The higher the temperature, the faster the intermolecular motion and the greater the diffusion rate of the harmful gas. It indicates that the test environment humidity data in the kth historical test cycle has a negative correlation with the test diffusion rate of the i-th harmful gas in the kth historical test cycle. The higher the humidity, the faster the gas dissolution rate, thereby limiting the diffusion of the gas and reducing the diffusion rate of the harmful gas.

[0039] According to one embodiment of the present invention, in formula (1), the conditional function The value of includes the following two cases, when it satisfies When the condition is , indicating that the concentration of harmful gases is too high, forming an air mass effect. The higher the gas concentration, the slower the diffusion rate. This means that under the condition of high concentration of harmful gases, the test harmful gas concentration data of the i-th harmful gas in the k-th historical test cycle has a negative correlation with the test diffusion rate. When it does not meet When the condition is , indicating that when the concentration of harmful gases is low, there is a positive correlation between the diffusion rate and concentration, and harmful gases will diffuse from locations with lower concentrations to locations with lower concentrations. Based on the above relationship, the equation for the unknown coefficients of the diffusion rate relationship function can be obtained.

[0040] According to one embodiment of the present invention, fitting can be performed based on multiple parameters involved in the above-mentioned undetermined coefficient equation, that is, fitting can be performed based on the test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind speed data and test diffusion speed in at least 11 historical test cycles to solve the above-mentioned multiple undetermined coefficients. There are 11 undetermined coefficients, namely, 、 、 、 、 、 、 、 、 、 and , the above 11 undetermined coefficients are solved according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind speed data and the test diffusion speed, and the solution values ​​of the above 11 undetermined coefficients are obtained, and the solution values ​​of the above 11 undetermined coefficients are substituted into the undetermined coefficient equation to obtain the diffusion speed relationship function.

[0041] In this way, the diffusion rate relationship function can be determined based on the test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind speed data and test diffusion speed, accurately describing the influence of gas concentration, temperature, humidity and wind speed on the diffusion speed, and improving the accuracy and objectivity of the diffusion rate relationship function.

[0042] According to one embodiment of the present invention, in step S4, a fire source warning coefficient is determined based on the real-time inspection video, the environmental gas data, and the thermal imaging data.

[0043] Figure 3 A flow chart for determining a fire source warning coefficient according to an embodiment of the present invention is exemplarily shown.

[0044] According to one embodiment of the present invention, step S4 includes: Step S41: In the real-time inspection video, the image detection model is used to identify whether there is a fire source and smoke at the inspection site, and a fire source identification result and a smoke identification result are determined; Step S42, when the fire source identification result is 1, determining the fire source warning coefficient to be 3; Step S43: When the fire source identification result is 0, a fire source warning coefficient is determined according to the smoke identification result, the ambient gas data, and the thermal imaging data.

[0045] For example, the image detection model is a deep learning neural network model such as a convolutional neural network model. The image detection model can be trained based on historical data so that the image detection model can identify whether there is flame or smoke in the image. According to the real-time inspection video, the inspection image at each moment of the inspection cycle is determined. The inspection image is identified according to the image detection model to identify whether there is a fire source and smoke at the inspection site, and the fire source identification result and the smoke identification result are determined. When there is a fire source, the fire source identification result is 1, otherwise, the fire source identification result is 0. When there is smoke, the smoke identification result is 1, otherwise, the smoke identification result is 0; when the fire source identification result is 1, it indicates that there is an open flame, and the fire source warning coefficient is determined to be 3; when the fire source identification result is 0, it indicates that there is no open flame. Further judgment is made based on the smoke identification result, environmental gas data and thermal imaging data to determine the fire source warning coefficient.

[0046] According to one embodiment of the present invention, step S43 includes: Step S431, determining the carbon monoxide concentration and the oxygen concentration according to the ambient gas data; Step S432, determining the regional temperature based on the thermal imaging data; Step S433, obtaining historical carbon monoxide concentration, historical oxygen concentration and historical regional temperature; Step S434, determining regional gas anomaly results based on the carbon monoxide concentration, the oxygen concentration, the historical carbon monoxide concentration, and the historical oxygen concentration; Step S435, determining a regional temperature anomaly result based on the regional temperature and the historical regional temperature; Step S436: determining a fire source warning coefficient based on the smoke recognition result, the regional gas anomaly result, and the regional temperature anomaly result.

[0047] For example, the carbon monoxide concentration and oxygen concentration are detected by a multi-channel gas sensor; the regional temperature is detected by an infrared thermal imager; the carbon monoxide concentration, oxygen concentration and regional temperature of each location in multiple historical inspection cycles are obtained, and the carbon monoxide concentration, oxygen concentration and regional temperature in multiple historical inspection cycles are averaged to determine the historical carbon monoxide concentration, historical oxygen concentration and historical regional temperature, indicating the normal carbon monoxide concentration, normal oxygen concentration and normal regional temperature of each location; if the carbon monoxide concentration is greater than the historical carbon monoxide concentration and the oxygen concentration is less than the historical oxygen concentration, it indicates that combustion may occur, and the regional gas abnormality result is determined to be 1, otherwise the regional gas abnormality result is 0; if If the regional temperature is much higher than the historical regional temperature (more than 40% of the historical regional temperature), for example, the regional temperature is 70 degrees Celsius and the historical regional temperature is 50 degrees Celsius, it indicates that combustion may occur, and the regional temperature anomaly result is 1. Otherwise, the regional temperature anomaly result is 0. The fire source warning coefficient is determined by summing the smoke identification results, regional gas anomaly results, and regional temperature anomaly results. For example, when the smoke identification results, regional gas anomaly results, and regional temperature anomaly results are all 1, it means that a hidden fire combustion phenomenon will definitely occur, and the fire source warning coefficient is 3. The larger the sum of the smoke identification results, regional gas anomaly results, and regional temperature anomaly results, the greater the possibility of a hidden fire combustion phenomenon and the larger the fire source warning coefficient.

[0048] According to one embodiment of the present invention, in step S5, the equipment warning coefficient is determined based on the thermal imaging data and the inspection audio data.

[0049] Figure 4 A flow chart for determining a device warning coefficient according to an embodiment of the present invention is exemplarily shown.

[0050] According to one embodiment of the present invention, step S5 includes: Step S51, determining device temperature data based on the thermal imaging data; Step S52, determining a device temperature anomaly identification result based on the device temperature data and a set device temperature threshold; Step S53, acquiring historical test operation status data of the test equipment in a plurality of second historical test cycles, wherein the historical test operation status data of the test equipment includes: historical equipment operation efficiency, historical equipment vibration value, and historical equipment working audio data; Step S54, obtaining the benchmark operating efficiency, benchmark vibration value and normal operating audio data of the test equipment; Step S55, determining a historical equipment normal operation coefficient based on the historical equipment operation efficiency, the historical equipment vibration value, the benchmark operation efficiency, and the benchmark vibration value; Step S56, processing the historical equipment working audio data according to the equipment operating status prediction model to obtain a training prediction equipment normal operating coefficient; Step S57, obtaining a training loss function of a device operation status prediction model based on the training prediction device normal operation coefficient, the historical device normal operation coefficient, the historical device operation audio data, and the normal operation audio data; Step S58, training the equipment operation status prediction model according to the training loss function of the equipment operation status prediction model to obtain a trained equipment operation status prediction model; Step S59, processing the inspection audio data according to the trained equipment operation status prediction model to determine the real-time equipment normal operation coefficient; Step S510, determining a device operation abnormality identification result based on the real-time device normal operation coefficient; Step S511: determining a device warning coefficient according to the device temperature anomaly identification result and the device operation anomaly identification result.

[0051] For example, the temperature data of the working equipment in the underground sewage treatment plant is detected by an infrared thermal imager. If the equipment temperature data is greater than or equal to the set equipment temperature threshold (which can be set to 70 degrees Celsius), it indicates that there is an electrical fire safety hazard, and the equipment temperature anomaly identification result is 1. Otherwise, the equipment temperature anomaly identification result is 0. In the second historical test cycle, equipment with the same model as the equipment in the underground sewage treatment plant is selected as the test equipment, and the historical equipment operating efficiency (the ratio of effective output power to total input power), historical equipment vibration value (for example, the equipment vibration value is 2mm / s) and historical equipment working audio data of the test equipment in the second historical test cycle are collected; the benchmark operating efficiency and benchmark vibration value of the equipment when it is just out of the factory are collected, and the normal working audio data of the test equipment during normal operation is collected; the operating efficiency difference is determined by subtracting the benchmark operating efficiency from the historical equipment operating efficiency, and the vibration value difference is determined by subtracting the historical equipment vibration value from the benchmark vibration value. The normal operating coefficient of the historical equipment is determined based on the ratio of the operating efficiency difference to the benchmark operating efficiency and the sum of the ratios of the vibration value difference to the benchmark vibration value. The larger the historical equipment normal operation coefficient, the higher the operating efficiency of the test equipment in the second historical test cycle, the lower the vibration value, and the better the operating condition; the historical equipment working audio data is processed according to the equipment operating condition prediction model to obtain the training predicted equipment normal operation coefficient; the training loss function of the equipment operating condition prediction model is obtained according to the training predicted equipment normal operation coefficient, the historical equipment normal operation coefficient, the historical equipment working audio data and the normal working audio data; the equipment operating condition prediction model is trained according to the training loss function of the equipment operating condition prediction model to improve the prediction accuracy of the equipment operating condition prediction model and obtain a trained equipment operating condition prediction model; the inspection audio data is processed according to the trained equipment operating condition prediction model to determine the real-time equipment normal operation coefficient; if the real-time equipment normal operation coefficient is less than 0, it means that there is an abnormality in the equipment's operating efficiency or vibration, and the equipment operation abnormality identification result is determined to be 1, otherwise, the equipment operation abnormality identification result is 0; the equipment warning coefficient is determined by summing the equipment temperature abnormality identification result and the equipment operation abnormality identification result.

[0052] According to one embodiment of the present invention, step S67 includes: Determining a historical operating root mean square value and a historical operating peak factor based on the historical device operating audio data; Determining a normal working root mean square value and a normal working peak factor based on the normal working audio data; Determine the training loss function of the equipment operation status prediction model according to formula (2) , (2) in, The normal operation coefficient of the training prediction equipment at the bth moment of the yth second historical test cycle is is the historical equipment normal operation coefficient of the test equipment at the bth moment in the yth second historical test cycle, is the historical operating RMS value of the test equipment at the bth moment of the yth second historical test cycle, is the RMS value of the normal operation of the test equipment, is the historical operating peak factor of the test equipment at the bth moment in the yth second historical test cycle, is the normal working peak factor of the test equipment, Y is the number of the second historical test cycle, y≤Y, B is the number of moments in the second historical test cycle, b≤B, y, Y, b and B are all positive integers.

[0053] According to one embodiment of the present invention, time-frequency domain feature extraction is performed on historical equipment working audio data and normal working audio data to determine the historical working RMS value and historical working peak factor, as well as the normal working RMS value and normal working peak factor, respectively.

[0054] According to one embodiment of the present invention, It is the ratio of the historical working RMS value of the test equipment at the bth moment of the yth second historical test cycle to the normal working RMS value. The larger the ratio, the larger the historical working RMS value. The ratio of the historical working peak factor of the test equipment at the bth moment of the yth second historical test cycle to the normal working peak factor is the same. The larger the ratio, the larger the historical working peak factor. It indicates that the root mean square value and peak factor are negatively correlated with the normal operating coefficient of the training prediction equipment. For example, the larger the root mean square value, the greater the overall sound energy intensity, which indicates severe friction or looseness, resulting in a decrease in equipment working efficiency and abnormal vibration values. The smaller the normal operating coefficient of the training prediction equipment, the larger the peak factor, which indicates a greater possibility of impact failure (such as bearing ball damage, blade breakage and collision), resulting in a decrease in equipment working efficiency and abnormal vibration values, and the smaller the normal operating coefficient of the training prediction equipment.

[0055] According to one embodiment of the present invention, The relative error between the normal operation coefficient of the training prediction equipment and the normal operation coefficient of the historical equipment is calculated by using The relative error between the training predicted device normal operation coefficient and the historical device normal operation coefficient at the bth moment of the yth second historical test cycle of the test device is weighted averaged to obtain a training loss function. During the training process, the above training loss function is reduced, thereby reducing the relative error between the training predicted device normal operation coefficient and the historical device normal operation coefficient, thereby improving the accuracy of the device operation status prediction model for device operation status prediction, thereby improving the accuracy of the device operation status prediction model.

[0056] In this way, the training loss function of the equipment operation status prediction model can be obtained based on the training prediction equipment normal operation coefficient, the historical equipment normal operation coefficient, the historical equipment working audio data and the normal working audio data. During the calculation process, the influence of the above data on the error of the training prediction equipment normal operation coefficient can be determined based on the possible influence of the root mean square value and the peak factor on the equipment working condition, and the training loss function is set based on the influence and the relative error of the training prediction equipment normal operation coefficient, so that the training loss function of the equipment operation status prediction model is reduced during the training process, and the accuracy of the equipment operation status prediction model is improved in a more targeted manner.

[0057] According to one embodiment of the present invention, in step S6, warning information is generated based on the location information, the harmful gas warning coefficient, the fire source warning coefficient, and the equipment warning coefficient.

[0058] For example, when the harmful gas warning coefficient is equal to 3, it means that the harmful gas hazard at this location is serious, and the highest warning information is generated. When the harmful gas warning coefficient is equal to 2 and 1, it means that there are certain harmful gas safety hazards at this location. When the harmful gas warning coefficient is equal to 0, it means that there are no harmful gas safety hazards, and the corresponding warning information is generated according to the position information of the inspection robot, such as the highest safety warning of the i-th harmful gas at position A; when the fire source warning coefficient is equal to 3, it means that there is a hidden fire or open flame. When the fire source warning coefficient is less than 3 and greater than 0, it means that there is a possibility of hidden fire combustion. When the fire source warning coefficient is equal to 0, it means that there is no fire situation, and the corresponding warning information is generated according to the position information of the inspection robot, such as the existence of an open flame at position A; when the equipment warning coefficient is greater than 0, it means that the temperature of the equipment is abnormal or there is an abnormal noise when the equipment is working, and the corresponding warning information is generated according to the position information of the inspection robot, such as the temperature of the working equipment at position A is abnormal.

[0059] According to the patrol robot safety warning method based on multimodal data fusion according to the embodiment of the present invention, multimodal data fusion can be achieved through the patrol robot, and real-time environmental perception and abnormal pattern recognition can be combined to effectively detect harmful gas hazards, abnormal fire source conditions, and abnormal equipment operation conditions, determine the harmful gas warning coefficient, fire source warning coefficient, and equipment warning coefficient, and achieve rapid accident warning and positioning. Video information of safety hazard points can be uploaded to management personnel to generate corresponding warning information, thereby improving the timeliness and accuracy of warnings. When determining the diffusion rate relationship function, the diffusion rate relationship function can be determined based on the test harmful gas concentration data, test environmental temperature data, test environmental humidity data, test wind speed data, and test diffusion speed, accurately describing the influence of gas concentration, temperature, humidity, and wind speed on the diffusion rate, thereby improving the accuracy and objectivity of the diffusion rate relationship function. When determining the training loss function, the training loss function of the equipment operation status prediction model can be obtained based on the training prediction equipment normal operation coefficient, the historical equipment normal operation coefficient, the historical equipment working audio data and the normal working audio data. During the calculation process, the influence of the above data on the error of the training prediction equipment normal operation coefficient can be determined based on the possible influence of the root mean square value and the peak factor on the equipment working condition, and the training loss function is set based on the influence and the relative error of the training prediction equipment normal operation coefficient, so that the training loss function of the equipment operation status prediction model is reduced during the training process, and the accuracy of the equipment operation status prediction model is improved in a more targeted manner.

[0060] Figure 5 A block diagram of a patrol robot safety warning system based on multimodal data fusion according to an embodiment of the present invention is exemplarily shown. The system includes: Position information module, used to obtain the robot's position information; An inspection data module is used to obtain inspection data at multiple moments during the inspection cycle through a combination of multimodal sensors installed in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind speed data, inspection audio data, and thermal imaging data; a gas warning module, configured to determine a harmful gas warning coefficient based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; A fire source warning module, configured to determine a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data; An equipment warning module, configured to determine an equipment warning coefficient based on the thermal imaging data and the inspection audio data; The early warning information module is used to generate early warning information based on the location information, the harmful gas early warning coefficient, the fire source early warning coefficient and the equipment early warning coefficient.

[0061] The present invention may be a method, an apparatus, a system and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for executing various aspects of the present invention.

[0062] Those skilled in the art will appreciate that the embodiments of the present invention described above and shown in the accompanying drawings are intended to be illustrative only and are not intended to limit the present invention. The objectives of the present invention have been fully and effectively achieved. The functional and structural principles of the present invention have been demonstrated and illustrated in the embodiments. Any variations or modifications may be made to the embodiments of the present invention without departing from the principles described.

Claims

1. A safety warning method for inspection robots based on multimodal data fusion, characterized in that: include: Get the robot's location information; At multiple moments during the inspection cycle, inspection data is acquired through a combination of multimodal sensors installed in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind speed data, inspection audio data, and thermal imaging data; determining a harmful gas warning coefficient according to the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; Determining a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data; Determining an equipment warning coefficient based on the thermal imaging data and the inspection audio data; Warning information is generated based on the location information, the harmful gas warning coefficient, the fire source warning coefficient and the equipment warning coefficient.

2. The inspection robot safety early warning method based on multimodal data fusion according to claim 1 is characterized in that: Determining a harmful gas warning coefficient according to the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data includes: Obtain historical harmful gas concentration data; determining a hazardous gas concentration risk factor based on the historical hazardous gas concentration data and the hazardous gas concentration data; determining a hazardous gas diffusion risk coefficient based on the hazardous gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; A harmful gas warning coefficient is determined based on the harmful gas concentration hazard coefficient and the harmful gas diffusion hazard coefficient.

3. The inspection robot safety early warning method based on multimodal data fusion according to claim 2 is characterized in that: Determining a hazardous gas concentration risk factor based on the historical hazardous gas concentration data and the hazardous gas concentration data includes: determining a change range of the harmful gas concentration based on the historical harmful gas concentration data and the harmful gas concentration data; Determining a hazardous gas concentration change hazard identification result based on the hazardous gas concentration change amplitude and a preset hazardous gas concentration change amplitude threshold; Determining a hazardous gas concentration hazard identification result based on the hazardous gas concentration data and a preset hazardous gas concentration threshold; A harmful gas concentration hazard coefficient is determined according to the harmful gas concentration change hazard identification result and the harmful gas concentration hazard identification result.

4. The inspection robot safety early warning method based on multimodal data fusion according to claim 2 is characterized in that: Determining a hazardous gas diffusion risk coefficient based on the hazardous gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data includes: Obtain test harmful gas concentration data, test environment temperature data, test environment humidity data, test wind speed data and test diffusion speed data in multiple historical test cycles; Determining a diffusion velocity relationship function according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind speed data, and the test diffusion velocity; Determining a predicted diffusion speed based on the diffusion speed relationship function, the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; The hazardous gas diffusion risk coefficient is determined based on the predicted diffusion speed.

5. The inspection robot safety early warning method based on multimodal data fusion according to claim 4 is characterized in that: Determining a diffusion velocity relationship function according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind speed data, and the test diffusion velocity includes: According to the formula Determine the unknown coefficient equation of the diffusion velocity relationship function, where if is the conditional function, is the test diffusion rate of the i-th harmful gas in the k-th historical test cycle, is the preset diffusion speed threshold, is the test wind data in the kth historical test cycle, To preset the wind data threshold, is the test environment humidity data in the kth historical test cycle, To preset the ambient humidity threshold, is the test harmful gas concentration data of the i-th harmful gas in the k-th historical test cycle, To preset the harmful gas concentration data threshold, is the test environment temperature data in the kth historical test cycle, is the preset ambient temperature threshold, 、 、 、 、 、 、 、 、 、 and is the coefficient to be determined; Solving the undetermined coefficients according to the test harmful gas concentration data, the test environment temperature data, the test environment humidity data, the test wind force data, and the test diffusion velocity to obtain solution values ​​of the undetermined coefficients; A diffusion velocity relationship function is obtained according to the solved value of the undetermined coefficient and the undetermined coefficient equation.

6. The inspection robot safety early warning method based on multimodal data fusion according to claim 1 is characterized in that: Determining a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data includes: In the real-time inspection video, an image detection model is used to identify whether there is a fire source and smoke at the inspection site, and a fire source identification result and a smoke identification result are determined; When the fire source identification result is 1, the fire source warning coefficient is determined to be 3; When the fire source identification result is 0, a fire source warning coefficient is determined according to the smoke identification result, the ambient gas data and the thermal imaging data.

7. The inspection robot safety early warning method based on multimodal data fusion according to claim 6 is characterized in that: When the fire source identification result is 0, determining a fire source warning coefficient according to the smoke identification result, the ambient gas data, and the thermal imaging data includes: determining carbon monoxide concentration and oxygen concentration according to the ambient gas data; determining a regional temperature based on the thermal imaging data; Get historical carbon monoxide concentration, historical oxygen concentration and historical regional temperature; Determining regional gas anomaly results based on the carbon monoxide concentration, the oxygen concentration, the historical carbon monoxide concentration, and the historical oxygen concentration; Determining a regional temperature anomaly result based on the regional temperature and the historical regional temperature; A fire source warning coefficient is determined based on the smoke identification result, the regional gas anomaly result, and the regional temperature anomaly result.

8. The inspection robot safety early warning method based on multimodal data fusion according to claim 1 is characterized in that: Determining a device warning coefficient based on the thermal imaging data and the inspection audio data includes: Determining device temperature data based on the thermal imaging data; Determining a device temperature anomaly identification result based on the device temperature data and a set device temperature threshold; Acquiring historical test operation status data of the test equipment during a plurality of second historical test cycles, wherein the historical test operation status data includes: historical equipment operation efficiency, historical equipment vibration value, and historical equipment working audio data; Obtain the benchmark operating efficiency, benchmark vibration value and normal operating audio data of the test equipment; Determining a historical equipment normal operation coefficient based on the historical equipment operating efficiency, the historical equipment vibration value, the benchmark operating efficiency, and the benchmark vibration value; Processing the historical equipment working audio data according to the equipment operating status prediction model to obtain a training prediction equipment normal operating coefficient; Obtaining a training loss function of a device operation status prediction model based on the training prediction device normal operation coefficient, the historical device normal operation coefficient, the historical device operation audio data, and the normal operation audio data; Training the device operation status prediction model according to the training loss function of the device operation status prediction model to obtain a trained device operation status prediction model; Processing the inspection audio data according to the trained equipment operation status prediction model to determine the real-time equipment normal operation coefficient; Determining an equipment operation abnormality identification result based on the real-time equipment normal operation coefficient; An equipment warning coefficient is determined based on the equipment temperature anomaly identification result and the equipment operation anomaly identification result.

9. The inspection robot safety early warning method based on multimodal data fusion according to claim 8 is characterized in that: Obtaining a training loss function of a device operation status prediction model based on the training prediction device normal operation coefficient, the historical device normal operation coefficient, the historical device operation audio data, and the normal operation audio data, including: Determining a historical operating root mean square value and a historical operating peak factor based on the historical device operating audio data; Determining a normal working root mean square value and a normal working peak factor based on the normal working audio data; According to the formula Determine the training loss function for the equipment health prediction model ,in, The normal operation coefficient of the training prediction equipment at the bth moment of the yth second historical test cycle is is the historical equipment normal operation coefficient of the test equipment at the bth moment in the yth second historical test cycle, is the historical operating RMS value of the test equipment at the bth moment of the yth second historical test cycle, is the RMS value of the normal operation of the test equipment, is the historical operating peak factor of the test equipment at the bth moment in the yth second historical test cycle, is the normal working peak factor of the test equipment, Y is the number of the second historical test cycle, y≤Y, B is the number of moments in the second historical test cycle, b≤B, y, Y, b and B are all positive integers.

10. A patrol robot safety warning system based on multimodal data fusion, characterized in that: include: Position information module, used to obtain the robot's position information; An inspection data module is used to obtain inspection data at multiple moments during the inspection cycle through a combination of multimodal sensors installed in the inspection robot, wherein the inspection data includes: harmful gas concentration data, real-time inspection video, ambient temperature data, ambient humidity data, ambient gas data, wind speed data, inspection audio data, and thermal imaging data; a gas warning module, configured to determine a harmful gas warning coefficient based on the harmful gas concentration data, the ambient temperature data, the ambient humidity data, and the wind speed data; A fire source warning module, configured to determine a fire source warning coefficient based on the real-time inspection video, the environmental gas data, and the thermal imaging data; An equipment warning module, configured to determine an equipment warning coefficient based on the thermal imaging data and the inspection audio data; The early warning information module is used to generate early warning information based on the location information, the harmful gas early warning coefficient, the fire source early warning coefficient and the equipment early warning coefficient.

Images