A robot fire intelligent identification and early warning method based on visual perception

Abstract

The application discloses a robot fire intelligent identification and early warning method and system based on visual perception, equipment and medium. The method comprises the following steps: acquiring a denoised panoramic image by a patrol robot; dividing the panoramic image into grids and calculating the burning feature confidence based on the pixel brightness, local contrast and gray standard deviation to screen out suspected subareas; continuously sampling the suspected subareas from multiple perspectives and multiple time sequences, extracting the burning feature saliency of the suspected target, calculating the transverse feature similarity and longitudinal scale trend value, and fusing to obtain the second burning feature confidence to determine the actual target position of the flame; based on the real-time burning scale of the target position, using a time series prediction model to obtain a future burning scale prediction value and establish a growth curve, determining the priority of fire extinguishing according to the geometric characteristics of the growth curve, and controlling the patrol robot to perform fire fighting warning and extinguishing actions. The application realizes accurate positioning and trend prediction of the initial flame.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and robot vision technology, specifically to a method, system, electronic device and storage medium for intelligent identification and early warning of fires by robots based on visual perception. Background Technology

[0002] Existing technologies use static or quasi-static image fusion analysis of visible light and infrared thermal imaging, focusing on confirmation through temperature changes after identifying suspected areas. However, dynamic interference from background and light in the early stages of flame formation makes it difficult for existing technologies based on static or single-frame image analysis to accurately locate the flame area, resulting in delayed warnings. Another existing technology relies on multi-source data fusion from various fixed sensors (such as environmental and electrical sensors) deployed within buildings. Fixed multi-sensor IoT systems have deployment limitations and insufficient adaptability to dynamic visual interference when dealing with non-fixed monitoring points and complex dynamic visual scenarios (such as flames on a robot's moving inspection path), potentially leading to delayed early warnings of moving or newly emerging fire sources. Summary of the Invention

[0003] This application discloses a robot-based intelligent fire identification and early warning method, system, electronic device, and storage medium based on visual perception. It aims to solve the technical problem that static image analysis technology is unable to locate the flame burning position in a timely and accurate manner due to background interference, which leads to insensitive fire handling and early warning, and thus causes early warning delay.

[0004] To achieve the above objectives, the first aspect of this application provides a method for intelligent identification and early warning of fires in robots based on visual perception, comprising the following steps: The inspection robot's camera module captures a panoramic view of the monitored area to obtain a noise-reduced panoramic image. The panoramic image is divided into multiple block images by grid. Based on the brightness level, local contrast and grayscale standard deviation of the pixels in the block images, the combustion feature confidence of each block image is calculated. Blocks with combustion feature confidence greater than a preset threshold are identified as suspected partitions. The inspection robot is scheduled to perform continuous image sampling of the suspected area from multiple perspectives and at different times, thereby acquiring multiple sampled images of the suspected target from different perspectives and at different times. Extract the saliency of combustion features of suspected targets in the sampled images, calculate the horizontal feature similarity of different viewpoints at the same time sequence, and the vertical scale trend value of different time sequences at the same viewpoint; fuse the horizontal feature similarity and the vertical scale trend value to calculate the second confidence of combustion features of suspected targets, and determine the suspected targets whose second confidence of combustion features meets the preset judgment conditions as the target locations where flames are actually generated. The real-time combustion scale of the target location at multiple time series is used as a feature factor and input into a pre-constructed time series prediction model to obtain the predicted values ​​of the combustion scale at multiple future times. A growth curve is established based on the real-time combustion scale and the predicted values ​​of the combustion scale. The fire-fighting priority of the corresponding target location is determined according to the geometric characteristics of the growth curve. Based on the fire-fighting priority, the inspection robot is controlled to perform the corresponding fire warning and fire-fighting actions.

[0005] Optionally, calculating the confidence level of the combustion features of each block image based on the brightness level value, local contrast, and grayscale standard deviation of pixels in the block image includes: By sliding a window within the block image, the difference between the maximum and minimum brightness values ​​within each window is calculated and accumulated. This difference is then combined with the average brightness value of the block image to calculate the local contrast convolution. A combustion feature confidence calculation function is constructed by combining the local contrast convolution with the gray standard deviation of all pixels in the block image; The confidence level of the combustion feature is positively correlated with the local contrast convolution and negatively correlated with the grayscale standard deviation. The smaller the grayscale standard deviation and the larger the local contrast convolution, the higher the probability that the block has flame features.

[0006] Optionally, extracting the saliency of combustion features of suspected targets in the sampled image includes: Obtain the grayscale histogram of the region corresponding to the suspected target in the sampled image; Identify the two largest peaks in the grayscale histogram, extract the peak areas corresponding to the two largest peaks, and the grayscale values ​​corresponding to the two largest peak points; The saliency of the combustion feature is determined by multiplying the absolute value of the difference in peak area between the two largest peaks with the absolute value of the difference in gray value; wherein, the saliency of the combustion feature is used to characterize the bimodal gray-scale physical characteristics of the coexistence of flame and smoke.

[0007] Optionally, the step of fusing the horizontal feature similarity and the vertical scale trend value to calculate the second confidence level of the combustion features of the suspected target includes: The mean of the sum of squared differences in the saliency of combustion features from any two different perspectives within the same time series is used as the lateral feature similarity. Based on sampled images from different time sequences under the same viewpoint, the product of the number of pixels in the suspected target area and the brightness level is calculated as the real-time combustion scale, and the difference between the real-time combustion scales in adjacent time sequences is accumulated as the longitudinal scale trend value. Calculate the longitudinal trend value of the horizontal feature similarity under different time series, and the horizontal similarity of the longitudinal scale trend value under different perspectives; The average horizontal feature similarity, the average vertical scale trend value, the vertical trend value of the horizontal feature similarity, and the horizontal similarity of the vertical scale trend value are weighted and fused to obtain the second confidence level of the combustion feature.

[0008] Optionally, the step of establishing a growth curve based on the real-time combustion scale and the predicted combustion scale, and determining the firefighting priority for the corresponding target location based on the geometric characteristics of the growth curve, includes: Based on the predicted combustion scale value output by the time-series prediction model and the historical calculated value, a combustion scale growth curve at the target location is fitted and generated. Extract the slope of the growth curve and the area of ​​the integral between the growth curve and the horizontal axis of time; The slope and the integral area are weighted and summed, and the sum is used as the processing priority for the target location; wherein, the slope represents the instantaneous spread rate of the fire, and the integral area represents the overall burning scale of the fire. The movement control commands and fire resource scheduling strategies for the inspection robot are generated in descending order of processing priority.

[0009] Optionally, the time-series prediction model is an autoregressive integral moving average model; before inputting the real-time combustion scale at the target location in multiple time series as a feature factor into the pre-constructed time-series prediction model to obtain the predicted combustion scale values ​​for multiple future times, the method further includes: If only one target location is determined, the nearest inspection robot is directly controlled to move to that target location and the fire extinguishing device is activated. If multiple target locations are identified, the firefighting priority calculation process described above will be triggered.

[0010] To achieve the above objectives, a second aspect of this application also provides a vision-based intelligent fire identification and early warning system for robots, applied to the vision-based intelligent fire identification and early warning method for robots provided above. The system includes: The image acquisition and preprocessing module is used to acquire panoramic images through the camera module of the inspection robot and perform noise reduction processing; The global coarse screening module is used to divide the panoramic image into multiple blocks, calculate the confidence level of combustion features based on the brightness level, local contrast and grayscale standard deviation of pixels, and screen out suspected partitions. The multi-view temporal sampling and fine recognition module is used to control the robot to perform multi-view and multi-temporal sampling on suspected areas. Based on the bimodal feature of the gray-level histogram of suspected targets in the sampled images, the saliency of combustion features is calculated. The second confidence of combustion features is calculated by combining the horizontal multi-view feature similarity and the vertical temporal scale trend value to determine the target location where the actual flame is generated. The trend prediction and scheduling control module is used to generate predicted values ​​for future times based on the real-time combustion scale of the target location using a time-series prediction model to construct a growth curve. Based on the slope and area of ​​the growth curve, it calculates the fire-fighting priority of multiple target locations and controls the inspection robot to perform early warning and fire-fighting actions according to the priority.

[0011] To achieve the above objectives, a third aspect of this application also provides an electronic device, comprising: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the vision-based robotic fire intelligent identification and early warning method provided above.

[0012] To achieve the above objectives, a fourth aspect of this application also provides a computer-readable storage medium storing computer instructions for causing a computer to execute the vision-based robotic fire intelligent identification and early warning method provided above.

[0013] To achieve the above objectives, the fifth aspect of this application also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the previously provided vision-based robotic fire intelligent identification and early warning method.

[0014] This application discloses a visual perception-based robotic intelligent fire identification and early warning method, comprising: taking panoramic photos of the monitored area using the camera module of an inspection robot to obtain a denoised panoramic image; dividing the panoramic image into multiple block images by gridding; calculating the combustion feature confidence score of each block image based on the brightness level, local contrast, and grayscale standard deviation of pixels in the block images; identifying blocks with combustion feature confidence scores greater than a preset threshold as suspected zones; scheduling the inspection robot to perform multi-view and multi-time-series continuous image sampling of the suspected zones to obtain multiple sampled images of the suspected targets at different views and time sequences; extracting the saliency of the combustion features of the suspected targets in the sampled images; and calculating the combustion feature saliency of different views at the same time sequence. The horizontal feature similarity and the vertical scale trend value at different time series under the same viewpoint are used to calculate the second confidence score of the combustion feature of the suspected target. The suspected target whose second confidence score of the combustion feature meets the preset judgment condition is determined as the target location of the actual flame. Based on the real-time combustion scale of the target location at multiple time series as a feature factor, it is input into a pre-constructed time series prediction model to obtain the predicted value of the combustion scale at multiple future times. A growth curve is established based on the real-time combustion scale and the predicted value of the combustion scale. The fire-fighting priority of the corresponding target location is determined according to the geometric features of the growth curve. Based on the fire-fighting priority, the inspection robot is controlled to perform the corresponding fire warning and fire-fighting actions. This application has the following technical effects: By performing a dual recognition mechanism of coarse screening and further screening on the acquired images, the characteristics of flame combustion are peeled off layer by layer from the global to the local, eliminating the interference of related background and illumination, and improving the accuracy of flame target recognition in the scene. Based on the continuous image sequence acquired by the inspection robot camera, the unique characteristics of the target area in the time and space dimensions are analyzed to calculate the characterization significance and dynamic change of flame combustion. Its function is to overcome the limitations of static or single-frame image analysis in the early stages of a fire, which is easily affected by background and lighting dynamics, thus achieving more accurate feature extraction and localization of fire targets. Through a historical trend prediction and dynamic early warning timing determination mechanism, it calculates the historical sequence of feature factors in real time, performs trend extrapolation or model prediction, and dynamically adjusts the threshold or timing of early warning triggering based on the prediction results. Its function is to achieve forward-looking judgment, optimize early warning processing logic, reduce losses from untimely handling of multi-target fires, and improve the system's response speed and early warning reliability for early fires. Attached Figure Description

[0015] Figure 1 A flowchart illustrating a visual perception-based intelligent fire identification and early warning method for robots, provided in an embodiment of this application; Figure 2This is a schematic diagram illustrating how to obtain multiple image samples corresponding to any suspected target, as provided in an embodiment of this application. Figure 3 This application provides a grayscale histogram corresponding to a flame combustion target image. Figure 4 This is a schematic diagram showing the integration of predicted and calculated values ​​of real-time combustion scale to establish a corresponding growth curve, as provided in the embodiments of this application.

[0016] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0017] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0018] Explanation of key terms: In the embodiments of this application, the following key terms are used: Panoramic images refer to image data collected by panoramic cameras on site and high-definition cameras on robots, which include image data of all areas monitored on site, and are usually processed to remove noise.

[0019] Block image: refers to multiple rectangular regions of equal length and width obtained after dividing a panoramic image into grids.

[0020] Suspected partitions: These are blocks in an image that, based on calculations of pixel brightness levels, local contrast, and grayscale standard deviation, are determined to have a combustion feature confidence level greater than a preset threshold, indicating that flame combustion behavior may exist.

[0021] Suspected target: refers to an object with flame-like appearance characteristics detected by visual target recognition algorithms (such as YOLO) within the suspected zone.

[0022] Combustion feature saliency: refers to the physical feature value calculated based on the difference in peak area and gray value of the two largest peaks in the gray-scale histogram of the region corresponding to the suspected target. It is used to characterize the bimodal gray-scale physical feature of the coexistence of flame and smoke.

[0023] Lateral feature similarity: refers to the similarity index of the salience of combustion features of suspected targets under different perspectives at the same time sequence.

[0024] Vertical scale trend value: refers to the growth trend indicator of the real-time burning scale of a suspected target at different time series from the same perspective.

[0025] Combustion feature second confidence level: refers to the comprehensive evaluation value obtained by fusing the average horizontal feature similarity, the average vertical scale trend value and their cross trend and similarity, which is used to accurately determine whether the suspected target is the actual target location that produces flames.

[0026] Time-series prediction model: refers to the autoregressive integral moving average model (ARIMA), which is used to predict the combustion scale at multiple future moments based on historical real-time combustion scale characteristic factors.

[0027] Growth curve: refers to the geometric curve generated by fitting the predicted combustion scale value output by the time-series prediction model with historical calculated values, reflecting the change of combustion scale at the target location over time.

[0028] Current fire early warning systems often employ static or quasi-static image fusion analysis using visible light and infrared thermal imaging, focusing on confirmation through temperature changes after identifying suspected areas. However, in the early stages of a fire, dynamic interference from the background and light makes it difficult for existing technologies based on static or single-frame image analysis to accurately locate the flame area, resulting in warning delays. Furthermore, existing technologies also rely on multi-source data fusion solutions using multiple fixed sensors deployed within buildings. Such fixed multi-sensor IoT systems have deployment limitations and insufficient adaptability to dynamic visual interference when dealing with non-fixed monitoring points and complex dynamic visual scenarios, potentially leading to delays in early warnings of moving or newly emerging fire sources. Existing technologies using static or quasi-static image fusion analysis using visible light and infrared thermal imaging focus on confirmation through temperature changes after identifying suspected areas; however, dynamic interference from the background and light in the early stages of a fire makes it difficult for existing technologies based on static or single-frame image analysis to accurately locate the flame area, resulting in warning delays. This application's technical solution focuses on analyzing time-series images captured by a robot's camera. By calculating characteristic factors of flame combustion trends and predicting historical trends, it aims to improve the accuracy of early warnings. Existing technology two relies on multi-source data fusion using various fixed sensors (such as environmental and electrical sensors) deployed within buildings. However, fixed multi-sensor IoT systems have limitations in deployment and adaptability to dynamic visual interference when dealing with non-fixed monitoring points and complex dynamic visual scenarios (such as flames on a robot's mobile inspection path), potentially leading to delays in early warnings of moving or newly emerging fire sources. This application's technical solution, based on visual perception from a mobile robot platform, extracts flame features and predicts trends through multi-view, multi-time-series image acquisition. First, the camera module of an inspectable robot captures a panoramic view of the monitored area. Based on the pixel features of different zones in the panoramic image, it identifies potential fire zones. Then, based on these potential zones, the inspection robot captures multi-view images of the suspected targets at different time sequences. Finally, it comprehensively analyzes the dynamic changes and flame feature salience of these images to filter and locate the actual fire source. Then, based on multi-time-series images of the target location, feature change factors are determined, and combustion trend is predicted based on these feature change factors. Finally, a reasonable early warning and fire-fighting mechanism is established based on the predicted combustion trend. In summary, this solution aims to optimize the problem of insensitive fire handling and early warning caused by the difficulty in timely and accurate location of flames due to background interference in static image analysis technology, thereby improving the system's response speed and early warning reliability in the early stages of a fire. Example 1

[0029] refer to Figure 1The first embodiment of this application provides a visual perception-based intelligent fire identification and early warning method for robots, to solve the problem in the background art where static image analysis technology is unable to locate the flame burning position in a timely and accurate manner due to background interference, resulting in insensitive fire handling and early warning. This method can be executed by a processor, which can be set in a terminal or server inside the inspection robot. The execution process of the visual perception-based intelligent fire identification and early warning method for robots can be as follows: Step S101: The inspection robot's camera module takes panoramic photos of the monitored area to obtain a denoised panoramic image.

[0030] With the development of deep learning and computer vision technologies, vision-based perception systems have become more mature. Visual perception technology can capture environmental images in real time through cameras and combine them with image processing algorithms to efficiently identify features such as smoke and flames in a fire. Unlike traditional sensors, visual perception systems can provide richer information, accurately identify various fire scenarios, and provide a foundation for timely early warning.

[0031] The relevant functional modules in this embodiment are shown below: 1. Image Acquisition Module: This module mainly includes a panoramic camera at the site and a high-definition camera on the robot. It sets parameters such as the camera's field of view, resolution, and frame rate to improve image quality. (Example: Fire prevention inspection robot) Figure 2 As shown (for reference only): Specifically, panoramic image data is collected by monitoring on-site inspection robots and related hardware camera modules. This mainly includes on-site panoramic cameras and high-definition cameras on the robots, with parameters such as camera field of view, resolution, and frame rate set to improve image quality. High-resolution panoramic images of the site are acquired by the image acquisition module, and after noise reduction processing, relevant data is obtained. The panoramic image needs to include all areas of the monitored site (using relevant image stitching or fusion techniques). Simultaneously, the multi-view acquisition program for the inspection robot is initialized and ready for use.

[0032] 2. Image Recognition and Processing Module: This module mainly uses relevant computer vision algorithms to analyze the collected image data and identify and locate the significant features of fires such as flames and smoke.

[0033] 3. Data storage and analysis module: Store all collected images and data in the cloud platform or local storage. This module predicts and provides early warning analysis of fire characteristics based on changes in time-series data.

[0034] 4. Robot Early Warning and Processing Module: Based on data analysis results, this module sends alarm signals through alarm devices or in conjunction with other emergency systems.

[0035] Since flames have distinct visual characteristics, this embodiment first uses the image acquisition module to obtain a high-resolution panoramic image of the scene, performs noise reduction processing, and then acquires relevant data. Note that the panoramic image needs to include all areas of the monitoring site (using relevant image stitching or fusion techniques). Simultaneously, the multi-view acquisition program for the inspection robot is initialized and put into standby mode. Subsequently, different modules are linked to accurately identify and promptly warn of fires.

[0036] Step S102: Divide the panoramic image into multiple block images by grid. Based on the brightness level, local contrast and grayscale standard deviation of the pixels in the block images, calculate the combustion feature confidence of each block image. Blocks with combustion feature confidence greater than a preset threshold are identified as suspected partitions.

[0037] For on-site monitoring, at any given sampling time, the presence and location of flames are unknown. Therefore, the primary goal of flame identification is to determine the location of the fire as soon as it occurs. Analyzing panoramic images can cover all possible fire locations, preventing fires from going undetected and unannounced due to blind spots. However, since panoramic images typically contain a large amount of pixel information, directly analyzing them for flame identification involves high complexity, which is detrimental to timely fire warnings. Therefore, this step first divides the panoramic image into blocks (by using a grid to divide the panoramic image into multiple rectangular regions of equal length and width), and then performs coarse screening by analyzing the generalized image features of flames in different blocks, thereby reducing the complexity of subsequent precise identification.

[0038] In one embodiment of this application, the confidence level of the combustion features of each block image is calculated based on the brightness level value, local contrast, and grayscale standard deviation of pixels in the block image, including: By sliding a window within the block image, the difference between the maximum and minimum brightness values ​​within each window is calculated and accumulated. This difference is then combined with the average brightness value of the block image to calculate the local contrast convolution. A confidence calculation function for combustion features is constructed by combining local contrast convolution with the gray standard deviation of all pixels in the block image. Among them, the confidence level of combustion features is positively correlated with local contrast convolution and negatively correlated with gray standard deviation; when the gray standard deviation is smaller and the local contrast convolution is larger, the probability of determining that the block has flame features is higher.

[0039] Specifically, high brightness and high contrast are generally important characteristics of flames, and burning flames typically do not produce complex textures. Therefore, for any block of image containing burning flames, the corresponding image will necessarily possess these characteristics, allowing for coarse screening. For any block of image, the following characteristics are present: ; In the formula The brightness level value of this block of image. and These are the first in this block. The maximum and minimum brightness values ​​within each sliding window. The number of sliding windows (using 5x5 small windows to slide within the block). This is the average brightness value of this block of image. Then... This represents the local contrast convolution of this block. The larger the value, the higher the contrast and the higher the brightness level, which makes it more likely that flame combustion behavior exists.

[0040] Meanwhile, the burning of flames also exhibits distinct characteristics in terms of image texture, therefore, its texture features can be used for evaluation: ; In the formula The confidence level of the combustion features in a certain block of image. The brightness level value of this block of image. The grayscale standard deviation of all pixels in the image block is used to determine the texture. The smaller the grayscale standard deviation, the less significant the texture. If the brightness level of an image block is higher and the grayscale standard deviation is smaller, the confidence level is higher, indicating that the block has obvious flame features and may exhibit flame combustion behavior.

[0041] The feature confidence scores of all blocks are calculated using the method described above, and the results are normalized to the range of [0,1]. Based on experience, the blocks corresponding to the confidence thresholds are set as suspected blocks that may contain flames. Information on all suspected blocks is then uploaded to the image recognition and processing module for subsequent accurate identification.

[0042] Step S103: The inspection robot is dispatched to perform continuous image sampling of the suspected area from multiple perspectives and at different times, and to obtain multiple sampled images of the suspected target from different perspectives and at different times.

[0043] Since suspected blocks represent areas where flames may exist, in actual scenarios, due to background or object highlights and the influence of external light, the above coarse screening may include images of blocks that meet the characteristics but do not produce flames (non-target blocks). Directly issuing warnings and fire prevention for these non-target blocks would lead to a waste of fire protection resources. Therefore, this step first detects suspected targets based on suspected blocks using visual target recognition algorithms (such as YOLO) and then performs further feature recognition.

[0044] Because the combustion of flames exhibits significant dynamic changes over time and does not shift with changes in viewing angle, this characteristic allows for multi-view, multi-time-series image acquisition of all suspected targets. This step involves multiple inspection robots simultaneously sampling the suspected targets from multiple perspectives (in this embodiment, the sampling interval is set to 5 seconds, and the sampling duration to 1 minute), thereby obtaining multiple image samples for any suspected target, such as... Figure 2 As shown (t represents timing).

[0045] Step S104: Extract the saliency of combustion features of suspected targets in the sampled image, calculate the horizontal feature similarity of different viewpoints under the same time sequence, and the vertical scale trend value of different time sequences under the same viewpoint; fuse the horizontal feature similarity and the vertical scale trend value, calculate the second confidence of combustion features of suspected targets, and determine the suspected targets whose second confidence of combustion features meets the preset judgment conditions as the target locations where flames are actually generated.

[0046] These images reflect the dynamic changes of the target to some extent, so the changing characteristics of these images can be analyzed for accurate identification. First, determine the similarity of their lateral combustion characteristics, that is, if the suspected target is burning with flames, its combustion characteristics from different perspectives at the same time will be similar.

[0047] Generally, flame combustion produces smoke; the simultaneous presence of smoke and flame is a key image feature of flame combustion, which is absent in other background interferences. Smoke and flame exhibit a significant grayscale difference in the image (flame's grayscale range is typically 180–255, while smoke's is typically 50–130). Furthermore, the initial smoke production is larger than the flame itself; that is, the grayscale histogram corresponding to the target image containing flame combustion shows two distinct peaks with differing areas. Figure 3 As shown.

[0048] In one embodiment of this application, extracting the saliency of combustion features of a suspected target in a sampled image includes: Obtain the grayscale histogram of the region corresponding to the suspected target in the sampled image; Identify the two largest peaks in the grayscale histogram, extract the peak areas corresponding to the two largest peaks, and the grayscale values ​​corresponding to the two largest peaks. The saliency of combustion characteristics is determined by multiplying the absolute value of the difference in peak area between the two largest peaks with the absolute value of the difference in gray value; wherein, the saliency of combustion characteristics is used to characterize the bimodal gray-scale physical characteristics of the coexistence of flame and smoke.

[0049] Furthermore, based on the above characteristics, the grayscale histogram of any image possesses the following: ; In the formula The significance of the combustion features of a suspected target in a sampled image. and These represent the peak areas corresponding to the two largest peaks in the grayscale histogram of the suspected target. and These are the grayscale values ​​corresponding to the peak values ​​of the two largest peaks. Then... The larger the value, the greater the difference in peak area. The larger the value, the greater the difference in grayscale range between the two peaks. The larger the value, the more it matches the characteristics of a burning flame for the suspected target.

[0050] Therefore, by using the above method, the feature saliency of multiple suspected target images from different perspectives at the same time is calculated, and then a horizontal comparison is performed to calculate their similarity index: ; In the formula In order to be in The horizontal feature similarity of this suspected target under different viewpoints at any given time. and These represent the saliency of the combustion features in the suspected target image from any viewpoint at this moment. This represents the number of sampled viewpoints. The smaller the value, the greater the similarity between the suspected targets in the image, and the more likely it is to be an image of a target with flames burning.

[0051] Since there are multiple sampling times, the feature similarity of this suspected target at multiple sampling times can be calculated using the above method. A comprehensive analysis will then be conducted, taking into account the longitudinal dynamic differences.

[0052] Then, the longitudinal combustion dynamics differences are determined based on the sampled images. Generally, flames burn more vigorously over time; that is, the size of suspected targets in images with burning flames gradually increases over time, while background highlights and illumination effects represent static targets. Here, the real-time combustion scale of suspected targets in any sampled image at the same viewpoint is calculated based on longitudinal time series: ; In the formula Therefore, the first sampled image under the same viewpoint Real-time burning scale of suspected targets at each sampling time. This refers to the total number of pixels in the image. This represents the number of pixels in the suspected target area. Therefore, this is the brightness level of the suspected target area. The larger the value, the larger the suspected target burning scale at that moment.

[0053] Therefore, from the same perspective, since the scale of flame combustion tends to gradually increase, it is also necessary to analyze the trend of combustion scale changes at different sampling times to determine whether it is the target image. As follows: ; In the formula From a certain perspective Vertical scale trend value of suspected targets. and These are the first two perspectives from which the following perspectives are presented. The sampling time and the first The real-time combustion scale of this suspected target at each sampling time. This represents the number of samples. This represents the growth trend of the real-time combustion scale. The larger the value, the more obvious the growth trend of the suspected target scale, and the more consistent it is with the trend characteristics of flame combustion. Conversely, the smaller the value and the closer it is to 0, the more likely the suspected target is a static target.

[0054] Since there are multiple sampling perspectives, it is necessary to calculate the scale trend of this suspected target under multiple perspectives using the method described above: Thus, for this suspected target, the similarity of its lateral multi-view features has been analyzed to obtain... Furthermore, by analyzing the trend changes of its longitudinal time series, we can obtain... Therefore, combining the above characteristics, if the horizontal similarity of the suspected target image at different times better satisfies the vertical trend and is smaller, while the vertical scale trend at different viewpoints better satisfies the horizontal similarity and is larger, then the likelihood of it being an actual flame burning target is greater. As shown below: ;

[0055] In the formula Therefore, the second confidence level of the combustion features of the suspected target image. and These represent the average horizontal feature similarity and the average vertical scale trend value of this image, respectively. Horizontal feature similarity at different time points The vertical trend value, Vertical scale trend values ​​from different perspectives The horizontal similarity.

[0056] Then if The larger the value, the more closely the suspected target matches the characteristics of flame combustion, and the greater the likelihood that it is an actual flame combustion target. Here, the suspected targets in all suspected blocks are calculated using the above method, and the results are normalized to the range [0,1]. The corresponding suspected target is the actual location of the burning flame (if there is no suspected target that meets the conditions, it can be assumed that no flame is generated and no processing is required). This further improves the accuracy of flame recognition in the scene.

[0057] In an optional embodiment of this application, the second confidence level of the combustion characteristics of a suspected target is calculated by fusing horizontal feature similarity and vertical scale trend value, including: The mean of the sum of squared differences in the saliency of combustion features from any two different perspectives within the same time series is used as the lateral feature similarity. Based on sampled images from different time sequences under the same viewpoint, the product of the number of pixels in the suspected target area and the brightness level is calculated as the real-time combustion scale, and the difference between the real-time combustion scales in adjacent time sequences is accumulated as the longitudinal scale trend value. Calculate the longitudinal trend value of the horizontal feature similarity under different time series, and the horizontal similarity of the longitudinal scale trend value under different perspectives; The average horizontal feature similarity, the average vertical scale trend value, the vertical trend value of the horizontal feature similarity, and the horizontal similarity of the vertical scale trend value are weighted and fused to obtain the second confidence level of the combustion feature.

[0058] Step S105: Based on the real-time combustion scale of the target location under multiple time series as a feature factor, input it into the pre-constructed time series prediction model to obtain the predicted values ​​of the combustion scale at multiple future times; establish a growth curve based on the real-time combustion scale and the predicted values ​​of the combustion scale, determine the fire-fighting priority of the corresponding target location according to the geometric characteristics of the growth curve, and control the inspection robot to execute the corresponding fire warning and fire-fighting actions based on the fire-fighting priority.

[0059] At this point, all target locations have been identified. If only one location in this scenario is burning, the fire inspection robot can immediately activate the early warning system and simultaneously control and extinguish the fire through its fire suppression system after confirming this target location. However, it is also necessary to consider the possibility that if the above detection results show multiple target locations, i.e., multiple fires, and the fires in different locations may not be able to be extinguished simultaneously if there are insufficient fire robots, a fire suppression priority mechanism needs to be established.

[0060] If multiple target locations catch fire, we want to prioritize the target locations with the greater burning growth trend to prevent the fire from getting out of control. Therefore, in this embodiment, the real-time burning scale of all target locations from any viewpoint is used as a feature factor, and then the burning scale at multiple future moments is predicted using the ARIMA model.

[0061] Then, the predicted and calculated values ​​of real-time combustion scale are integrated to establish a corresponding growth curve, such as... Figure 4 As shown.

[0062] In one embodiment of this application, the time-series prediction model is an autoregressive integral moving average model; before inputting the real-time combustion scale of the target location at multiple time series as a feature factor into the pre-constructed time-series prediction model to obtain the predicted combustion scale values ​​for multiple future time periods, the method further includes: If only one target location is determined through step S104, the nearest inspection robot is directly controlled to move to the target location and the fire extinguishing device is activated. If multiple target locations are determined through step S104, the firefighting priority calculation process in step S105 is triggered.

[0063] In one optional embodiment of this application, a growth curve is established based on the real-time combustion scale and the predicted combustion scale, and the firefighting priority for the corresponding target location is determined according to the geometric characteristics of the growth curve, including: Based on the predicted combustion scale output by the time-series prediction model and the historical calculated values, a combustion scale growth curve at the target location is generated by fitting. Extract the slope of the growth curve and the area of ​​the integral between the growth curve and the horizontal time axis; The slope and the integral area are weighted and summed, and the sum is used as the processing priority for the target location; where the slope represents the instantaneous spread rate of the fire, and the integral area represents the overall burning scale of the fire. The movement control commands for the inspection robot and the fire resource scheduling strategy are generated in descending order of processing priority.

[0064] Furthermore, since the growth curve slope is steeper for target locations with more pronounced growth trends, and the curve is steeper for larger combustion scales, the processing priority for different target locations can be expressed as follows: ; In the formula For a certain target location The processing priority. The slope of the curve corresponding to this target location. Therefore, the area between the curve and the horizontal axis corresponding to this target location. The larger the value, the larger the fire scale at the target location and the more intense the predicted growth trend. The higher the processing priority, the more the limited inspection robots should extinguish the fire at this location as much as possible (this priority is unrelated to the early warning), and so on.

[0065] Thus, the aforementioned method, based on visual perception, has achieved accurate monitoring of initial flames and fire early warning processing. Subsequent collection and analysis of new fire image data can continuously update relevant parameters, enhance recognition capacity, maintain efficient recognition capabilities, and establish corresponding early warning mechanisms. Furthermore, for more complex scenarios, it is recommended to combine relevant sensors for multimodal data fusion to further improve fire identification accuracy.

[0066] Example 2

[0067] To address the aforementioned technical problems, the second embodiment of this application provides a visual perception-based intelligent fire identification and early warning system for robots, solving the same technical problems as the method embodiment. This system 1000 may include the following modules: The image acquisition and preprocessing module 1001 is used to acquire panoramic images through the camera module of the inspection robot and perform noise reduction processing; The global coarse screening module 1002 is used to divide the panoramic image into multiple blocks, calculate the confidence level of combustion features based on the brightness level, local contrast and grayscale standard deviation of pixels, and screen out suspected partitions. The multi-view temporal sampling and fine recognition module 1003 is used to control the robot to perform multi-view and multi-temporal sampling on suspected areas. Based on the bimodal feature of the gray-level histogram of suspected targets in the sampled images, the saliency of combustion features is calculated. The second confidence of combustion features is calculated by combining the horizontal multi-view feature similarity and the vertical temporal scale trend value to determine the target location where the actual flame is generated. The trend prediction and scheduling control module 1004 is used to generate predicted values ​​for future times based on the real-time combustion scale of the target location using a time-series prediction model to construct a growth curve. Based on the slope and area of ​​the growth curve, it calculates the fire-fighting priority of multiple target locations and controls the inspection robot to perform early warning and fire-fighting actions according to the priority.

[0068] It is understood that this embodiment is the apparatus / system claim corresponding to the method claim, and therefore this embodiment has all the technical effects of the method claim, which will not be repeated here.

[0069] Example 3

[0070] The third embodiment of this application also provides an electronic device, including: At least one processor; and memory that is communicatively connected to at least one processor; The memory stores a computer program that can be executed by at least one processor, which enables the at least one processor to execute the vision-based robotic fire intelligent identification and early warning method provided in the previous embodiment.

[0071] Example 4

[0072] The fourth embodiment of this application also provides a computer-readable storage medium storing computer instructions for causing a computer to execute the vision-based robotic fire intelligent identification and early warning method provided in the preceding embodiments.

[0073] Example 5

[0074] The fifth embodiment of this application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the vision-based robotic fire intelligent identification and early warning method provided in the previous embodiments.

[0075] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. A method for intelligent identification and early warning of fires using robots based on visual perception, characterized in that, Includes the following steps: The inspection robot's camera module captures a panoramic view of the monitored area to obtain a noise-reduced panoramic image. The panoramic image is divided into multiple block images by grid. Based on the brightness level, local contrast and grayscale standard deviation of the pixels in the block images, the combustion feature confidence of each block image is calculated. Blocks with combustion feature confidence greater than a preset threshold are identified as suspected partitions. The inspection robot is scheduled to perform continuous image sampling of the suspected area from multiple perspectives and at different times, thereby acquiring multiple sampled images of the suspected target from different perspectives and at different times. Extract the saliency of combustion features of suspected targets in the sampled images, calculate the horizontal feature similarity of different viewpoints at the same time sequence, and the vertical scale trend value of different time sequences at the same viewpoint; By combining the horizontal feature similarity and the vertical scale trend value, the second confidence level of the combustion feature of the suspected target is calculated, and the suspected target whose second confidence level of the combustion feature meets the preset judgment condition is determined as the target location where the flame is actually generated. Based on the real-time combustion scale of the target location in multiple time series as a feature factor, it is input into a pre-built time series prediction model to obtain the predicted values ​​of combustion scale at multiple future times. A growth curve is established based on the real-time combustion scale and the predicted combustion scale. The fire suppression priority of the corresponding target location is determined according to the geometric characteristics of the growth curve. Based on the fire suppression priority, the inspection robot is controlled to perform the corresponding fire warning and fire suppression actions.

2. The method for intelligent identification and early warning of robot fires based on visual perception according to claim 1, characterized in that, The calculation of the combustion feature confidence level of each block image based on the brightness level value, local contrast, and grayscale standard deviation of pixels in the block image includes: By sliding a window within the block image, the difference between the maximum and minimum brightness values ​​within each window is calculated and accumulated. This difference is then combined with the average brightness value of the block image to calculate the local contrast convolution. A combustion feature confidence calculation function is constructed by combining the local contrast convolution with the gray standard deviation of all pixels in the block image; The confidence level of the combustion feature is positively correlated with the local contrast convolution and negatively correlated with the grayscale standard deviation. The smaller the grayscale standard deviation and the larger the local contrast convolution, the higher the probability that the block has flame features.

3. The method for intelligent identification and early warning of robot fires based on visual perception according to claim 1, characterized in that, The extraction of the saliency of combustion features of suspected targets in the sampled image includes: Obtain the grayscale histogram of the region corresponding to the suspected target in the sampled image; Identify the two largest peaks in the grayscale histogram, extract the peak areas corresponding to the two largest peaks, and the grayscale values ​​corresponding to the two largest peak points; The saliency of the combustion feature is determined by multiplying the absolute value of the difference in peak area between the two largest peaks with the absolute value of the difference in gray value; wherein, the saliency of the combustion feature is used to characterize the bimodal gray-scale physical characteristics of the coexistence of flame and smoke.

4. The method for intelligent identification and early warning of robot fires based on visual perception according to claim 3, characterized in that, The process of fusing the horizontal feature similarity and the vertical scale trend value to calculate the second confidence level of the combustion features of the suspected target includes: The mean of the sum of squared differences in the saliency of combustion features from any two different perspectives within the same time series is used as the lateral feature similarity. Based on sampled images from different time sequences under the same viewpoint, the product of the number of pixels in the suspected target area and the brightness level is calculated as the real-time combustion scale, and the difference between the real-time combustion scales in adjacent time sequences is accumulated as the longitudinal scale trend value. Calculate the longitudinal trend value of the horizontal feature similarity under different time series, and the horizontal similarity of the longitudinal scale trend value under different perspectives; The average horizontal feature similarity, the average vertical scale trend value, the vertical trend value of the horizontal feature similarity, and the horizontal similarity of the vertical scale trend value are weighted and fused to obtain the second confidence level of the combustion feature.

5. The method for intelligent identification and early warning of robot fires based on visual perception according to claim 4, characterized in that, The step of establishing a growth curve based on the real-time combustion scale and the predicted combustion scale, and determining the firefighting priority for the corresponding target location based on the geometric characteristics of the growth curve, includes: Based on the predicted combustion scale value output by the time-series prediction model and the historical calculated value, a combustion scale growth curve at the target location is fitted and generated. Extract the slope of the growth curve and the area of ​​the integral between the growth curve and the horizontal axis of time; The slope and the integral area are weighted and summed, and the sum is used as the processing priority for the target location; wherein, the slope represents the instantaneous spread rate of the fire, and the integral area represents the overall burning scale of the fire. The movement control commands and fire resource scheduling strategies for the inspection robot are generated in descending order of processing priority.

6. The method for intelligent identification and early warning of robot fires based on visual perception according to claim 1, characterized in that, The time-series prediction model is an autoregressive integral moving average model; before inputting the real-time combustion scale at the target location under multiple time series as a feature factor into the pre-constructed time-series prediction model to obtain the predicted combustion scale values ​​for multiple future times, the method further includes: If only one target location is determined through step S4, then the nearest inspection robot is directly controlled to move to that target location and the fire extinguishing device is activated. If multiple target locations are determined through step S4, the firefighting priority calculation process in step S5 will be triggered.

7. A robot-based intelligent fire identification and early warning system based on visual perception, characterized in that, The system, applicable to the method of any one of claims 1-6, comprises: The image acquisition and preprocessing module is used to acquire panoramic images through the camera module of the inspection robot and perform noise reduction processing; The global coarse screening module is used to divide the panoramic image into multiple blocks, calculate the confidence level of combustion features based on the brightness level, local contrast and grayscale standard deviation of pixels, and screen out suspected partitions. The multi-view temporal sampling and fine recognition module is used to control the robot to perform multi-view and multi-temporal sampling on suspected areas. Based on the bimodal feature of the gray-level histogram of suspected targets in the sampled images, the saliency of combustion features is calculated. The second confidence of combustion features is calculated by combining the horizontal multi-view feature similarity and the vertical temporal scale trend value to determine the target location where the actual flame is generated. The trend prediction and scheduling control module is used to generate predicted values ​​for future times based on the real-time combustion scale of the target location using a time-series prediction model to construct a growth curve. Based on the slope and area of ​​the growth curve, it calculates the fire-fighting priority of multiple target locations and controls the inspection robot to perform early warning and fire-fighting actions according to the priority.

8. An electronic device, characterized in that, include: At least one processor; and a memory communicatively connected to the at least one processor; The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the method as described in any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the method as described in any one of claims 1-6.

10. A computer program product, characterized in that, The invention includes a computer program / instruction, characterized in that, when executed by a processor, the computer program / instruction implements the method described in any one of claims 1-6.

Images