Electrical equipment operation monitoring system and method based on flame smoke video analysis

Abstract

The invention discloses an electrical equipment operation monitoring system and method based on flame smoke video analysis, and relates to the field of electrical equipment operation monitoring, and the method comprises the steps: carrying out the preliminary evaluation of an electrical equipment region, and constructing a three-dimensional model of the electrical equipment region; key monitoring positions are analyzed and identified, a high-definition camera is installed, and a visual angle test is carried out; an electrical equipment area video is collected in real time, and a suspected flame or smoke area is marked; extracting flame and smoke areas according to the suspected flame or smoke area; calculating the position of a fire source in a three-dimensional space according to the flame area; when the smoke area is extracted but the flame area is not extracted, analyzing the position of the fire source in the three-dimensional space by combining the smoke shape, the flow direction and the convergent point detected by the plurality of high-definition cameras; and when the fire source is marked, triggering an alarm device, sending the three-dimensional model of the electrical equipment area with the fire source position mark to a terminal of a user, and carrying out emergency power-off.

Description

Technical Field

[0001] The present invention relates to the field of operation monitoring of electrical equipment, and specifically to an operation monitoring system and method for electrical equipment based on flame and smoke video analysis. Background Art

[0002] With the wide application of electrical equipment, traditional electrical equipment monitoring systems generally rely on manual inspections or basic temperature sensors and smoke detectors for monitoring. However, these methods have defects such as lagging response and insufficient accuracy. Especially in the complex layout of electrical equipment, it is difficult for traditional monitoring means to comprehensively, real-time and accurately monitor the specific location of a fire. In recent years, fire monitoring technologies based on video image analysis have gradually been applied. However, the existing flame and smoke recognition methods still have problems such as insufficient recognition accuracy and long processing delay, and cannot accurately locate the fire source position in the first time, affecting the effect of fire warning.

[0003] The prior art mostly relies on single monitoring devices, such as smoke sensors or infrared temperature sensors, and these sensors have limitations in terms of accuracy, response speed, etc. Especially when the electrical equipment area is complex and the spatial layout is irregular, there are many blind spots in traditional monitoring means, and it is often impossible to accurately locate the fire source position. Summary of the Invention

[0004] The purpose of the present invention is to provide an operation monitoring system and method for electrical equipment based on flame and smoke video analysis to solve the problems raised in the prior art.

[0005] To achieve the above purpose, the present invention provides the following technical solution: An operation monitoring method for electrical equipment based on flame and smoke video analysis, the method comprising:

[0006] Preliminarily evaluate the layout and spatial position of the electrical equipment area, and construct a three-dimensional model of the electrical equipment area;

[0007] Based on the three-dimensional model of the electrical equipment area, analyze and identify key monitoring positions, install high-definition cameras, and conduct perspective tests to ensure that there are no blind spots in key areas;

[0008] Based on the installed high-definition cameras, collect videos of the electrical equipment area in real time; and identify whether there is flame or smoke in the videos, and mark the suspected flame or smoke areas;

[0009] When it is identified that there is flame or smoke in the video, extract the flame or smoke area according to the suspected flame or smoke area;

[0010] According to the flame area, calculate the position of the fire source in three-dimensional space, and mark the position of the fire source in three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area;

[0011] When the smoke area is extracted but the flame area is not, combine the shapes, flow directions, and convergence points of the smoke detected by multiple high-definition cameras to analyze the position of the fire source in three-dimensional space, and mark the position of the fire source in three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area;

[0012] After marking the fire source in the three-dimensional model of the electrical equipment area, immediately trigger the alarm device, send the three-dimensional model of the electrical equipment area with the marked fire source position to the user's terminal, and perform emergency power-off.

[0013] According to the above solution, install high-definition cameras based on each of the key monitoring positions and conduct perspective tests; the perspective tests include: analyzing the perspective range covered by each high-definition camera, the size of the monitoring area, and the space and layout around the electrical equipment to ensure that the high-definition cameras cover all key areas and the pictures of the key areas are clear without blind spots;

[0014] When the perspective test finds that there are blind spots or unclear pictures in the key areas, adjust the installation angles and positions of the high-definition cameras or increase the number of high-definition cameras.

[0015] According to the above solution, process the video of the electrical equipment area into frames of images to form video frames;

[0016] Perform Gaussian filtering denoising on the video frames to remove high-frequency noise and smooth the images;

[0017] Use the Gaussian mixture model to establish multiple Gaussian distributions for the color or brightness values of each pixel, and perform background modeling on each pixel based on the multiple Gaussian distributions. Different Gaussian distributions represent different states in the background;

[0018] When the pixel value in the background differs from the Gaussian distribution in the modeling by more than the set threshold, the pixel value will be marked as a suspected flame or smoke area.

[0019] According to the above solution, when a flame exists in the video, extract the flame area based on the suspected flame or smoke area, including:

[0020] Based on the suspected flame or smoke area, perform color feature analysis to obtain the color information of the flame and mark it as the first verified flame area; based on the suspected flame or smoke area, perform edge detection to obtain the contour information of the flame and mark it as the second verified flame area; combine the first verified flame area and the second verified flame area to analyze and extract the flame area.

[0021] According to the above solution, the color feature analysis includes:

[0022] Convert the suspected flame or smoke area from the RGB color space to the HSV color space; for the HSV color space, based on hue, saturation, and value, by setting the ranges of hue, saturation, and value, and comparing the HSC values of each pixel in the suspected flame or smoke area with the ranges of hue, saturation, and value; if the HSV value of a pixel is within the ranges of hue, saturation, and value, then mark this pixel as the first verified flame area;

[0023] The edge detection includes:

[0024] Convert the suspected flame or smoke area from an RGB color image to a grayscale image;

[0025] Perform Gaussian smoothing on the grayscale image. For the Gaussian smoothing, construct a Gaussian kernel centered on each pixel in the grayscale image, and through the convolution operation of the Gaussian kernel and the pixels of the grayscale image, achieve the Gaussian smoothing process of the grayscale image to reduce the influence of noise;

[0026] Calculate the gradient magnitude and gradient direction of each pixel in the Gaussian-smoothed grayscale image. The gradient magnitude represents the degree of sharp change in the pixel values in the grayscale image, and the gradient direction represents the direction in which the pixel values change fastest;

[0027] According to the gradient direction θ1 of pixel s1 in the grayscale image, determine the two adjacent pixels s0 and s2 of the pixel in the gradient direction in the grayscale image, and compare the gradient magnitude G1 of pixel s1 in the grayscale image with the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2; when the gradient magnitude G1 of pixel s1 in the grayscale image is greater than the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2, then retain the gradient magnitude G1 of pixel s1 in the grayscale image; when the gradient magnitude G1 of pixel s1 in the grayscale image is smaller than any one of the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2, then set the gradient magnitude G1 of pixel s1 in the grayscale image to 0;

[0028] Set a high threshold T h and a low threshold T l , which are used to detect edge pixels; when the gradient magnitude of a pixel in the grayscale image is greater than the high threshold T h , then determine that the pixel in the grayscale image is a strong edge pixel; when the gradient magnitude of a pixel in the grayscale image is less than the low threshold T l , then determine that the pixel in the grayscale image is not an edge pixel; when the gradient magnitude of a pixel in the grayscale image is between the high threshold T h and the low threshold T lWhen it is between them, it is determined that the pixel in the grayscale image is a weak edge pixel; starting from the strong edge pixel, check the weak edge pixels adjacent to the strong edge pixel. When the adjacent weak edge pixel is connected to the strong edge pixel, mark the adjacent weak edge pixel as an edge pixel; complete the connection of the edges, obtain clear flame contour information, and mark the pixel area in the flame contour information as the second verified flame area;

[0029] Perform an intersection operation on the pixels in the first verified flame area and the second verified flame area; when the pixel is simultaneously a pixel in the first verified flame area and the second verified flame area, it is determined as a pixel in the flame area; extract all pixels that are simultaneously in the first verified flame area and the second verified flame area to form the flame area.

[0030] According to the above solution, when it is recognized that there is smoke in the video, based on the suspected flame or smoke area, the smoke area is extracted, including:

[0031] Based on the suspected flame or smoke area, perform grayscale threshold segmentation to generate a binary region image and mark it as the first verified smoke area; based on the first verified smoke area, use edge detection and contour detection to obtain the smoke shape within the smoke template range and mark it as the second verified smoke area; based on the suspected flame or smoke area, perform optical flow method calculation to obtain the smoke motion characteristics and mark it as the third verified smoke area; combine the second verified smoke area and the third verified smoke area to analyze and extract the smoke area.

[0032] According to the above solution, the grayscale threshold segmentation includes:

[0033] Convert the suspected flame or smoke area from an RGB color image to a grayscale image;

[0034] Based on the smoke grayscale characteristics, set the grayscale threshold H through sample analysis; compare the grayscale value of each pixel in the grayscale image with the grayscale threshold H; when the grayscale value of the pixel in the grayscale image is less than or equal to the grayscale threshold H, mark the pixel as a pixel in the first verified smoke area; when the grayscale value of the pixel in the grayscale image is greater than the grayscale threshold H, mark it as a non-smoke area; all pixels marked as the first verified smoke area are combined into the first verified smoke area;

[0035] The edge detection and contour detection include:

[0036] Perform edge detection on the first verified smoke area to obtain smoke edge information;

[0037] Based on the smoke edge information, use a contour detection algorithm to detect the contour of the smoke area, calculate the perimeter of the smoke area contour, determine the area enclosed by the smoke area contour, and calculate the aspect ratio through the length and width of the circumscribed rectangle of the smoke area contour;

[0038] Compare the perimeter, area, and aspect ratio of the smoke area contour with the smoke model; when the perimeter, area, and aspect ratio of the smoke area contour are within the range of the smoke template, then mark the area within the smoke area contour as the second verified smoke area;

[0039] The optical flow method includes:

[0040] Based on the video of the electrical equipment area, extract two consecutive frames of images containing the suspected flame or smoke area, perform grayscale and Gaussian filtering processing on the suspected flame or smoke area in the images to obtain a denoised grayscale image;

[0041] Based on the denoised grayscale image, use the Shi-Tomasi corner detection algorithm to select feature points; the Shi-Tomasi corner detection algorithm obtains points with significant gray-scale changes in multiple directions as corner points by calculating the gray-scale changes of each pixel point in the denoised grayscale image in different directions; screen the corner points, remove points with overly smooth edges or insignificant gray-scale changes, and retain a representative set of feature points P = {p1, p2,..., p n}, where p n represents a feature point and n represents the total number of feature points; set the adjacent pixel points of the feature point P i as the points within the neighborhood of the feature point P i , establish an optical flow constraint equation for each point within the neighborhood of the feature point P i , and obtain the optical flow vector of the feature point P i as (u i , v i ), where (u, v) represents the displacement of the feature point between the two consecutive frames of images containing the suspected flame or smoke area; calculate the optical flow vector of each feature point P to obtain an optical flow vector set V = {(u1, v1), (u2, v2),..., (u n , v n )}; calculate the direction of each optical flow vector and perform statistical analysis on the optical flow directions of all feature points; when the optical flow direction shows a diffusion trend, then judge it as a smoke area and mark it as the third verified smoke area;

[0042] Combined with the second verified smoke area and the third verified smoke area, analyze the overlapping pixel area of the second verified smoke area and the third verified smoke area, and the overlapping pixel area is the smoke area.

[0043] According to the above solution, calculating the position of the fire source in three-dimensional space based on the flame area and marking the position of the fire source in three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area includes:

[0044] When only one high-definition camera detects the flame area, calculate the position of the fire source in three-dimensional space based on the installation position, viewing angle of the high-definition camera, and the three-dimensional model of the electrical equipment area;

[0045] When multiple high-definition cameras detect the flame area, utilize the viewing angle and position differences of each camera to jointly estimate the position of the fire source through spatial geometric relationships;

[0046] Mark the position of the fire source in three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area.

[0047] According to the above solution, establish a projection relationship from the two-dimensional image coordinates captured by the high-definition camera to the three-dimensional space coordinates based on the installation position, viewing angle, and imaging principle of the high-definition camera;

[0048] Based on the smoke area detected by the high-definition camera, convert the smoke shape, flow direction, and convergence point of the two-dimensional image into coordinate information in three-dimensional space according to the projection relationship;

[0049] Based on the smoke flow direction and convergence point detected by the multiple high-definition cameras, draw the motion vector and convergence point range of the smoke in the three-dimensional space;

[0050] Based on the comprehensive analysis of the fire source direction inferred from the smoke flow direction and the position information of the convergence point, estimate the position of the fire source in three-dimensional space and mark the position of the fire source in three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area.

[0051] An electrical equipment operation monitoring system based on flame and smoke video analysis, the system includes: a three-dimensional model module, a flame or smoke detection module, a fire source positioning module, and a fire source alarm module;

[0052] The 3D model module includes a perspective test module, a video acquisition module, and a model construction module. The model construction module conducts a preliminary assessment of the electrical equipment area and constructs a 3D model of the electrical equipment area. The perspective test module identifies key monitoring positions based on the model construction module and conducts perspective tests by installing high-definition cameras to ensure that each key monitoring area is clearly covered by the high-definition cameras. The video acquisition module real-time collects video data of the electrical equipment area through the installed high-definition cameras.

[0053] The flame or smoke detection module includes a flame or smoke extraction module, a flame verification module, and a smoke verification module. The flame or smoke extraction module analyzes the color or brightness change of each pixel using a Gaussian mixture model, detects the difference from the background, and thus determines whether there is a suspected flame or smoke area. The flame verification module extracts the suspected flame or smoke area through color feature analysis and edge detection, and analyzes it in combination with the first verified flame area and the second verified flame area to finally confirm the accurate position of the flame area. The smoke verification module generates a first smoke verification area through gray threshold segmentation, then obtains a second smoke verification area through edge detection and contour detection, and further analyzes the motion characteristics of the smoke using the optical flow method to obtain a third smoke verification area. Combining the second smoke verification area and the third smoke verification area, it confirms the accurate position of the smoke area.

[0054] The fire source positioning module includes a flame fire source positioning module and a smoke fire source positioning module. The flame fire source positioning module calculates the fire source position based on a single or multiple cameras. The smoke fire source positioning module calculates the fire source position based on the shape, flow direction, and convergence point of the smoke.

[0055] The fire source alarm module displays the 3D model marked with the fire source position to the user through the terminal interface and provides real-time monitoring, alarm, and operation feedback functions.

[0056] Compared with the prior art, the beneficial effects of the present invention are:

[0057] 1. Through a multi-step verification process, the present invention reduces the misjudgment of flames and smoke, and improves the accuracy and reliability of the operation monitoring of electrical equipment.

[0058] 2. Based on various situations, the present invention locates the fire source and provides the accurate 3D model position of the fire source, providing accurate position information for taking timely measures.

[0059] 3. Through the alarm device, the present invention sends a 3D model marked with the fire source position to the user terminal and performs emergency power-off to ensure the safety of personnel and equipment. Brief Description of the Drawings

[0060] Figure 1The step process of the electrical equipment operation monitoring method based on flame and smoke video analysis of the present invention;

[0061] Figure 2 The step process of the multi-step verification of flame and smoke for the electrical equipment operation monitoring method based on flame and smoke video analysis of the present invention;

[0062] Figure 3 The structural schematic diagram of the electrical equipment operation monitoring system based on flame and smoke video analysis of the present invention. Specific embodiments

[0063] Next, the technical solutions in the embodiments of the present invention will be clearly and completely described in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

[0064] Embodiment: As Figures 1-3 shown, the present invention provides a technical solution, an electrical equipment operation monitoring method based on flame and smoke video analysis; the method includes the steps:

[0065] S1. Conduct a preliminary assessment of the layout and spatial position of the electrical equipment area, and construct a three-dimensional model of the electrical equipment area;

[0066] Specifically, for example: a large electrical equipment room with an area of 50m×30m, including distribution cabinets, transformers, cable channels, and equipment operation consoles inside; through laser scanning, conduct a preliminary assessment of the layout and spatial position of the electrical equipment area, and use three-dimensional modeling software to create a three-dimensional model of the electrical equipment area;

[0067] S2. Based on the three-dimensional model of the electrical equipment area, analyze and identify key monitoring positions, install high-definition cameras, and conduct perspective tests to ensure that there are no blind spots in the key areas;

[0068] Specifically, based on each key monitoring location, high-definition cameras are installed. For example, 4 high-definition cameras are installed near the power distribution cabinet, covering the front and sides of the power distribution cabinet; 6 high-definition cameras are installed on the transformer, covering the four corners and the middle area of the transformer; 2 high-definition cameras are installed in the cable channel, located at both ends of the channel; 2 high-definition cameras are installed on the equipment operation console, covering above and on the side of the equipment operation console. Next, a perspective test is carried out. The perspective test includes: analyzing the perspective range covered by each high-definition camera, the size of the monitoring area, and the space and layout around the electrical equipment to ensure that the high-definition cameras cover the key areas and the pictures of the key areas are clear without blind spots. When blind spots or unclear pictures are found in the key areas during the perspective test, adjust the installation angle and position of the high-definition cameras or increase the number of high-definition cameras.

[0069] S3. Based on the installed high-definition cameras, the video of the electrical equipment area is collected in real time; and it is identified whether there is fire or smoke in the video, and the suspected fire or smoke area is marked.

[0070] Specifically, the video of the electrical equipment area is processed into one frame of image after another to form video frames; Gaussian filtering denoising is performed on the video frames to remove high-frequency noise and smooth the images; using the Gaussian mixture model, by establishing multiple Gaussian distributions for the color or brightness value of each pixel, and based on the multiple Gaussian distributions, background modeling is performed for each pixel. Different Gaussian distributions represent different states in the background. When in the background, the pixel value differs from the Gaussian distribution in the modeling by more than the set threshold, the pixel value will be marked as a suspected fire or smoke area. For example: in a certain frame of image, the pixel point located at (500, 600) has a large difference in color value from the background model, exceeding the set threshold, and the threshold is set to 3 times the standard deviation, then this pixel point is marked as a suspected fire or smoke area.

[0071] S4. When it is identified that there is fire in the video, the fire area is extracted according to the suspected fire or smoke area.

[0072] Specifically, based on the suspected fire or smoke area, color feature analysis is carried out to obtain the color information of the fire and mark it as the first verified fire area; based on the suspected fire or smoke area, edge detection is carried out to obtain the contour information of the fire and mark it as the second verified fire area; combining the first verified fire area and the second verified fire area, the fire area is analyzed and extracted.

[0073] Further, the color feature analysis includes: converting the suspected flame or smoke area from the RGB color space to the HSV color space; in the HSV color space, based on hue, saturation, and value, by setting the ranges of hue, saturation, and value, for example: setting the hue range of the flame to [0, 30] (0 - 360 degrees), the saturation range to [120, 255], and the value range to [120, 255], and comparing the HSC values of each pixel in the suspected flame or smoke area with the ranges of hue, saturation, and value; if the HSV value of the pixel is within the ranges of hue, saturation, and value, then mark the pixel as the first verified flame area; for example: a pixel at a certain pixel coordinate (550, 650) in the suspected flame or smoke area, its HSV value is (20, 150, 180), which meets the set range and is marked as a pixel in the first verified flame area.

[0074] Further, edge detection includes: converting the suspected flame or smoke area from an RGB color image to a grayscale image; performing Gaussian smoothing on the grayscale image. Gaussian smoothing constructs a Gaussian kernel centered on each pixel in the grayscale image, and through the convolution operation of the Gaussian kernel and the pixels of the grayscale image, the Gaussian smoothing process of the grayscale image is realized to reduce the influence of noise; calculating the gradient magnitude and gradient direction of each pixel in the Gaussian-smoothed grayscale image. The gradient magnitude represents the degree of change of the pixel value in the grayscale image, and the gradient direction represents the direction in which the pixel value changes fastest; according to the gradient direction θ1 of the pixel s1 in the grayscale image, determine the two adjacent pixels s0 and s2 of the pixel in the gradient direction in the grayscale image, and compare the gradient magnitude G1 of the pixel s1 in the grayscale image with the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2; when the gradient magnitude G1 of the pixel s1 in the grayscale image is larger than the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2, then retain the gradient magnitude G1 of the pixel s1 in the grayscale image; when the gradient magnitude G1 of the pixel s1 in the grayscale image is smaller than any one of the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2, then set the gradient magnitude G1 of the pixel s1 in the grayscale image to 0; for example: for the pixel s1, its gradient magnitude G1 = 50, and the gradient direction θ1 = 45°; determine the two adjacent pixels s0 and s2 in this direction according to the gradient direction. Assume that the gradient magnitude G0 of s0 = 30 and the gradient magnitude G2 of s2 = 40. Since G1 > G0 and G1 > G2, then retain G1.

[0075] Set a high threshold T h and a low threshold T l , for detecting edge pixels; when the gradient magnitude of a pixel in the grayscale image is greater than the high threshold T h , then determine that the pixel in the grayscale image is a strong edge pixel; when the gradient magnitude of a pixel in the grayscale image is less than the low threshold T lIf so, it is determined that the pixel in the grayscale image is not an edge pixel; when the gradient amplitude of the pixel in the grayscale image is between the high threshold T h and the low threshold T l , it is determined that the pixel in the grayscale image is a weak edge pixel; starting from the strong edge pixel, check the weak edge pixels adjacent to the strong edge pixel. When the adjacent weak edge pixel is connected to the strong edge pixel, mark the adjacent weak edge pixel as an edge pixel; complete the connection of the edges, obtain the clear flame contour information, and mark the pixel area in the flame contour information as the second verified flame area;

[0076] Perform an intersection operation on the pixels in the first verified flame area and the second verified flame area; when the pixel is a pixel in both the first verified flame area and the second verified flame area at the same time, it is determined as a pixel in the flame area; extract all the pixels that are in both the first verified flame area and the second verified flame area at the same time to form the flame area.

[0077] S5. When it is recognized that there is smoke in the video, extract the smoke area according to the suspected flame or smoke area;

[0078] Specifically, based on the suspected flame or smoke area, perform grayscale threshold segmentation to generate a binary region image and mark it as the first verified smoke area; based on the first verified smoke area, use edge detection and contour detection to obtain the smoke shape within the smoke template range and mark it as the second verified smoke area; based on the suspected flame or smoke area, perform optical flow method calculation to obtain the smoke motion characteristics and mark it as the third verified smoke area; combine the second verified smoke area and the third verified smoke area to analyze and extract the smoke area.

[0079] Further, the grayscale threshold segmentation includes: converting the suspected flame or smoke area from an RGB color image to a grayscale image; setting the grayscale threshold H through sample analysis based on the smoke grayscale characteristics; comparing the grayscale value of each pixel in the grayscale image with the grayscale threshold H; when the grayscale value of the pixel in the grayscale image is less than or equal to the grayscale threshold H, mark the pixel as a pixel in the first verified smoke area; when the grayscale value of the pixel in the grayscale image is greater than the grayscale threshold H, mark it as a non-smoke area; all the pixels marked as the first verified smoke area are combined into the first verified smoke area;

[0080] Further, edge detection and contour detection include: performing edge detection on the first verified smoke area to obtain smoke edge information; based on the smoke edge information, using a contour detection algorithm to detect the contour of the smoke area, calculating the perimeter of the smoke area contour, determining the area enclosed by the smoke area contour, and calculating the aspect ratio through the length and width of the circumscribed rectangle of the smoke area contour; comparing the perimeter, area, and aspect ratio of the smoke area contour with the smoke model; when the perimeter, area, and aspect ratio of the smoke area contour are within the range of the smoke template, the area within the smoke area contour is marked as the second verified smoke area; for example: detecting the contour of the smoke area, calculating that the perimeter of the contour is 200 pixels, the area enclosed by the contour is 1500 pixel², through the length and width of the circumscribed rectangle of the contour, the length is 50 pixels, the width is 30 pixels, and the calculated aspect ratio is 5 / 3; the set perimeter range of the smoke model is [150, 250] pixels, the area range is [1000, 2000] pixel², and the aspect ratio range is [1, 2]. If the parameters of the detected smoke area contour are within the range of the smoke template, the area within the detected smoke area contour is marked as the second verified smoke area; only examples are given here, without any limitations.

[0081] Further, the optical flow method includes: based on the video of the electrical equipment area, extracting two consecutive frames of images containing suspected flame or smoke areas, grayscaling and Gaussian filtering the suspected flame or smoke areas in the images to obtain a denoised grayscale image; based on the denoised grayscale image, using the Shi-Tomasi corner detection algorithm to select feature points; the Shi-Tomasi corner detection algorithm obtains points with significant gray-scale changes in multiple directions as corner points by calculating the gray-scale changes of each pixel point in the denoised grayscale image in different directions; screening the corner points, removing points with overly smooth edges or insignificant gray-scale changes, and retaining a representative set of feature points P = {p1, p2, …, p n}, where p n represents a feature point and n represents the total number of feature points; setting the adjacent pixel points of the feature point P i as the points within the neighborhood of the feature point P i , establishing an optical flow constraint equation for each point within the neighborhood of the feature point P i to obtain the optical flow vector of the feature point P i as (u i , v i ), where (u, v) represents the displacement of the feature point between two consecutive frames of images containing suspected flame or smoke areas; calculating the optical flow vector of each feature point P to obtain an optical flow vector set V = {(u1, v1), (u2, v2), …, (u n , v n)}; Calculate the direction of each optical flow vector, and conduct a statistical analysis on the optical flow directions of all feature points; when the optical flow directions show a diffusing trend, it is determined as a smoke area and marked as the third verified smoke area;

[0082] Combine the second verified smoke area and the third verified smoke area, and analyze the overlapping pixel area of the second verified smoke area and the third verified smoke area. The overlapping pixel area is the smoke area.

[0083] S6. According to the flame area, calculate the position of the fire source in three-dimensional space, and mark the position of the fire source in three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area;

[0084] Specifically, when only one high-definition camera detects the flame area, based on the installation position, viewing angle of the high-definition camera, and the three-dimensional model of the electrical equipment area, calculate the position of the fire source in three-dimensional space; for example: the high-definition camera is installed at the position of (10, 5, 2) (unit: meters, with a corner of the room as the origin to establish a coordinate system), the horizontal viewing angle is 120°, and the vertical viewing angle is 90°; based on the installation position, viewing angle of the high-definition camera, and the three-dimensional model of the electrical equipment area, through the triangulation method, calculate the position of the fire source in three-dimensional space as (15, 8, 3) (unit: meters);

[0085] When multiple high-definition cameras detect the flame area, utilize the viewing angle and position differences of each camera, and conduct a joint estimation of the fire source position through spatial geometric relationships; for example: 3 cameras detect the flame area, located at the positions of (10, 5, 2), (20, 10, 2.5), and (30, 15, 3) (unit: meters) respectively; utilize the viewing angle and position differences of each camera, through the principle of triangulation positioning, combined with the position information of the flame area captured by the three cameras in the image, calculate the position of the fire source in three-dimensional space as (18, 12, 3.2) (unit: meters), and mark it at the corresponding position of the three-dimensional model.

[0086] S7. When the smoke area is extracted but the flame area is not extracted, combine the smoke shapes, flow directions, and convergence points detected by multiple high-definition cameras, analyze the position of the fire source in three-dimensional space, and mark the position of the fire source in three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area;

[0087] Specifically, according to the installation location, viewing angle, and imaging principle of the high-definition camera, a projection relationship from the two-dimensional image coordinates captured by the high-definition camera to the three-dimensional space coordinates is established; based on the smoke area detected by the high-definition camera, according to the projection relationship, the shape, flow direction, and convergence point of the smoke in the two-dimensional image are converted into coordinate information in the three-dimensional space; based on the smoke flow direction and convergence point detected by multiple high-definition cameras, the movement vector and convergence point range of the smoke are drawn in the three-dimensional space; based on the comprehensive analysis of the fire source direction and the position information of the convergence point obtained by backtracking the smoke flow direction, the position of the fire source in the three-dimensional space is estimated, and the position of the fire source in the three-dimensional space is marked at the corresponding position of the three-dimensional model of the electrical equipment area.

[0088] S8. When the fire source is marked in the three-dimensional model of the electrical equipment area, the alarm device is immediately triggered, a three-dimensional model of the electrical equipment area with the fire source position marked is sent to the user's terminal, and an emergency power-off is performed;

[0089] Specifically, for example, when the fire source position is determined, the system sends a three-dimensional model picture containing the fire source position (19, 14, 3.5) to the user through the terminal interface and automatically cuts off the power of the relevant electrical equipment to prevent the fire source from spreading further.

[0090] The present invention provides another technical solution, an electrical equipment operation monitoring system based on flame and smoke video analysis; the system includes: a three-dimensional model module, a flame or smoke detection module, a fire source positioning module, and a fire source alarm module;

[0091] The three-dimensional model module includes a viewing angle test module, a video acquisition module, and a model construction module; the model construction module makes a preliminary evaluation of the electrical equipment area and constructs a three-dimensional model of the electrical equipment area; the viewing angle test module identifies key monitoring positions according to the model construction module and conducts a viewing angle test by installing high-definition cameras to ensure that each key monitoring area is clearly covered by the high-definition cameras; the video acquisition module real-time collects video data of the electrical equipment area through the installed high-definition cameras.

[0092] The flame or smoke detection module includes a flame or smoke extraction module, a flame verification module, and a smoke verification module. The flame or smoke extraction module analyzes the color or brightness change of each pixel using a Gaussian mixture model, detects the difference from the background, and thus determines whether there is a suspected flame or smoke area. The flame verification module extracts the suspected flame or smoke area through color feature analysis and edge detection, and analyzes it in combination with the first verified flame area and the second verified flame area to finally confirm the accurate position of the flame area. The smoke verification module generates a first smoke verification area through gray-scale threshold segmentation, then obtains a second smoke verification area through edge detection and contour detection, and further analyzes the motion characteristics of the smoke using the optical flow method to obtain a third smoke verification area. By combining the second smoke verification area and the third smoke verification area, the accurate position of the smoke area is confirmed.

[0093] The fire source location module includes a flame fire source location module and a smoke fire source location module. The flame fire source location module calculates the fire source position based on a single or multiple cameras. The smoke fire source location module calculates the fire source position based on the shape, flow direction, and convergence point of the smoke.

[0094] The fire source alarm module displays the three-dimensional model marked with the fire source position to the user through the terminal interface and provides functions of real-time monitoring, alarm, and operation feedback.

[0095] For those skilled in the art, it is obvious that the present invention is not limited to the details of the above exemplary embodiments, and can be implemented in other specific forms without departing from the spirit or basic characteristics of the present invention. Therefore, from any point of view, the embodiments should be regarded as exemplary and non-restrictive. The scope of the present invention is defined by the appended claims rather than the above description. Therefore, all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present invention. Any reference signs in the claims should not be regarded as limiting the claims involved.

Claims

1. An electrical equipment operation monitoring method based on flame and smoke video analysis, characterized in that: The method includes: Conduct a preliminary assessment of the layout and spatial position of the electrical equipment area, and construct a 3D model of the electrical equipment area; Based on the 3D model of the electrical equipment area, analyze and identify key monitoring positions, install high-definition cameras, and conduct perspective tests to ensure there are no blind spots in key areas; Based on the installed high-definition cameras, collect videos of the electrical equipment area in real time; and identify whether there is fire or smoke in the video, and mark the suspected fire or smoke area; When it is identified that there is fire or smoke in the video, extract the fire or smoke area according to the suspected fire or smoke area; According to the fire area, calculate the position of the fire source in 3D space, and mark the position of the fire source in 3D space at the corresponding position of the 3D model of the electrical equipment area; When the smoke area is extracted but the fire area is not extracted, analyze the position of the fire source in 3D space by combining the smoke shape, flow direction and convergence point detected by multiple high-definition cameras, and mark the position of the fire source in 3D space at the corresponding position of the 3D model of the electrical equipment area; When the fire source is marked in the 3D model of the electrical equipment area, immediately trigger the alarm device, send the 3D model of the electrical equipment area with the marked fire source position to the user's terminal, and perform emergency power-off.

2. The method for monitoring the operation of electrical equipment based on fire and smoke video analysis according to claim 1, characterized in that: Based on each of the key monitoring positions, install high-definition cameras and conduct perspective tests; The perspective test includes: analyzing the perspective range covered by each high-definition camera, the size of the monitoring area, and the space and layout around the electrical equipment to ensure that the high-definition camera covers all key areas and the pictures of the key areas are clear without blind spots; When the perspective test finds that there are blind spots or unclear pictures in the key areas, adjust the installation angle and position of the high-definition camera or increase the number of high-definition cameras.

3. The system and method for monitoring the operation of electrical equipment based on fire and smoke video analysis according to claim 1, characterized in that: Process the video of the electrical equipment area into frames of images to form video frames; Perform Gaussian filtering denoising on the video frames to remove high-frequency noise and smooth the images; Use the Gaussian mixture model to establish multiple Gaussian distributions for the color or brightness value of each pixel, and based on the multiple Gaussian distributions, perform background modeling on each pixel, and different Gaussian distributions represent different states in the background; When in the background, the pixel value differs from the Gaussian distribution in the modeling by more than the set threshold, the pixel value will be marked as a suspected fire or smoke area.

4. The method for monitoring the operation of an electrical device based on flame and smoke video analysis according to claim 1, characterized in that, When it is identified that there is fire in the video, extracting the fire area according to the suspected fire or smoke area includes: Based on the suspected flame or smoke area, color feature analysis is performed to obtain the color information of the flame and mark it as the first verified flame area; based on the suspected flame or smoke area, edge detection is performed to obtain the contour information of the flame and mark it as the second verified flame area; combining the first verified flame area and the second verified flame area, the flame area is analyzed and extracted.

5. The electrical equipment operation monitoring method based on flame and smoke video analysis according to claim 4, characterized in that: The color feature analysis includes: Converting the suspected flame or smoke area from the RGB color space to the HSV color space; the HSV color space is based on hue, saturation, and value. By setting the ranges of hue, saturation, and value, and comparing the HSC values of each pixel in the suspected flame or smoke area with the ranges of hue, saturation, and value; if the HSV value of a pixel is within the ranges of hue, saturation, and value, then mark this pixel as the first verified flame area; The edge detection includes: Converting the suspected flame or smoke area from an RGB color image to a grayscale image; Performing Gaussian smoothing on the grayscale image. The Gaussian smoothing constructs a Gaussian kernel centered on each pixel in the grayscale image. By performing a convolution operation between the Gaussian kernel and the pixels of the grayscale image, the Gaussian smoothing process of the grayscale image is realized to reduce the influence of noise; Calculating the gradient magnitude and gradient direction of each pixel in the Gaussian-smoothed grayscale image. The gradient magnitude represents the degree of change of the pixel value in the grayscale image, and the gradient direction represents the direction in which the pixel value changes fastest; According to the gradient direction θ1 of pixel s1 in the grayscale image, determine the two adjacent pixels s0 and s2 of pixel s1 in the gradient direction in the grayscale image, and compare the gradient magnitude G1 of pixel s1 in the grayscale image with the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2; when the gradient magnitude G1 of pixel s1 in the grayscale image is larger than the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2, then retain the gradient magnitude G1 of pixel s1 in the grayscale image; when the gradient magnitude G1 of pixel s1 in the grayscale image is smaller than any one of the gradient magnitudes G0 and G2 of the two adjacent pixels s0 and s2, then set the gradient magnitude G1 of pixel s1 in the grayscale image to 0; Set a high threshold T h and a low threshold T l , for detecting edge pixels; when the gradient magnitude of a pixel in the grayscale image is greater than the high threshold T h , then it is determined that the pixel in the grayscale image is a strong edge pixel; when the gradient magnitude of a pixel in the grayscale image is less than the low threshold T l , then it is determined that the pixel in the grayscale image is not an edge pixel; when the gradient magnitude of a pixel in the grayscale image is between the high threshold T h and the low threshold T l , then it is determined that the pixel in the grayscale image is a weak edge pixel; starting from the strong edge pixel, check the weak edge pixels adjacent to the strong edge pixel. When the adjacent weak edge pixel is connected to the strong edge pixel, mark the adjacent weak edge pixel as an edge pixel; complete the connection of the edges, obtain clear flame contour information, and mark the pixel area in the flame contour information as the second verified flame area; Performing an intersection operation on the pixels of the first verified flame area and the second verified flame area; when a pixel is simultaneously a pixel within the first verified flame area and the second verified flame area, then it is determined as a pixel of the flame area; extract all pixels that are simultaneously within the first verified flame area and the second verified flame area to form the flame area.

6. The method for monitoring the operation of an electrical device based on flame and smoke video analysis according to claim 1, characterized in that, When it is recognized that there is smoke in the video, according to the suspected flame or smoke area, the smoke area is extracted, including: Based on the suspected flame or smoke area, perform grayscale threshold segmentation to generate a binary region image, which is marked as the first verified smoke area; based on the first verified smoke area, use edge detection and contour detection to obtain the smoke shape within the smoke template range, which is marked as the second verified smoke area; based on the suspected flame or smoke area, perform optical flow method calculation to obtain the smoke motion characteristics, which is marked as the third verified smoke area; combine the second verified smoke area and the third verified smoke area to analyze and extract the smoke area.

7. The method for monitoring the operation of an electrical device based on flame and smoke video analysis according to claim 6, characterized in that: The grayscale threshold segmentation includes: Convert the suspected flame or smoke area from an RGB color image to a grayscale image; Based on the grayscale characteristics of smoke, set the grayscale threshold H through sample analysis; compare the grayscale value of each pixel in the grayscale image with the grayscale threshold H; when the grayscale value of the pixel in the grayscale image is less than or equal to the grayscale threshold H, mark the pixel as a pixel in the first verified smoke area; when the grayscale value of the pixel in the grayscale image is greater than the grayscale threshold H, mark it as a non-smoke area; all pixels marked as the first verified smoke area are combined to form the first verified smoke area; The edge detection and contour detection include: Perform edge detection on the first verified smoke area to obtain smoke edge information; Based on the smoke edge information, use a contour detection algorithm to detect the smoke area contour, calculate the perimeter of the smoke area contour, determine the area enclosed by the smoke area contour, and calculate the aspect ratio through the length and width of the circumscribed rectangle of the smoke area contour; Compare the perimeter, area, and aspect ratio of the smoke area contour with the smoke model; when the perimeter, area, and aspect ratio of the smoke area contour are within the smoke template range, mark the area within the smoke area contour as the second verified smoke area; The optical flow method includes: Based on the video of the electrical device area, extract two consecutive frames of images containing the suspected flame or smoke area, perform grayscale conversion and Gaussian filtering on the suspected flame or smoke area in the images to obtain a denoised grayscale image; Based on the denoised grayscale image, feature points are selected using the Shi-Tomasi corner detection algorithm; the Shi-Tomasi corner detection algorithm obtains points with significant gray-scale changes in multiple directions as corner points by calculating the gray-scale changes of each pixel point in the denoised grayscale image in different directions; the corner points are screened to remove points with overly smooth edges or insignificant gray-scale changes, and a representative set of feature points P = {p1, p2, …, p n} is retained, where p n represents a feature point and n represents the total number of feature points; the pixel points adjacent to the feature points P i are set as the points within the neighborhood of the feature points P i ; for each point within the neighborhood of the feature points P i , an optical flow constraint equation is established to obtain the optical flow vector of the feature points P i as (u i , v i ), where (u, v) represents the displacement of the feature point between the two consecutive frames of images containing the suspected flame or smoke region; the optical flow vectors of each feature point P are calculated to obtain an optical flow vector set V = {(u1, v1), (u2, v2), …, (u n , v n )}; the direction of each optical flow vector is calculated, and a statistical analysis is performed on the optical flow directions of all feature points; when the optical flow directions show a diffusion trend, it is determined as a smoke region and marked as the third verified smoke region; Combine the second verified smoke area and the third verified smoke area, and analyze the overlapping pixel area of the second verified smoke area and the third verified smoke area, and the overlapping pixel area is the smoke area.

8. The method for monitoring the operation of an electrical device based on flame and smoke video analysis according to claim 1, wherein: The calculation of the position of the fire source in three-dimensional space according to the flame area and the annotation of the position of the fire source in three-dimensional space to the corresponding position of the three-dimensional model of the electrical device area include: When only one high-definition camera detects the flame area, calculate the position of the fire source in three-dimensional space based on the installation position, viewing angle of the high-definition camera, and the three-dimensional model of the electrical device area; When multiple high-definition cameras detect the flame area, use the viewing angle and position differences of each camera to jointly estimate the fire source position through spatial geometric relationships; Mark the position of the fire source in the three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area.

9. The method for monitoring the operation of an electrical equipment based on flame and smoke video analysis according to claim 1, characterized in that: Based on the installation position, viewing angle and imaging principle of the high-definition camera, establish a projection relationship from the two-dimensional image coordinates captured by the high-definition camera to the three-dimensional space coordinates; Based on the smoke area detected by the high-definition camera, according to the projection relationship, convert the smoke shape, flow direction and convergence point of the two-dimensional image into coordinate information in the three-dimensional space; Based on the smoke flow direction and convergence point detected by the multiple high-definition cameras, draw the motion vector and convergence point range of the smoke in the three-dimensional space; Based on the comprehensive analysis of the fire source direction inferred from the smoke flow direction and the position information of the convergence point, estimate the position of the fire source in the three-dimensional space, and mark the position of the fire source in the three-dimensional space at the corresponding position of the three-dimensional model of the electrical equipment area.

10. An electrical equipment operation monitoring system based on flame and smoke video analysis, which is applied to the electrical equipment operation monitoring method based on flame and smoke video analysis according to any one of claims 1-9, and is characterized in that, The system includes: a three-dimensional model module, a flame or smoke detection module, a fire source positioning module and a fire source alarm module; The three-dimensional model module includes a viewing angle test module, a video acquisition module and a model construction module; the model construction module conducts a preliminary evaluation of the electrical equipment area and constructs a three-dimensional model of the electrical equipment area; the viewing angle test module, according to the model construction module, identifies key monitoring positions and conducts a viewing angle test by installing high-definition cameras to ensure that each key monitoring area is clearly covered by the high-definition cameras; the video acquisition module, through the installed high-definition cameras, real-time collects video data of the electrical equipment area; The flame or smoke detection module includes a flame or smoke extraction module, a flame verification module and a smoke verification module; the flame or smoke extraction module analyzes the color or brightness change of each pixel using the Gaussian mixture model, detects the difference from the background, and thus judges whether there is a suspected flame or smoke area; the flame verification module extracts the suspected flame or smoke area through color feature analysis and edge detection, and analyzes it in combination with the first verified flame area and the second verified flame area to finally confirm the accurate position of the flame area; the smoke verification module generates a first smoke verification area through gray-scale threshold segmentation, then obtains a second smoke verification area through edge detection and contour detection, and further analyzes the motion characteristics of the smoke using the optical flow method to obtain a third smoke verification area, and combines the second smoke verification area and the third smoke verification area to confirm the accurate position of the smoke area; The fire source positioning module includes a flame fire source positioning module and a smoke fire source positioning module; the flame fire source positioning module calculates the fire source position based on a single or multiple cameras; the smoke fire source positioning module calculates the fire source position based on the smoke shape, flow direction and convergence point; The fire source alarm module displays the three-dimensional model marked with the fire source position to the user through the terminal interface and provides real-time monitoring, alarm and operation feedback functions.

Images