Fan fire early warning method and device based on multi-source video data fusion, equipment and storage medium

Abstract

The invention discloses a fan fire early warning method, device and equipment based on multi-source video data fusion and a storage medium, and relates to the technical field of image data fusion, and the method comprises the steps: determining a fire early warning position according to video data when the video data of a wind turbine generator has a fire early warning feature, determining a plurality of target monitoring devices according to the fire early warning position, extracting video data of the plurality of target monitoring devices, taking the fire early warning position as a center point, performing video fusion on the video data and the video data of the plurality of target monitoring devices to obtain fused video data, and sending the fused video data to a server; and performing fire early warning detection on the fused video data to obtain a fire detection result, and when the fire detection result is a preset fire detection result, outputting a fire position according to the fire detection result, thereby realizing fire early warning detection on the wind turbine generator from multiple angles, and ensuring the accuracy of fire early warning detection.

Description

Technical Field

[0001] The present application relates to the technical field of image data fusion, and in particular to a fan fire warning method, device, equipment and storage medium for multi-source video data fusion. Background Art

[0002] Wind turbine fire is a prominent problem in wind power accidents, and the probability of fire in large wind turbines is higher. Lightning strikes and high-temperature operation are key factors that cause electrical appliances and lines to catch fire. At present, in wind power systems, smoke detection is performed through smoke alarms to determine whether there is a fire in the wind turbine. However, this method is limited by the smoke alarm's judgment of smoke concentration, and when smoke is generated, a fire source has already appeared. At this time, the warning has already caused the loss of wind turbine equipment, which will seriously affect the normal operation of the wind turbine. Therefore, the current fire warning method cannot meet the production needs of wind turbines and cannot quickly and accurately locate potential fire sources.

[0003] The above contents are only used to assist in understanding the technical solution of the present application and do not constitute an admission that the above contents are prior art. Summary of the invention

[0004] The main purpose of this application is to provide a fan fire warning method, device, equipment and storage medium with multi-source video data fusion, aiming to solve the technical problem that the existing technology cannot intervene in the occurrence of fire in advance.

[0005] To achieve the above purpose, the present application proposes a fan fire early warning method based on multi-source video data fusion, and the fan fire early warning method based on multi-source video data fusion includes:

[0006] When the video data of the wind turbine generator set has fire warning features, determining the fire warning location according to the video data;

[0007] Determine a plurality of target monitoring devices according to the fire warning position, and extract video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position;

[0008] Taking the fire warning position as the center point, fusing the video data with the video data of a plurality of target monitoring devices to obtain fused video data;

[0009] Perform fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, output the fire location according to the fire detection result.

[0010] In one embodiment, when the video data of the wind turbine generator set has a fire warning feature, the step of determining the fire warning location according to the video data includes:

[0011] Performing feature recognition on the video data of the wind turbine generator set to determine a temperature feature graph of the wind turbine generator set in the video data;

[0012] Determine a temperature gradient of a temperature characteristic diagram of the wind turbine generator set, and when the temperature gradient is greater than a preset temperature gradient, determine a temperature change vector according to the temperature gradient;

[0013] The temperature change vectors are aggregated to obtain a vector center, and the vector center is determined as a fire warning position.

[0014] In one embodiment, the step of determining a plurality of target monitoring devices according to the fire warning position and extracting video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position, comprises:

[0015] Determining the spatial position coordinates of the fire warning position according to the fire warning position;

[0016] Obtaining a monitoring location map of a wind turbine monitoring device and a monitoring area of ​​each wind turbine monitoring device;

[0017] Determine the wind turbine group monitoring device to be selected according to the spatial position coordinates and the monitoring site map of the wind turbine group monitoring device;

[0018] Determine a plurality of target wind turbine monitoring devices according to the monitoring area of ​​the selected wind turbine monitoring device and the spatial position coordinates;

[0019] The video data of the plurality of target wind turbine group monitoring devices are extracted.

[0020] In one embodiment, the step of taking the fire warning position as the center point and fusing the video data with the video data of a plurality of target monitoring devices to obtain fused video data comprises:

[0021] Determine the fire warning position, and determine the fire warning position as the center point;

[0022] respectively determining the fire warning positions in the video data of the plurality of target monitoring devices, and determining the fire warning positions in the video data of the target monitoring devices as fusion reference points;

[0023] Determine the edge feature of the center point, and determine the edge feature of the center point as a fusion base;

[0024] Determine the fusion point edge feature of the fusion reference point, and make the fusion reference point coincide with the center point;

[0025] Based on the fusion base, the fusion point edge feature and the center point edge feature are fused to obtain fused video data.

[0026] In one embodiment, the step of fusing the fusion point edge feature with the center point edge feature to obtain fused video data includes:

[0027] Determine similar features between the fusion point edge feature and the center point edge feature, wherein the similar features exist simultaneously in the fusion point edge feature and the center point edge feature;

[0028] Performing positive fusion of the fusion reference point and the center point based on the video frame to obtain a positive fusion video;

[0029] Determine the feature offset of the fusion point edge feature and the center point edge feature in the orthogonal fusion video, where the feature offset is feature offset data of the fusion point edge feature relative to the center point edge feature;

[0030] The orthogonal fused video is offset calibrated according to the characteristic offset to obtain fused video data.

[0031] In one embodiment, the step of performing offset calibration on the orthogonal fused video according to the characteristic offset to obtain fused video data comprises:

[0032] Determine the offset distance and the offset angle of the feature offset;

[0033] Using the offset angle as a first calibration parameter, the edge feature of the fusion point is rotated and offset;

[0034] After the rotational offset is completed, the offset direction and the offset distance can be updated based on the edge features of the fusion point after the rotational offset and the edge features of the center point, and an offset vector can be obtained according to the offset direction and the offset distance;

[0035] The edge feature of the fusion point is adjusted by using the offset vector so that the edge feature of the fusion point coincides with the edge feature of the center point, thereby obtaining fused video data.

[0036] In one embodiment, the step of performing fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, outputting the fire location according to the fire detection result comprises:

[0037] Performing fire warning detection on the fused video data, determining infrared detection data in the fused video data, and determining an infrared imaging image corresponding to the fused video data;

[0038] Determining abnormally high temperature points based on temperature thresholds of components of the wind turbine generator set and the infrared imaging image;

[0039] Outputting a fire detection result according to the abnormally high temperature point;

[0040] When the fire detection result is a preset fire detection result, the fire location is output according to the fire detection result.

[0041] In addition, to achieve the above purpose, the present application also proposes a fan fire warning device with multi-source video data fusion, and the fan fire warning device with multi-source video data fusion includes:

[0042] A point warning module, used to determine the fire warning position according to the video data when there are fire warning features in the video data of the wind turbine generator set;

[0043] a video extraction module, configured to determine a plurality of target monitoring devices according to the fire warning position, and extract video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position;

[0044] A video fusion module, used to fuse the video data with the video data of a plurality of target monitoring devices with the fire warning position as the center point to obtain fused video data;

[0045] The fire warning module is used to perform fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, output the fire location according to the fire detection result.

[0046] In addition, to achieve the above-mentioned purpose, the present application also proposes a fan fire warning device with multi-source video data fusion, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the fan fire warning method with multi-source video data fusion as described above.

[0047] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by the processor, the steps of the fan fire warning method of multi-source video data fusion as described above are implemented.

[0048] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, it implements the steps of the fan fire warning method of multi-source video data fusion as described above.

[0049] One or more technical solutions proposed in the present application have at least the following technical effects: when there are fire warning features in the video data of a wind turbine generator set, a fire warning position is determined according to the video data, a plurality of target monitoring devices are determined according to the fire warning position, and the video data of the plurality of target monitoring devices are extracted, and the video data of the plurality of target monitoring devices are fused with the video data of the plurality of target monitoring devices with the fire warning position as the center point to obtain fused video data, and fire warning detection is performed on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, the fire position is output according to the fire detection result, so as to realize fire warning detection of the wind turbine generator set from multiple angles and ensure the accuracy of fire warning detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0051] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0052] Figure 1 A flow chart of a fan fire warning method embodiment 1 of the multi-source video data fusion of the present application is provided;

[0053] Figure 2 A schematic diagram of infrared imaging of a wind turbine set provided in accordance with an embodiment of a wind turbine fire warning method using multi-source video data fusion in this application;

[0054] Figure 3 A schematic diagram of the center point position provided in an embodiment of a fan fire warning method for multi-source video data fusion of the present application;

[0055] Figure 4 A schematic diagram for determining the offset distance and offset angle provided in an embodiment of a fan fire warning method for multi-source video data fusion of the present application;

[0056] Figure 5 This is a schematic diagram of the module structure of a fan fire warning device with multi-source video data fusion according to an embodiment of the present application;

[0057] Figure 6 Schematic diagram of the equipment structure of the hardware operating environment involved in the fan fire warning method of multi-source video data fusion in the embodiment of the present application.

[0058] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0059] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0060] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0061] The main solution of the embodiment of the present application is: when there are fire warning features in the video data of the wind turbine set, determine the fire warning position according to the video data; determine multiple target monitoring devices according to the fire warning position, and extract the video data of the multiple target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position; with the fire warning position as the center point, perform video fusion on the video data of the multiple target monitoring devices to obtain fused video data; perform fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, output the fire position according to the fire detection result.

[0062] In this embodiment, for ease of description, the following description is made by taking a fan fire warning device that recognizes multi-source video data fusion as the execution subject.

[0063] Since the existing technology usually installs smoke alarms at fire-prone locations when conducting fire warnings, the smoke alarms can be triggered to sound an alarm when a fire occurs and smoke is generated. However, the process from fire to warning is an intervention measure to prevent the fire from further expanding after the fire has already occurred and a combustion reaction has occurred. In other words, the current fire warning is actually only issued after the fire has occurred. In addition, in order to avoid false alarms, the smoke alarm will also set a threshold for the smoke concentration and will only sound an alarm when the smoke concentration exceeds the threshold. Therefore, a situation may occur where a fire occurs but no warning can be issued. Therefore, it can be seen that the current fire warning cannot provide early intervention before the fire occurs.

[0064] The present application provides a solution that can lock and output dangerous locations where fire may occur before the fire actually occurs, and execute relevant fire warning measures to reduce the probability of fire and improve the operating safety of wind turbines.

[0065] It can be seen from the above embodiments that the present application determines the fire warning position according to the video data when there are fire warning features in the video data of the wind turbine generator set, determines multiple target monitoring devices according to the fire warning position, and extracts the video data of the multiple target monitoring devices, takes the fire warning position as the center point, performs video fusion on the video data of the multiple target monitoring devices to obtain fused video data, performs fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, outputs the fire position according to the fire detection result, thereby realizing fire warning detection of the wind turbine generator set from multiple angles and ensuring the accuracy of the fire warning detection.

[0066] It should be noted that the execution subject of this embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, a fan fire warning device with multi-source video data fusion, etc. The fan fire warning device with multi-source video data fusion is taken as an example to illustrate this embodiment and the following embodiments.

[0067] Based on this, the embodiment of the present application provides a fan fire warning method based on multi-source video data fusion, referring to Figure 1 , Figure 1 This is a flow chart of the first embodiment of the fan fire warning method based on multi-source video data fusion of the present application.

[0068] In this embodiment, the fan fire warning method based on multi-source video data fusion includes steps S10 to S40:

[0069] Step S10: when the video data of the wind turbine generator set has fire warning features, determine the fire warning position according to the video data.

[0070] It should be noted that a wind turbine is a device that converts wind energy into mechanical energy and then converts it into electrical energy through a generator. Various factors may cause fires during the operation of a wind turbine, especially mechanical or electrical failures. According to the principle of fire occurrence, before a fire occurs, the temperature at the ignition point will rise abnormally. Therefore, local temperature anomalies can be used as a benchmark as a fire warning feature, and the point of local temperature anomaly is the fire warning position, which is used to indicate that a fire may occur at the current location.

[0071] In a specific implementation, for the fire warning of wind turbines, in order to respond to the occurrence of fire in wind turbines, video monitoring is performed at multiple angles on various areas of the wind turbines. In order to determine the accuracy of the data, two or more operable monitoring devices can be provided at any position of the wind turbine for video monitoring. The video monitoring can include the temperature information of each position in the wind turbine, and the temperature characteristics of the wind turbine are output in the form of infrared imaging images, where the fire warning characteristics are local high temperature points. When the temperature characteristics in the video data of the wind turbine are detected and local high temperature points are found, the determined local high temperature points can be used as fire warning positions where there is a fire risk, and the current fire warning position is output.

[0072] In a feasible implementation manner, when the video data of the wind turbine generator set has fire warning features, the step of determining the fire warning location according to the video data includes:

[0073] Performing feature recognition on the video data of the wind turbine generator set to determine a temperature feature graph of the wind turbine generator set in the video data;

[0074] Determine a temperature gradient of a temperature characteristic diagram of the wind turbine generator set, and when the temperature gradient is greater than a preset temperature gradient, determine a temperature change vector according to the temperature gradient;

[0075] The temperature change vectors are aggregated to obtain a vector center, and the vector center is determined as a fire warning position.

[0076] It should be noted that the temperature characteristic map refers to the image captured by infrared thermal imaging technology, which can reflect the temperature distribution on the surface of an object. In these images, different temperatures represent different temperatures. Usually, blue represents low temperature and red represents high temperature. The temperature gradient refers to the rate of temperature change of the surface temperature of an object. The larger the temperature gradient, the faster the current temperature change rate. It also reflects that there is a large gap between the current temperature and the temperature at the adjacent position. From another perspective, it means that the current position is a local high temperature point. Accordingly, in order to describe the degree of temperature change, the temperature gradient is converted into a temperature change vector. While describing the temperature change amount, it also describes the direction of temperature change. At the same time, under the joint action of multiple temperature change vectors, the center of the vector can be determined, where the center of the vector and the fire warning position are basically in the same area, and the center of the vector is consistent with the fire warning position.

[0077] In a specific implementation, before performing feature recognition on the video data of a wind turbine, it is also necessary to perform preprocessing operations on the acquired video data of the wind turbine, including denoising, contrast enhancement, etc., to improve the accuracy of subsequent feature recognition. After preprocessing the video data, it is possible to extract infrared imaging data from the video data, output a temperature characteristic map, and perform edge detection and region growing on the video data. The wind turbine or the components of the wind turbine are separated from the background from the video data, and independent wind turbines or the components of the wind turbine are extracted and fused with the temperature characteristic map to obtain a temperature characteristic map of the wind turbine. When determining the temperature characteristic map of a wind turbine, such as Figure 2 As shown, Figure 2 It is a schematic diagram of infrared imaging of a wind turbine. When obtaining a temperature characteristic diagram, different color partitions can be determined according to the color partitions of infrared imaging corresponding to the temperature conditions, and the same color partition is determined as a temperature interval. There is a temperature difference between different temperature intervals, and the temperature difference determines the direction of the vector. In this embodiment, the direction of the vector is from the high temperature interval to the low temperature interval. Therefore, the temperature gradient of the temperature is determined according to the temperature characteristic diagram. The temperature gradient is the degree of temperature change. The degree of temperature change is positively correlated with the temperature distribution. When the temperature suddenly changes, the temperature gradient is large. Therefore, when the temperature gradient is detected, when the temperature gradient is greater than the preset temperature gradient, the temperature change vector can be determined according to the temperature gradient. The temperature vector is aggregated, and an aggregated area can be formed according to the tail of the temperature vector. The aggregated area is the center of the vector. At this time, the area corresponding to the center of the vector is an area with a higher temperature, and a local high temperature point is formed compared with the adjacent area. Therefore, the center of the vector can be determined as the fire warning position.

[0078] Step S20, determining a plurality of target monitoring devices according to the fire warning position, and extracting video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position.

[0079] It should be noted that the monitoring equipment is used for video monitoring of wind turbines and can obtain monitoring videos of wind turbines, while the target monitoring equipment is a monitoring equipment among many monitoring equipment that can monitor the fire warning location.

[0080] In a specific implementation, when the fire warning position is determined, the target monitoring device that can monitor the current fire warning position can be determined according to the current fire warning position, and the monitoring video recorded by the corresponding target monitoring device can be extracted. When determining the target monitoring device, the spatial position coordinates of the fire warning position can be determined according to the fire warning position; the monitoring site map of the wind turbine monitoring device and the monitoring area of ​​each of the wind turbine monitoring devices are obtained; the wind turbine monitoring device to be selected is determined according to the spatial position coordinates and the monitoring site map of the wind turbine monitoring device; multiple target wind turbine monitoring devices are determined according to the monitoring areas of the selected wind turbine monitoring devices and the spatial position coordinates; and the video data of multiple target wind turbine monitoring devices are extracted.

[0081] When arranging monitoring equipment, the environment in which the wind turbine is located and the wind turbine itself can be three-dimensionally modeled to determine a three-dimensional spatial coordinate system. The spatial position coordinates in the three-dimensional spatial coordinate system can point to a unique position in space. When arranging monitoring equipment, the monitoring range of the monitoring equipment and its own position can be recorded accordingly to obtain a monitoring point map of the wind turbine monitoring equipment. Therefore, when the current fire warning position is detected, the spatial position coordinates of the fire position can be determined, and at the same time, the monitoring point map of the wind turbine monitoring equipment and the monitoring area of ​​each wind turbine monitoring equipment can be obtained. By aligning the monitoring area with the fire warning position, the monitoring area containing the fire warning position can be determined as the target monitoring area, the corresponding target monitoring equipment can be locked, and the video data of the corresponding target monitoring equipment can be extracted.

[0082] Step S30, taking the fire warning position as the center point, performing video fusion on the video data with the video data of the plurality of target monitoring devices to obtain fused video data.

[0083] It should be noted that the center point refers to the reference point in the video fusion process, which is used to indicate that the fire warning position in the image for determining the fire warning position is the main reference point. Setting it as the center point means that in the subsequent video fusion process, the center point needs to be used as a reference for video fusion, and the data after fusing multiple video data is the fused video data.

[0084] In the specific implementation, the fire warning position is used as the center point of video fusion, and the center point is used as the reference point of video fusion. First, in the process of video fusion, the video data of multiple target monitoring devices can be time-calibrated with the timestamp data of the video data, and the time axis of each video data can be unified to ensure that the video at the same time is fused, avoiding monitoring errors caused by video data asynchronization due to video delay. After the time information is unified, a suitable video fusion algorithm can be selected, such as splicing video images of different cameras into a panoramic image, or using image blending technology to smoothly transition between different video images for fusion, and optimizing the fused video, and optimizing the fused video, including eliminating splicing seams, adjusting color consistency, enhancing details, etc., to improve the naturalness and usability of the fused video, and obtain fused video data.

[0085] In a feasible implementation manner, the step of taking the fire warning position as the center point and fusing the video data with the video data of a plurality of target monitoring devices to obtain fused video data comprises:

[0086] Determine the fire warning position, and determine the fire warning position as the center point;

[0087] respectively determining the fire warning positions in the video data of the plurality of target monitoring devices, and determining the fire warning positions in the video data of the target monitoring devices as fusion reference points;

[0088] Determine the edge feature of the center point, and determine the edge feature of the center point as a fusion base;

[0089] Determine the fusion point edge feature of the fusion reference point, and make the fusion reference point coincide with the center point;

[0090] Based on the fusion base, the fusion point edge feature and the center point edge feature are fused to obtain fused video data.

[0091] It should be noted that the fusion reference point is the fire warning position in the target monitoring device, the fusion base refers to the reference point in the video fusion process, the center point edge features and the fusion point edge features are the object features near the center point and the object features near the fusion reference point, respectively.

[0092] In a specific implementation, firstly, the image where the fire warning position is found can be determined, and the fire warning position is determined, and the position of the fire warning position in the image is determined as the center point, and the reference Figure 3 , Figure 3It should be noted that the center point is not the center point of the monitoring video, but the fusion center in the subsequent fusion process. Figure 3 As shown, it can be the edge area of ​​the image. At the same time, the fire warning position in the video data of other target monitoring devices can be determined respectively. Due to the angle problem of the monitoring device, the video data of the target monitoring device and the video data of the fire warning position are angle-deviated. The fire warning position in any video data is the same, so the fire warning position is used as a reference point to fuse multiple video data.

[0093] During the fusion process, the edge features of the center point can be determined, and the edge degree of the center point can be determined according to the system preset, can be extended according to fixed pixel points, or can be determined according to the outline of the recognized object, and this embodiment is not limited. At this time, when the edge features of the center point are determined, the edge features of the middle points can be used as reference features to guide the fusion of the edge features of the fusion points. At this time, the fusion reference point and the center point can be overlapped first, and then the fusion edge point features and the center point edge features can be feature fused, which may involve the rotation and scaling of the video to obtain fused video data.

[0094] In a feasible implementation manner, the step of fusing the fusion point edge feature with the center point edge feature to obtain fused video data includes:

[0095] Determine similar features between the fusion point edge feature and the center point edge feature, wherein the similar features exist simultaneously in the fusion point edge feature and the center point edge feature;

[0096] Performing positive fusion of the fusion reference point and the center point based on the video frame to obtain a positive fusion video;

[0097] Determine the feature offset of the fusion point edge feature and the center point edge feature in the orthogonal fusion video, where the feature offset is feature offset data of the fusion point edge feature relative to the center point edge feature;

[0098] The orthogonal fused video is offset calibrated according to the characteristic offset to obtain fused video data.

[0099] In the specific implementation, the edge features of the fusion point and the center point are first analyzed for similarity to identify the common edge features of the two. The edge features of the fusion point and the center point are detected according to the Canny algorithm to obtain the edge features of the fusion point and the center point respectively, and then the edge features of the fusion point and the center point are vectorized to facilitate similarity calculation. When determining the edge features of the fusion point and the center point, the formula can be used:

[0100]

[0101] Among them, μ x and μ y are the average brightness of the edge features of the fusion point and the edge features of the center point, is the variance, σ xy is the covariance, C 1 and C 2 is a constant used to maintain stability.

[0102] Then, based on the video screen, the fusion reference point and the center point are accurately fused to generate an orthogonal fusion video. Next, the feature offset is calculated for the edge features of the fusion point and the edge features of the center point in the orthogonal fusion video, the feature points in both the edge features of the fusion point and the edge features of the center point are determined, and their feature offset is calculated. The specific calculation formula is:

[0103]

[0104] Among them, P 1 =(x 1 ,y 1 ) and P 2 =(x 2 ,y 2 ) are the coordinates of the feature points.

[0105] The offset reflects the displacement data of the edge feature of the fusion point relative to the edge feature of the center point. Based on the calculated feature offset, the offset calibration is performed on the orthogonal fusion video to finally obtain the calibrated fusion video data.

[0106] In a feasible implementation manner, the step of performing offset calibration on the orthogonal fused video according to the feature offset to obtain fused video data includes:

[0107] Determine the offset distance and the offset angle of the feature offset;

[0108] Using the offset angle as a first calibration parameter, the edge feature of the fusion point is rotated and offset;

[0109] After the rotational offset is completed, the offset direction and the offset distance can be updated based on the edge features of the fusion point after the rotational offset and the edge features of the center point, and an offset vector can be obtained according to the offset direction and the offset distance;

[0110] The edge feature of the fusion point is adjusted by using the offset vector so that the edge feature of the fusion point coincides with the edge feature of the center point, thereby obtaining fused video data.

[0111] In the specific implementation, when the feature offset of the feature point in the center point edge feature and the fusion point edge feature is used, the offset distance and offset angle in the feature offset can be determined. The offset distance refers to the distance deviation and deviation angle of the feature point in the center point edge feature and the fusion point edge feature. Since the center point and the fusion point have overlapped during fusion, they can radiate outward from the center point. Therefore, when determining the offset distance and offset angle, refer to Figure 4 , Figure 4 The schematic diagram for determining the offset distance and the offset angle. When determining the offset angle, the clockwise direction can be taken as a positive angle, and the counterclockwise direction can be taken as a positive angle. For ease of description, the offset angle α can be set to (-180°, 180°). 1 ~L 4 They respectively represent the distance between the feature point and the center point. The circle and diamond represent two pairs of feature points. For example, the circular feature point is used as an example. First, the center point edge feature point and the fusion point edge feature point are determined. The center point edge feature point is used as the starting point to determine the relative position of the fusion point edge feature point and the relative position of the center point edge feature point. In this figure, the fusion point edge feature point is located counterclockwise from the center point edge feature point, so the offset angle is a negative angle. At the same time, the distance between the center point edge feature point and the center point can be determined as L. 1 , the distance between the edge feature point of the fusion point and the center point is L 2 , when determining the offset distance, the edge feature point of the center point is also used as the starting point, the side close to the center point is the negative direction, and the side far from the center point is the positive direction. Therefore, when determining the offset distance L, it can be based on L = L 1 -L 2 The offset distance is calculated. The offset angle is used as the first calibration parameter. -α can be used as the calibration parameter to rotate and offset the edge feature of the fusion point. After the offset is completed, the offset vector can be obtained according to the offset distance and the offset angle after the rotation offset. The fusion edge feature is adjusted with the offset vector so that the fusion edge feature coincides with the edge feature of the center point to obtain fusion video data.

[0112] Step S40, performing fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, outputting the fire location according to the fire detection result.

[0113] In the specific implementation, the fused video data is subjected to fire warning detection, the infrared detection data in the fused video data is determined, and the corresponding infrared imaging image in the fused video data is determined. The temperature thresholds of each component in the wind turbine and the abnormally high temperature points in the corresponding infrared imaging image are determined based on the infrared imaging image. It should be noted that when the temperature in a certain area is high but not extremely high, it does not mean that the current temperature is normal. When determining whether it is an abnormally high temperature point, it is necessary to determine the component temperature threshold at the current position based on the position and characteristics of the current component, and judge the current temperature with the component temperature threshold to determine the abnormally high temperature point in the image. After obtaining the abnormally high temperature point, the fire detection result can be output. When judging the fire detection result, the fire detection result can be compared with the preset fire detection result, indicating that there is a possible location of the fire at present. Therefore, the fire location can be determined based on the fire detection result, and the location where the fire may occur can be output, so that the staff can respond in advance and perform shutdown, maintenance and other operations.

[0114] The present embodiment provides a wind turbine fire warning method by fusing multi-source video data. When there is a fire warning feature in the video data of the wind turbine set, the fire warning position is determined according to the video data, a plurality of target monitoring devices are determined according to the fire warning position, and the video data of the plurality of target monitoring devices are extracted. The video data is fused with the video data of the plurality of target monitoring devices with the fire warning position as the center point to obtain fused video data, a fire warning detection is performed on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, the fire position is output according to the fire detection result, thereby realizing fire warning detection of the wind turbine set from multiple angles and ensuring the accuracy of the fire warning detection.

[0115] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the fan fire warning method based on multi-source video data fusion of the present application. More simple transformations based on this technical concept are all within the protection scope of the present application.

[0116] This application also provides a fan fire warning device with multi-source video data fusion, please refer to Figure 5 The fan fire warning device with multi-source video data fusion includes:

[0117] A point warning module 10 is used to determine a fire warning position according to the video data when there are fire warning features in the video data of the wind turbine generator set;

[0118] A video extraction module 20, configured to determine a plurality of target monitoring devices according to the fire warning position, and extract video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position;

[0119] A video fusion module 30 is used to fuse the video data with the video data of multiple target monitoring devices with the fire warning position as the center point to obtain fused video data;

[0120] The fire warning module 40 is used to perform fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, output the fire location according to the fire detection result.

[0121] In one embodiment, the point warning module 10 is also used to perform feature recognition on the video data of the wind turbine group, determine the temperature characteristic graph of the wind turbine group in the video data; determine the temperature gradient of the temperature characteristic graph of the wind turbine group, and when the temperature gradient is greater than a preset temperature gradient, determine the temperature change vector according to the temperature gradient; aggregate the temperature change vector to obtain the vector center, and determine the vector center as the fire warning position.

[0122] In one embodiment, the video extraction module 20 is also used to determine the spatial position coordinates of the fire warning position based on the fire warning position; obtain the monitoring site map of the wind turbine monitoring device and the monitoring area of ​​each of the wind turbine monitoring devices; determine the wind turbine monitoring device to be selected based on the spatial position coordinates and the monitoring site map of the wind turbine monitoring device; determine multiple target wind turbine monitoring devices based on the monitoring areas of the selected wind turbine monitoring devices and the spatial position coordinates; and extract video data of multiple target wind turbine monitoring devices.

[0123] In one embodiment, the video fusion module 30 is further used to determine the fire warning position, and determine the fire warning position as the center point; respectively determine the fire warning positions in the video data of multiple target monitoring devices, and determine the fire warning positions in the video data of the target monitoring devices as the fusion reference point; determine the edge features of the center point, and determine the edge features of the center point as the fusion base; determine the fusion point edge features of the fusion reference point, and make the fusion reference point coincide with the center point; based on the fusion base, fuse the fusion point edge features with the center point edge features to obtain fused video data.

[0124] In one embodiment, the video fusion module 30 is also used to determine the similar features between the edge features of the fusion point and the edge features of the center point, and the similar features exist in both the edge features of the fusion point and the edge features of the center point; the fusion reference point and the center point are fused in an orthogonal manner based on the video frame to obtain an orthogonal fused video; the feature offset of the edge features of the fusion point and the edge features of the center point in the orthogonal fused video is determined, and the feature offset is the feature offset data of the edge features of the fusion point relative to the edge features of the center point; the orthogonal fused video is offset calibrated according to the feature offset to obtain fused video data.

[0125] In one embodiment, the video fusion module 30 is also used to determine the offset distance and offset angle of the feature offset; using the offset angle as the first calibration parameter, the fusion point edge feature is rotationally offset; after completing the rotational offset, the offset direction and the offset distance can be updated based on the fusion point edge feature after the rotational offset and the center point edge feature, and an offset vector is obtained according to the offset direction and the offset distance; the fusion point edge feature is adjusted with the offset vector so that the fusion point edge feature coincides with the center point edge feature, thereby obtaining fused video data.

[0126] In one embodiment, the fire warning module 40 is further used to perform fire warning detection on the fused video data, determine the infrared detection data in the fused video data, and determine the infrared imaging image corresponding to the fused video data; determine the abnormally high temperature point based on the temperature threshold of each component of the wind turbine and the infrared imaging image; output the fire detection result according to the abnormally high temperature point; when the fire detection result is a preset fire detection result, output the fire location according to the fire detection result.

[0127] The fan fire warning device with multi-source video data fusion provided by the present application adopts the fan fire warning method with multi-source video data fusion in the above embodiment, which can solve the technical problem that the prior art cannot intervene in advance when a fire occurs. Compared with the prior art, the beneficial effects of the fan fire warning device with multi-source video data fusion provided by the present application are the same as the beneficial effects of the fan fire warning method with multi-source video data fusion provided by the above embodiment, and the other technical features of the fan fire warning device with multi-source video data fusion are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0128] The present application provides a fan fire warning device with multi-source video data fusion, and the fan fire warning device with multi-source video data fusion includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the fan fire warning method with multi-source video data fusion in the above-mentioned embodiment one.

[0129] Reference below Figure 6 , which shows a schematic diagram of the structure of a fan fire warning device for implementing multi-source video data fusion in the embodiment of the present application. The fan fire warning device for multi-source video data fusion in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 6 The fan fire warning device with multi-source video data fusion shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0130] like Figure 6As shown, the fan fire warning device with multi-source video data fusion may include a processing device 1001 (such as a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the fan fire warning device with multi-source video data fusion are also stored. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the wind turbine fire warning device with multi-source video data fusion to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a wind turbine fire warning device with multi-source video data fusion having various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or have instead.

[0131] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0132] The fan fire warning device with multi-source video data fusion provided by the present application adopts the fan fire warning method with multi-source video data fusion in the above embodiment, which can solve the technical problem that the prior art cannot intervene in advance when a fire occurs. Compared with the prior art, the beneficial effects of the fan fire warning device with multi-source video data fusion provided by the present application are the same as the beneficial effects of the fan fire warning method with multi-source video data fusion provided by the above embodiment, and the other technical features of the fan fire warning device with multi-source video data fusion are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0133] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0134] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0135] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, the computer-readable program instructions being used to execute the fan fire warning method of multi-source video data fusion in the above-mentioned embodiment.

[0136] The computer-readable storage medium provided in the present application may be, for example, a USB flash drive, but is not limited to electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, systems or devices, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, system or device. The program code contained on the computer-readable storage medium may be transmitted using any appropriate medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination of the above.

[0137] The computer-readable storage medium may be included in the fan fire warning device with multi-source video data fusion; or may exist independently without being assembled into the fan fire warning device with multi-source video data fusion.

[0138] The computer-readable storage medium carries one or more programs. When the one or more programs are executed by the fan fire warning device with multi-source video data fusion, the fan fire warning device with multi-source video data fusion:

[0139] When the video data of the wind turbine generator set has fire warning features, determining the fire warning location according to the video data;

[0140] Determine a plurality of target monitoring devices according to the fire warning position, and extract video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position;

[0141] Taking the fire warning position as the center point, fusing the video data with the video data of a plurality of target monitoring devices to obtain fused video data;

[0142] Perform fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, output the fire location according to the fire detection result.

[0143] Computer program code for performing the operations of the present application may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0144] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present application. In this regard, each box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a sequence different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flow chart, and the combination of the boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0145] The modules involved in the embodiments of the present application may be implemented by software or hardware, wherein the name of the module does not limit the unit itself in some cases.

[0146] The readable storage medium provided by the present application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned fan fire warning method of multi-source video data fusion, which can solve the technical problem that the prior art cannot intervene in advance when a fire occurs. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided by the present application are the same as the beneficial effects of the fan fire warning method of multi-source video data fusion provided by the above-mentioned embodiment, and will not be repeated here.

[0147] The present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps of the fan fire warning method of multi-source video data fusion as described above.

[0148] The computer program product provided by the present application can solve the technical problem that the prior art cannot intervene in advance when a fire occurs. Compared with the prior art, the beneficial effects of the computer program product provided by the present application are the same as the beneficial effects of the fan fire early warning method of multi-source video data fusion provided by the above embodiment, which will not be elaborated here.

[0149] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A fan fire warning method based on multi-source video data fusion, characterized in that: The fan fire warning method using multi-source video data fusion includes: When the video data of the wind turbine generator set has fire warning features, determining the fire warning location according to the video data; Determine a plurality of target monitoring devices according to the fire warning position, and extract video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position; Taking the fire warning position as the center point, fusing the video data with the video data of a plurality of target monitoring devices to obtain fused video data; Perform fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, output the fire location according to the fire detection result.

2. The method according to claim 1, characterized in that When the video data of the wind turbine generator set has fire warning features, the step of determining the fire warning location according to the video data comprises: Performing feature recognition on the video data of the wind turbine generator set to determine a temperature feature graph of the wind turbine generator set in the video data; Determine a temperature gradient of a temperature characteristic diagram of the wind turbine generator set, and when the temperature gradient is greater than a preset temperature gradient, determine a temperature change vector according to the temperature gradient; The temperature change vectors are aggregated to obtain a vector center, and the vector center is determined as a fire warning position.

3. The method according to claim 1, characterized in that The step of determining a plurality of target monitoring devices according to the fire warning position and extracting video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position, comprises: Determining the spatial position coordinates of the fire warning position according to the fire warning position; Obtaining a monitoring location map of a wind turbine monitoring device and a monitoring area of ​​each wind turbine monitoring device; Determine the wind turbine group monitoring device to be selected according to the spatial position coordinates and the monitoring site map of the wind turbine group monitoring device; Determine a plurality of target wind turbine monitoring devices according to the monitoring area of ​​the selected wind turbine monitoring device and the spatial position coordinates; The video data of the plurality of target wind turbine group monitoring devices are extracted.

4. The method according to claim 1, characterized in that The step of fusing the video data with the video data of a plurality of target monitoring devices with the fire warning position as the center point to obtain fused video data comprises: Determine the fire warning position, and determine the fire warning position as the center point; respectively determining the fire warning positions in the video data of the plurality of target monitoring devices, and determining the fire warning positions in the video data of the target monitoring devices as fusion reference points; Determine the edge feature of the center point, and determine the edge feature of the center point as a fusion base; Determine the fusion point edge feature of the fusion reference point, and make the fusion reference point coincide with the center point; Based on the fusion base, the fusion point edge feature and the center point edge feature are fused to obtain fused video data.

5. The method according to claim 4, characterized in that The step of fusing the fusion point edge feature with the center point edge feature to obtain fused video data comprises: Determine similar features between the fusion point edge feature and the center point edge feature, wherein the similar features exist simultaneously in the fusion point edge feature and the center point edge feature; Performing positive fusion of the fusion reference point and the center point based on the video frame to obtain a positive fusion video; Determine the feature offset of the fusion point edge feature and the center point edge feature in the orthogonal fusion video, where the feature offset is feature offset data of the fusion point edge feature relative to the center point edge feature; The orthogonal fused video is offset calibrated according to the characteristic offset to obtain fused video data.

6. The method according to claim 5, characterized in that The step of performing offset calibration on the orthogonal fusion video according to the characteristic offset to obtain fusion video data comprises: Determine the offset distance and the offset angle of the feature offset; Using the offset angle as a first calibration parameter, the edge feature of the fusion point is rotated and offset; After the rotational offset is completed, the offset direction and the offset distance can be updated based on the edge features of the fusion point after the rotational offset and the edge features of the center point, and an offset vector can be obtained according to the offset direction and the offset distance; The edge feature of the fusion point is adjusted by using the offset vector so that the edge feature of the fusion point coincides with the edge feature of the center point, thereby obtaining fused video data.

7. The method according to claim 1, characterized in that The step of performing fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, outputting the fire location according to the fire detection result comprises: Performing fire warning detection on the fused video data, determining infrared detection data in the fused video data, and determining an infrared imaging image corresponding to the fused video data; Determining abnormally high temperature points based on temperature thresholds of components of the wind turbine generator set and the infrared imaging image; Outputting a fire detection result according to the abnormally high temperature point; When the fire detection result is a preset fire detection result, the fire location is output according to the fire detection result.

8. A fan fire warning device with multi-source video data fusion, characterized in that: The device comprises: A point warning module, used to determine the fire warning position according to the video data when there are fire warning features in the video data of the wind turbine generator set; a video extraction module, configured to determine a plurality of target monitoring devices according to the fire warning position, and extract video data of the plurality of target monitoring devices, wherein the video data of the target monitoring devices includes the fire warning position; A video fusion module, used to fuse the video data with the video data of a plurality of target monitoring devices with the fire warning position as the center point to obtain fused video data; The fire warning module is used to perform fire warning detection on the fused video data to obtain the fire detection result, and when the fire detection result is a preset fire detection result, output the fire location according to the fire detection result.

9. A fan fire warning device with multi-source video data fusion, characterized in that: The device comprises: a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the fan fire warning method of multi-source video data fusion as described in any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the fan fire warning method based on multi-source video data fusion as described in any one of claims 1 to 7 are implemented.

Images