Smoke recognition method, device, equipment and storage medium

Abstract

The application provides a smoke identification method and device, equipment and a storage medium, and relates to the field of data analysis. The method comprises the following steps: acquiring a first image of a to-be-identified scene; the first image comprises a to-be-identified region; the to-be-identified region is a region where a fog-shaped object appears; acquiring a plurality of meteorological data corresponding to the to-be-identified scene; judging whether the fog-shaped object is smoke according to the plurality of meteorological data and a preset neural network judgment model; the neural network judgment model is used for outputting an identification type according to an image comprising a fog-shaped object and a plurality of meteorological data; the identification type comprises smoke or water mist. The method is suitable for the process of identifying smoke and is used for improving the accuracy of smoke identification.

Description

Technical Field

[0001] This application relates to the field of data analysis, and in particular to a smoke recognition method, apparatus, device, and storage medium. Background Technology

[0002] Currently, for projects such as straw burning and forest fire prevention, images of the scene to be identified can be used for analysis, detection, and alarm.

[0003] For example, the image can be analyzed and detected, and if smoke is detected in the image, an alarm can be issued.

[0004] However, false alarms due to smoke often occur in actual projects. Summary of the Invention

[0005] Based on the above-mentioned technical problems, this application provides a smoke recognition method, device, equipment and storage medium, which can detect smoke objects in an image and then use meteorological data and a preset neural network judgment model to judge and identify smoke, thereby improving the accuracy of smoke recognition of smoke objects.

[0006] In a first aspect, this application provides a smoke recognition method, which includes: acquiring a first image of a scene to be recognized; the first image includes a region to be recognized; the region to be recognized is a region where a fog-like object appears; acquiring multiple meteorological data corresponding to the scene to be recognized; determining whether the fog-like object is smoke based on the multiple meteorological data and a preset neural network judgment model; the neural network judgment model is used to output a recognition type based on the image including the fog-like object and the multiple meteorological data; the recognition type includes smoke or water mist.

[0007] Optionally, before acquiring the first image of the scene to be identified, the method further includes: acquiring a first training sample set; the first training sample set includes multiple first training samples; each first training sample includes various meteorological data and labels when a fog-like object appears; the labels are smoke or water mist; and training a preset neural network model based on the first training sample set to obtain a neural network judgment model.

[0008] In one possible implementation, obtaining the first training sample set includes: obtaining candidate meteorological data corresponding to each of the multiple candidate meteorological factors; obtaining the distribution of smoke events under different values ​​of each candidate meteorological data; obtaining the distribution of water mist events under different values ​​of each candidate meteorological data; determining the target meteorological factor from the multiple candidate meteorological factors based on the smoke distribution and water mist distribution; and determining the multiple meteorological data in the first training sample based on the target meteorological factor to obtain the first training sample set.

[0009] Optionally, the target meteorological factors include at least one of the following: time, surface temperature, relative humidity, downward shortwave radiation flux, PM2.5 concentration, surface air pressure, wind direction, and wind speed.

[0010] Optionally, based on various meteorological data and a preset neural network judgment model, before or after determining whether the fog-like object is smoke, the method further includes: acquiring a second image, the second image being a close-up image of the region to be identified included in the first image, the second image including the region to be identified after being centered and magnified; determining whether the fog-like object is smoke based on the proportion of the region to be identified after being centered and magnified to the second image and a first proportion threshold; and / or, if a skyline is identified in the second image, determining whether the fog-like object is smoke based on the proportion of the portion of the region to be identified below the skyline to the region to be identified and a second proportion threshold.

[0011] The smoke recognition method provided in this application can determine the type of fog object (smoke or water mist) by using various meteorological data corresponding to the scene to be identified after detecting the fog object. The meteorological data can reflect the surrounding environment of the scene to be identified. Combining the surrounding environment of the scene to be identified with the determination of the type of fog object is more conducive to the recognition of fog object and improves the accuracy of smoke recognition of fog object.

[0012] Secondly, this application provides a smoke recognition device, which includes various functional modules for the method described in the first aspect above.

[0013] Thirdly, this application provides an electronic device including a processor and a memory; the memory stores processor-executable instructions; when the processor is configured to execute the instructions, the electronic device performs the method described in the first aspect above.

[0014] Fourthly, this application provides a computer program product that, when run in an electronic device, causes the electronic device to perform the steps of the related method described in the first aspect, so as to implement the method described in the first aspect.

[0015] Fifthly, this application provides a readable storage medium comprising: software instructions; when the software instructions are executed in an electronic device, they cause the electronic device to perform the method described in the first aspect above.

[0016] In a sixth aspect, this application provides a smoke recognition system, which includes an image acquisition device and an electronic device as described in the third aspect, wherein the image acquisition device and the electronic device are used to cooperate with each other to implement the smoke recognition method described in the first aspect.

[0017] The beneficial effects of the second to sixth aspects mentioned above can be referred to the first aspect, and will not be repeated here. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0019] Figure 1 This is a schematic diagram of the composition of the smoke recognition system provided in the embodiments of this application;

[0020] Figure 2 A schematic diagram illustrating the composition of the electronic device provided in the embodiments of this application;

[0021] Figure 3 A schematic flowchart illustrating the smoke recognition method provided in this application embodiment;

[0022] Figure 4 A schematic diagram of smoke and water mist distribution under relative humidity factors provided in this application embodiment;

[0023] Figure 5 This is a schematic diagram of the distribution of smoke and water mist under the downward shortwave radiation flux factor provided in an embodiment of this application;

[0024] Figure 6 A schematic diagram illustrating the distribution of smoke and water mist at perceived temperature and air temperature 2 meters above the ground, provided in an embodiment of this application.

[0025] Figure 7 This is a schematic diagram illustrating the cumulative contribution rate of the target meteorological factors provided in the embodiments of this application.

[0026] Figure 8 This is a schematic diagram of the BP neural network model structure provided in the embodiments of this application;

[0027] Figure 9 Another schematic flowchart of the smoke recognition method provided in the embodiments of this application;

[0028] Figure 10 Close-up schematic diagram provided for embodiments of this application;

[0029] Figure 11 A schematic diagram of water mist provided for an embodiment of this application;

[0030] Figure 12 A skyline diagram provided for an embodiment of this application;

[0031] Figure 13 A schematic diagram of factory smoke provided for an embodiment of this application;

[0032] Figure 14 A schematic diagram of smoke in a residential building provided in an embodiment of this application;

[0033] Figure 15 This is a schematic diagram comparing motion features provided in an embodiment of this application;

[0034] Figure 16 This is another schematic flowchart of the smoke recognition method provided in the embodiments of this application;

[0035] Figure 17 This is a schematic diagram illustrating the smoke recognition effect provided in an embodiment of this application.

[0036] Figure 18 This is a comparative illustration of the version effects provided for embodiments of this application;

[0037] Figure 19 This is a schematic diagram illustrating the filtering effect provided in an embodiment of this application;

[0038] Figure 20 This is a schematic diagram illustrating the composition of the smoke recognition device provided in the embodiments of this application. Detailed Implementation

[0039] Hereinafter, the terms "first," "second," and "third," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first," "second," or "third," etc., may explicitly or implicitly include one or more of that feature.

[0040] Currently, for projects such as straw burning and forest fire prevention, images of the scene to be identified can be used for analysis, detection, and alarm.

[0041] For example, the image can be analyzed and detected, and if smoke is detected in the image, an alarm can be issued.

[0042] However, false alarms due to smoke often occur in actual projects.

[0043] For example, false alarms may be caused by smoke from factory chimneys or residential chimneys.

[0044] For example, there may be various false alarms, such as false alarms about clouds in the sky or false alarms about water mist.

[0045] Based on this, embodiments of this application provide a smoke recognition method, apparatus, device, and storage medium.

[0046] After detecting fog-like objects in an image, meteorological data and a pre-set neural network analysis model are used to identify smoke, thereby improving the accuracy of smoke identification for fog-like objects. The following description, in conjunction with accompanying figures, illustrates this process.

[0047] Figure 1 This is a schematic diagram illustrating the composition of a smoke recognition system provided in an embodiment of this application. Figure 1 As shown, the system includes an image acquisition device 100 and a smoke detection device 200. The image acquisition device 100 and the smoke detection device 200 can be connected via a wired network or a wireless network.

[0048] The image acquisition device 100 can be used to acquire images of the scene to be identified.

[0049] The scene to be identified can be a village, field, forest, mountain forest, or other similar scene.

[0050] The image acquisition device 100 can be a variable zoom camera.

[0051] In some possible embodiments, the image acquisition device 100 may be mounted on a pan-tilt head. For example, the pan-tilt head may be an electric pan-tilt head, which can drive the image acquisition device 100 to rotate horizontally or vertically to adjust the field of view of the image acquisition device 100.

[0052] In some possible embodiments, the image acquisition device 100 may also send the acquired images to the smoke recognition device 200.

[0053] As described above, the image acquisition device 100 and the smoke detection device 200 can be connected via a wired network or a wireless network. This wired or wireless network may include one or more media or devices capable of transmitting image data from the image acquisition device 100 to the smoke detection device 200.

[0054] In some embodiments, the wired or wireless network may include one or more communication media that enable the image acquisition device 100 to transmit image data directly to the smoke detection device 200 in real time. In this embodiment, the image acquisition device 100 may modulate the image data according to a communication standard (e.g., a wireless communication protocol) and transmit the modulated image data to the smoke detection device 200. The one or more communication media may include wireless and / or wired communication media, such as radio frequency (RF) spectrum or one or more physical transmission lines. Optionally, the one or more communication media may form part of a packet-based network, such as a local area network, a wide area network, or a global network (e.g., the Internet). Optionally, the one or more communication media may also include routers, switches, base stations, or other devices that facilitate communication between the image acquisition device 100 and the smoke detection device 200.

[0055] The smoke recognition device 200 is used to identify whether smoke appears in an image. The specific recognition process can be referred to the smoke recognition method in the following embodiments, which will not be repeated here.

[0056] In some possible embodiments, the smoke recognition device 200 may also be used to receive images sent by the image acquisition device 100 before the images are recognized.

[0057] The smoke detection device 200 can be an electronic device with computing processing capabilities, such as a computer or server.

[0058] The server can be a single server or a server cluster consisting of multiple servers. In some embodiments, the server cluster can also be a distributed cluster. Optionally, the server can also be implemented on a cloud platform, such as a private cloud, public cloud, hybrid cloud, community cloud, distributed cloud, inter-cloud, and multi-cloud, or any combination thereof. This application does not impose any limitations on this.

[0059] It should be noted that the above Figure 1 The description uses the image acquisition device 100 and the smoke recognition device 200 as separate devices as an example. Optionally, the image acquisition device 100 and the smoke recognition device 200 can also be combined into one device, that is, the image acquisition device 100 or its function and the smoke recognition device 200 or its function can be integrated into one device. For example, a camera with smoke recognition function (or with built-in smoke recognition algorithm). This application does not limit this.

[0060] The execution entity of the smoke recognition method provided in this application embodiment can be the smoke recognition device 200 described above. As mentioned above, the smoke recognition device 200 can be an electronic device with computing processing capabilities, such as a computer or server. Optionally, the smoke recognition device 200 can also be a processor (e.g., a central processing unit (CPU)) in the aforementioned electronic device; or, the smoke recognition device 200 can also be a software system or platform arranged in the aforementioned electronic device (e.g., the platform can be called a smoke alarm platform or other platforms, etc., which is not limited in this application embodiment); or, the smoke recognition device 200 can also be an application (APP) with smoke recognition function installed in the aforementioned electronic device; or, the smoke recognition device 200 can also be a functional module with smoke recognition function in the aforementioned electronic device, etc., which is not limited in this application embodiment.

[0061] For simplicity, the following description will use the smoke detection device 200 as an example of an electronic device.

[0062] Figure 2 This is a schematic diagram illustrating the composition of an electronic device provided in an embodiment of this application. For example... Figure 2 As shown, the electronic device may include: a processor 10, a memory 20, a communication line 30, a communication interface 40, and an input / output interface 50.

[0063] The processor 10, memory 20, communication interface 40, and input / output interface 50 can be connected via communication line 30.

[0064] The processor 10 is used to execute instructions stored in the memory 20 to implement the smoke recognition method provided in the following embodiments of this application. The processor 10 may be a CPU, a network processor (NP), a digital signal processor (DSP), a microprocessor, a microcontroller (MCU) / single-chip microcomputer / microcontroller, a programmable logic device (PLD), or any combination thereof. The processor 10 may also be any other device with processing capabilities, such as a circuit, device, or software module; this application embodiment does not limit this. In one example, the processor 10 may include one or more CPUs, for example... Figure 2 CPU0 and CPU1 are mentioned. As an optional implementation, the electronic device may include multiple processors; for example, in addition to processor 10, it may also include processor 60. Figure 2 (The example shown is a dashed line).

[0065] The memory 20 is used to store instructions. For example, the instructions may be computer programs. Optionally, the memory 20 may be a read-only memory (ROM) or other types of static storage devices that can store static information and / or instructions; it may also be a random access memory (RAM) or other types of dynamic storage devices that can store information and / or instructions; it may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, universal optical discs, Blu-ray discs, etc.), magnetic disk storage media, or other magnetic storage devices, etc. The embodiments of this application do not limit this.

[0066] It should be noted that the memory 20 can exist independently of the processor 10 or it can be integrated with the processor 10. The memory 20 can be located inside or outside the electronic device, and this application embodiment does not impose any restrictions on this.

[0067] Communication line 30 is used to transmit information between the components included in the electronic device.

[0068] The communication interface 40 is used to communicate with other devices (such as the image acquisition device 100 described above) or other communication networks. These other communication networks can be Ethernet, radio access network (RAN), wireless local area network (WLAN), etc. The communication interface 40 can be a module, circuit, transceiver, or any device capable of enabling communication.

[0069] Input / output interface 50 is used to enable human-computer interaction between users and electronic devices. For example, it enables action interaction or information exchange between users and electronic devices.

[0070] For example, the input / output interface 50 can be a mouse, keyboard, display screen, or touch screen. Action or information interaction between the user and the electronic device can be achieved through a mouse, keyboard, display screen, or touch screen.

[0071] It should be noted that, Figure 2 The structures shown do not constitute a limitation on electronic devices, except... Figure 2 In addition to the components shown, electronic devices may include more or fewer components than illustrated, or combinations of certain components, or different component arrangements.

[0072] The smoke recognition method provided in the embodiments of this application will be described below.

[0073] Figure 3 This is a flowchart illustrating the smoke recognition method provided in an embodiment of this application. Optionally, the method can be implemented by a person having the above-mentioned... Figure 2 The electronic device with the hardware structure shown performs, such as Figure 3 As shown, the method includes S101 to S103.

[0074] S101, The electronic device acquires the first image of the scene to be identified.

[0075] Optionally, as described above, the electronic device (smoke recognition device 200) and the image acquisition device 100 can be connected via a wired or wireless network. In this case, the electronic device (smoke recognition device 200) can acquire a first image of the scene to be recognized by receiving images sent by the image acquisition device 100.

[0076] S102. The electronic device acquires various meteorological data corresponding to the scene to be identified.

[0077] The process of acquiring meteorological data by electronic devices can be referred to in relevant technologies, and will not be repeated here.

[0078] S103. The electronic device determines whether the fog-like object is smoke based on various meteorological data and a preset neural network judgment model.

[0079] The neural network analysis model is used to identify the type of fog based on images of fog-like objects and various meteorological data. The identified types include water fog or smoke.

[0080] Optionally, prior to S103 above, the electronic device may also acquire a neural network judgment model.

[0081] In one possible implementation, the electronic device can directly obtain the trained neural network judgment model from other devices.

[0082] In another possible implementation, the electronic device can train a neural network judgment model using training samples. In this case, before S103 above, the method may further include steps 1 and 2:

[0083] Step 1: The electronic device acquires the first training sample set.

[0084] The first training sample set includes multiple first training samples, each of which includes various meteorological data and labels for the occurrence of fog-like objects, with the labels being smoke or water mist.

[0085] In one possible implementation, the electronic device can acquire multiple candidate meteorological data and select meteorological data relevant to the assessment of smoke and water mist as the first training sample. In this case, step 1 above can specifically include the following steps:

[0086] Step 1.1: Electronic devices acquire candidate meteorological data corresponding to various candidate meteorological factors.

[0087] The candidate meteorological factors may include local precipitation, total cloud cover, time, surface temperature, relative humidity, downward shortwave radiation flux, PM2.5 concentration, surface air pressure, wind direction, wind speed, carbon monoxide content, and nitrogen dioxide content, etc. This application does not limit these factors.

[0088] Step 1.2: The electronic device acquires the smoke distribution of the smoke event under different values ​​of each candidate meteorological data.

[0089] Step 1.3: The electronic device acquires the water mist distribution of the water mist event under different values ​​of each candidate meteorological data.

[0090] For example, Figure 4 This is a schematic diagram illustrating the distribution of smoke and water mist under a relative humidity factor provided in an embodiment of this application. Figure 4 As shown in (a), the horizontal axis represents relative humidity, and the vertical axis represents the number of smog events. Approximately 77.1% of smog events occur when the loudness and humidity meet at 73%. Figure 4 As shown in (b), the horizontal axis represents relative humidity, and the vertical axis represents the number of water mist events. Approximately 74.5% of water mist events occur when the humidity reaches 73% or higher. Therefore, relative humidity can, to some extent, distinguish between water mist and smoke.

[0091] For example, Figure 5 This is a schematic diagram illustrating the distribution of smoke and water mist under the downward shortwave radiation flux factor provided in an embodiment of this application. Figure 5 As shown in (a), the horizontal axis represents the downward shortwave radiation flux, and the vertical axis represents the number of smog events. The radiation flux of smog events is mainly in the range of (300, 650) (in watts). Shortwave radiation is the main component of solar radiation and can directly reflect the solar radiation intensity at different times. Fitting the time of smog events with the downward shortwave radiation flux factor reveals that this factor is highly correlated with time, peaking between 7:00-9:00 AM and reaching its peak between 12:00-2:00 PM. Figure 5 As shown in (b), the horizontal axis represents the downward shortwave radiation flux, and the vertical axis represents the number of water mist events. The radiation flux of water mist events is mainly in the range of [0,50].

[0092] For example, Figure 6 This is a schematic diagram showing the distribution of smoke and water mist at perceived temperature and air temperature 2 meters above the ground, as provided in the embodiments of this application. Figure 6 (a) in the figure shows the variation trends of the perceived temperature and the air temperature 2 meters below the ground in some smoke events, with the horizontal axis representing smoke events and the vertical axis representing temperature. Figure 6 (b) shows the variation trends of perceived temperature and air temperature at 2 meters above the ground for some water mist events, with the horizontal axis representing water mist events and the vertical axis representing temperature. It can be seen that the two temperature factors (perceived temperature and air temperature at 2 meters above the ground) follow the changes of (smoke or water mist) events in almost the same way, so either temperature factor can be retained as the target meteorological factor.

[0093] Step 1.4: The electronic equipment determines the target meteorological factor from multiple candidate meteorological factors based on the distribution of smoke and water mist.

[0094] The target meteorological factors may include at least one of the following: time, surface temperature, relative humidity, downward shortwave radiation flux, PM2.5 concentration, surface air pressure, wind direction, and wind speed.

[0095] Optionally, for any first candidate meteorological data corresponding to a first candidate meteorological factor among multiple candidate meteorological factors, the electronic device can determine whether the first candidate meteorological factor is the target meteorological factor based on whether the difference between the first distribution and the second distribution is greater than a preset difference threshold.

[0096] The first distribution is the distribution of smoke events under different values ​​of the first candidate meteorological data, and the second distribution is the distribution of smoke events under different values ​​of the first candidate meteorological data.

[0097] In one possible implementation, the aforementioned smoke distribution can be specifically implemented as a smoke event statistics chart, and the aforementioned water mist distribution can be specifically implemented as a water mist event statistics chart (the smoke event statistics chart and the water mist event statistics chart can be referred to the above). Figures 4 to 6 As shown (and will not be elaborated further), the difference threshold can be specifically implemented as a pixel value threshold. In this case, the electronic device can perform differential processing on the smoke event statistics map and the water mist event statistics map to obtain a difference image. The electronic device can determine whether the sum of pixel values ​​in the difference image is greater than the pixel value threshold, in order to determine whether the degree of difference between the first distribution and the second distribution meets the preset difference threshold, thereby determining whether the first candidate meteorological factor is the target meteorological factor.

[0098] Optionally, after the target meteorological factors are selected, the electronic device can also use principal component analysis (PCA) to reduce the dimensionality of the target meteorological factors.

[0099] Alternatively, the electronic device can specifically follow these steps to use PCA to reduce the dimensionality of the target meteorological factors:

[0100] X1. Convert the meteorological data corresponding to the target meteorological factor into a two-dimensional feature matrix.

[0101] X2. Decentering the two-dimensional feature matrix, which means subtracting the average value from each eigenvalue in the two-dimensional feature matrix.

[0102] X3. Calculate the covariance matrix and perform eigenvalue decomposition on it.

[0103] X4. Sort the eigenvalues ​​(contribution rates) from largest to smallest, and select the k largest eigenvalues ​​to obtain the dimensionality-reduced matrix through data transformation.

[0104] The specific process of the PCA method for X1 to X4 described above can be found in the relevant technical documents, and will not be repeated here.

[0105] For example, Figure 7 This is a schematic diagram illustrating the cumulative contribution rate of the target meteorological factors provided in the embodiments of this application. Figure 7 As shown, taking the target meteorological factors including time, surface temperature, relative humidity, downward shortwave radiation flux, PM2.5 concentration, surface air pressure, wind direction, and wind speed as an example, the contribution rate of these eight target meteorological factors can reach 88% of the contribution rate of all meteorological factors.

[0106] Step 1.5: The electronic device determines various meteorological data in the first training sample based on the target meteorological factors, and obtains the first training sample set.

[0107] For example, an electronic device can use multiple meteorological data corresponding to a target meteorological factor as multiple meteorological data in the first training sample to obtain the first training sample, and then repeatedly obtain the first training sample to obtain the first training sample set.

[0108] Optionally, the electronic device can divide multiple first training samples into a first positive sample set and a first negative sample set according to labels. In this case, the first training samples can include a first positive sample set and a first negative sample set. The first positive sample set can include multiple first positive training samples, each of which is various meteorological data when smoke occurs, and the label of each first positive training sample is "smoke". The first negative sample set can include multiple first negative training samples, each of which is various meteorological data when water mist occurs, and the label of each first negative training sample is "water mist".

[0109] Step 2: The electronic device trains the preset neural network model based on the first training sample set to obtain the neural network judgment model.

[0110] The preset neural network model can be a back propagation (BP) neural network model.

[0111] For example, Figure 8 This is a schematic diagram of the BP neural network model structure provided in an embodiment of this application. Figure 8 As shown, the BP neural network model can include an input layer, a hidden layer, and an output layer. Electronic devices can input various meteorological data into the input layer of the BP neural network model, process it through the hidden layer, and then output the judgment result as smoke or water mist from the output layer.

[0112] Alternatively, the electronic device may specifically train the BP neural network model according to the following process:

[0113] (1) Data normalization:

[0114] In order to make the established neural network model have a faster convergence speed and a stronger generalization ability, the original second training samples need to be preprocessed (normalized) to avoid the saturation region of the transfer function and keep the value of the processed training samples between 0.2 and 0.8. The normalization formula is shown in the following formula (1):

[0115]

[0116] In formula (1), x i x represents the initial value of any meteorological data (e.g., the first meteorological data) in the second training sample. max This indicates the maximum value reached by this type of meteorological data. x′ i This represents the normalized value of the meteorological data.

[0117] (2) Model training:

[0118] a. Input the normalized data into the preset neural network model.

[0119] b. Calculate the output of each hidden layer neuron according to the following formula (2):

[0120]

[0121] In formula (2), z k This represents the output of the k-th hidden layer neuron. ki This represents the connection weights between the input layer and the hidden layer.

[0122] c. Calculate the output of the output layer node according to the following formula (3):

[0123]

[0124] In formula (3), y j This represents the output of the j-th output layer node. jk This represents the connection weights between the hidden layer and the output layer.

[0125] d. Calculate the error according to the following formula (4):

[0126]

[0127] In formula (4), E P This represents the error value during forward propagation. This represents the actual value. This indicates the calculated output value.

[0128] e. Calculate the updated weights using backpropagation of the error values ​​according to the following formula (5):

[0129]

[0130] f. Use the first five steps to complete one forward propagation of data and backpropagation of error. Repeat this process to continuously update the connection weights so that the calculated output value is close to the actual value, until the preset number of training iterations is reached.

[0131] In the smoke recognition method provided in this application embodiment, after detecting a fog-like object, the electronic device can use various meteorological data corresponding to the scene to be recognized to determine the type of fog-like object (smoke or water mist). The meteorological data can reflect the surrounding environmental conditions of the scene to be recognized. Combining the surrounding environmental conditions of the scene to be recognized to determine the type of fog-like object is more conducive to the recognition of fog-like objects and improves the accuracy of smoke recognition of fog-like objects.

[0132] In some possible embodiments, before or after making a judgment using a neural network analysis model, the electronic device may also take a close-up view of the area to be identified and make a preliminary judgment or further judgment based on the close-up image. In this case, Figure 9 This is another schematic flowchart illustrating the smoke recognition method provided in an embodiment of this application. Figure 9 As shown, before or after S103 above, the method may further include:

[0133] S201, The electronic device acquires the second image.

[0134] The second image is a close-up of the region to be identified included in the first image. The second image includes the region to be identified, magnified and centered. In one possible implementation, the electronic device can use a preset detection algorithm to detect the fog-like object in the first image and mark the location of the fog-like object using a detection box. The area defined by the detection box is the region to be identified in S101 above. Specific processes can be found in related technologies and will not be repeated here.

[0135] Optionally, the electronic device may also record the acquisition time of the first image, the position information of the region to be detected (detection box) in the first image (e.g., the position coordinates of the pixel in the first image in the Xth row and Yth column or in other coordinate systems), the field of view orientation of the image acquisition device 100 when acquiring the first image, the current field of view orientation of the image acquisition device 100, the acquisition time of the second image, and other information.

[0136] For example, during normal cruise, the electronic device can extract one frame every M frames as the first image for analysis and record the field of view (T1) of the image acquisition device 100 when acquiring the first image. If a region containing a fog-like object is identified in the first image, the electronic device can record the current field of view (T2) of the image acquisition device 100, control the image acquisition device 100 to return to the field of view (T1) when acquiring the first image, adjust the gimbal position and zoom parameters (or camera magnification), center and magnify the identified region, and capture the second image. The electronic device can then perform further identification on the second image captured by the image acquisition device 100 to further determine whether the fog-like object is smoke. After the determination is completed, the electronic device can control the image acquisition device 100 to return to the previous field of view (T2) and continue cruise according to the set cruise path.

[0137] Wherein, M is a positive integer, which can be preset in the electronic device by the administrator. For example, M can be 3, 4, or 5, etc. The specific value of M is not limited in the embodiments of this application.

[0138] For example, Figure 10 A close-up schematic diagram provided for an embodiment of this application. For example... Figure 10 As shown in (a), in the image acquired by the image acquisition device 100, which is positioned at a higher location, some foggy objects are blurry and cannot be clearly identified at normal cruise magnification because they are far away from the image acquisition device 100. Figure 10 As shown in (b), in order to clearly identify the fog-like object, after the fog-like object is identified at the normal cruise magnification, the area to be identified (detection box) can be centered and enlarged, and the area to be identified can be zoomed in again. The second image obtained by zooming in is then verified.

[0139] S202, the electronic device determines whether the fog-like object is smoke based on the proportion of the centered and magnified area to be identified in the second image and a first proportion threshold; and, or, if a skyline is identified in the second image, the electronic device determines whether the fog-like object is smoke based on the proportion of the area to be identified below the skyline to the area to be identified and a second proportion threshold.

[0140] The first proportional threshold can be preset in the electronic device by the administrator; for example, the first proportional threshold can be 30%, 40%, or 50%. This application embodiment does not limit the specific value of the first proportional threshold. The second proportional threshold can also be preset in the electronic device by the administrator; for example, the second proportional threshold can be 40%, 50%, or 60%. This application embodiment does not limit the specific value of the second proportional threshold.

[0141] For example, Figure 11 This is a schematic diagram of water mist provided for an embodiment of this application. Figure 11 As shown, in the second image, the region to be identified ( Figure 11 (As shown in the example with the black solid line box) The proportion is relatively large, so it can be considered as a large area of ​​water mist produced by a large forest, and thus determined to be a false alarm.

[0142] It should be understood that in areas with extensive forest cover, a large amount of water vapor can evaporate, easily forming water mist. Since smoke and water mist produced by combustion are not significantly different in color and shape, and both exhibit significant variations depending on wind speed and direction, they are easily confused. The smoke recognition method provided in this application can filter out larger fog-like objects by determining whether the proportion of the area to be identified in the second image is less than a threshold. Water mist is produced by the large amount of water vapor evaporated from extensive forests, and its area is typically large, while smoke produced in the early stages of combustion typically has a smaller area. Therefore, the larger fog-like objects filtered out by this application can be considered water mist, thus avoiding interference from water mist and improving the accuracy of smoke recognition.

[0143] For example, Figure 12 A skyline diagram provided for an embodiment of this application. For example... Figure 12 As shown, clouds (or mist) floating in the sky (above the horizon) have similar characteristics to smoke and may be identified as fog-like objects, thus allowing the area to be identified above the horizon to be identified. Figure 12 (The example shown is a black solid line frame). Therefore, if most of the area to be identified is above the horizon, it can be considered a false alarm of clouds (color); if most of the area to be identified is below the horizon, further identification of fog-like objects in the area to be identified can continue.

[0144] Alternatively, the electronic device can also determine whether a fog-like object is smoke based on the following conditions:

[0145] 1) When a false alarm reference object is identified in the second image, the positional relationship between the area to be identified and the false alarm reference object satisfies the preset relationship.

[0146] For example, an electronic device can use a semantic segmentation model to perform semantic segmentation on a second image, segmenting it into multiple different types of regions, which may include false alarm references.

[0147] Optionally, the electronic device can also acquire the trained semantic segmentation model.

[0148] Alternatively, the electronic device can directly obtain the trained semantic segmentation model from other devices.

[0149] Optionally, the electronic device can also acquire training samples to train the semantic segmentation algorithm and obtain a semantic segmentation model.

[0150] For example, electronic devices can acquire images of factory chimneys, residential buildings, and clouds in the sky, and obtain the user's semantic segmentation annotation operations on the images. Semantic labels are then added to the segmented regions, and the semantically labeled images are used to train the semantic segmentation algorithm to obtain a semantic segmentation model.

[0151] In one possible implementation, the false alarm reference may specifically include factories and / or residential buildings, and the preset relationship may specifically include: there is no intersection between the area to be identified and the area where the factory is located and / or the area where the residential buildings are located.

[0152] For example, Figure 13 This is a schematic diagram of factory smoke provided for an embodiment of this application. Figure 13 As shown, in the area to be identified ( Figure 13 (As shown in the example with a black solid line box) If there is overlap or intersection between the area to be identified and the factory area, the fog-like object in the area to be identified can be considered as smoke generated by the factory, not smoke generated by straw burning or forest fire. Therefore, if the area to be identified intersects with the factory area obtained by semantic segmentation, it can be considered a false alarm; if the area to be identified does not intersect with the factory area obtained by semantic segmentation, the fog-like object in the area to be identified can continue to be identified.

[0153] For example, Figure 14 This is a schematic diagram of smoke in a residential building provided as an embodiment of this application. Figure 14 As shown, in the area to be identified ( Figure 14(As shown in the example with the black solid line box) When there is overlap or intersection between the area to be identified and the area where the residential houses are located, the fog-like objects in the area to be identified can be considered as smoke generated by the residential houses, not smoke generated by straw burning or forest fire. Therefore, if the area to be identified intersects with the residential house area obtained by semantic segmentation, it can be considered a false alarm; if the area to be identified does not intersect with the residential house area obtained by semantic segmentation, the fog-like objects in the area to be identified can continue to be identified.

[0154] 2) The motion characteristics of the fog-like object meet the preset characteristics.

[0155] Optionally, the preset features may include the motion characteristics of smoke, and the electronic device may also determine whether the motion characteristics of the fog-like object satisfy the motion characteristics of smoke. In this case, the method may further include the following steps:

[0156] Step 1: The electronic device acquires multiple frames of images of the foggy object.

[0157] The multiple frames of the fog-like object can be continuous video or images with intervals. This application does not impose any limitations on this.

[0158] For example, as described in S201 above, if the electronic device can record the acquisition time of the first image, then after identifying the region to be identified in the first image, the electronic device can read multiple frames of images before or after the acquisition time of the first image.

[0159] Step 2: Based on the multi-frame images of the fog object and the preset classification model, determine the types of motion features of the fog object.

[0160] The classification model is used to determine the types of motion features of the fog object based on multi-frame images of the fog object. The types of motion features of the fog object can include the motion features of water mist or the motion features of smoke.

[0161] Optionally, prior to step 2 above, the electronic device may also acquire a classification model.

[0162] In one possible implementation, the electronic device can directly obtain the trained classification model from other devices.

[0163] In another possible implementation, the electronic device can train a classification model using training samples. In this case, the method may further include S1 and S2 before step 2 above:

[0164] S1. The electronic device acquires the second training sample set.

[0165] The second training sample set includes a second positive sample set and a second negative sample set. The second positive sample set contains multiple second positive training samples, each a video clip of smoke, and each labeled with the motion features of the smoke. The second negative sample set contains multiple second negative training samples, each a video clip of water mist, and each labeled with the motion features of the water mist.

[0166] S2. The electronic device uses the second training sample set to train the preset video classification algorithm to obtain a classification model.

[0167] Optionally, the electronic device can input two or more second training samples (second positive training samples or second negative training samples) into the video classification algorithm each time to obtain a predicted value (the type of motion feature corresponding to a certain video segment predicted by the video classification algorithm), calculate the loss function based on the predicted value and the label of the training sample, and adjust the parameters of the video classification algorithm. In this way, multiple second training samples from the second training sample set are input into the video classification algorithm for iterative training until convergence.

[0168] Optionally, the conditions for the video classification algorithm to converge (end training) may include: the electronic device inputting the second training sample into the preset video classification algorithm a preset number of times reaches a preset number threshold, or the error between the predicted value and the label is less than a preset error threshold.

[0169] For example, Figure 15 This is a schematic diagram comparing motion features provided in an embodiment of this application. Figure 15 As shown in (a), water mist typically floats in the air, exhibiting horizontal movement (left and right) and slow speed. For example... Figure 15 As shown in (b), when straw, weeds, or trees burn, they continuously produce smoke. This smoke has the characteristics of moving vertically upwards from the burning roots on the ground, continuously spreading outwards, and moving at a high speed.

[0170] It should be noted that the above-mentioned conditions for determining whether a fog-like object is smoke or water mist (judgment based on meteorological data and neural network analysis model, judgment based on the proportion of the center-enlarged area to be identified in the second image and the first proportion threshold, judgment based on the proportion of the area below the horizon to the area to be identified and the second proportion threshold, judgment based on whether the positional relationship between the area to be identified and the false alarm reference object meets the preset relationship, and judgment based on the motion characteristics of the fog-like object) can be any one of them, can be combined arbitrarily, or can all be satisfied simultaneously. This application embodiment does not impose any restrictions on this. The order of the judgment steps can be arbitrarily combined or executed simultaneously. This application embodiment also does not impose any restrictions on the specific judgment order.

[0171] Based on the understanding of the above embodiments, Figure 16 This is another schematic flowchart illustrating the smoke recognition method provided in an embodiment of this application. Figure 16 As shown, the method may include S301 to S309.

[0172] S301. The electronic device acquires the alarm image and the coordinate information of the alarm location box in the alarm image.

[0173] S301 can be referred to as described in S201 above, and will not be repeated here.

[0174] S302. Center and enlarge the alarm location frame.

[0175] S302 can be referred to as described in S201 above, and will not be repeated here.

[0176] S303. Segmentation is performed using a semantic segmentation algorithm.

[0177] S303 can be referred to in the above embodiments, and will not be repeated here.

[0178] S304, False alarm reference filtering.

[0179] Specifically, S304 may include S3041 and S3042.

[0180] S3041, filtration for factories and residential buildings.

[0181] S3041 can be referred to the above. Figure 13 and Figure 14 As mentioned above, it will not be repeated here.

[0182] S3042, Skyline Filtering.

[0183] S3042 can be referred to the above. Figure 12 As mentioned above, it will not be repeated here.

[0184] S305, Area Size Filtering.

[0185] S305 can be referred to the above. Figure 11 As mentioned above, it will not be repeated here.

[0186] S306, Video classification algorithm filtering.

[0187] S306 can be referred to the above. Figure 15 As mentioned above, it will not be repeated here.

[0188] S307, Neural Network Judgment Model Filtering.

[0189] S307 can be referred to in S103 above, and will not be repeated here.

[0190] S308, Smoke alarm.

[0191] S309, Push to user for review.

[0192] For example, Figure 17 This is a schematic diagram illustrating the smoke recognition effect provided in an embodiment of this application. Figure 17 As shown, taking the alarm data from two days of cloudy and heavy rain as an example, there is actually a lot of water mist. Using the improved smoke recognition method V5.5.45_220829 provided in this application embodiment, 350 devices generated alarms on site, with a total of 1506 smoke alarms generated in two days, of which 928 were false alarms, with an average of 1.33 false alarms / day / device.

[0193] For example, Figure 18 This is a comparative illustration of the version effects provided for embodiments of this application. For example... Figure 18 As shown, compared with the previous version V5.5.45_220617, the improved device has reduced the total number of false alarms, water vapor (water mist) false alarms, building smoke false alarms, and the average number of non-smoke false alarms.

[0194] For example, Figure 19 This is a schematic diagram illustrating the filtering effect provided in an embodiment of this application. Figure 19 As shown, taking the 856 suspected smoke events (fog-like objects) detected from November 1st to November 16th as an example, there were a total of 402 water mist events. The algorithm detected 177 water mist events and 149 correct water mist events, with a detection rate of 44.03% and a precision rate of 83.03%. That is, in the original alarm, it can reduce the review workload for users by 44%.

[0195] The foregoing primarily describes the solutions provided by the embodiments of this application from a methodological perspective. To achieve the aforementioned functions, it includes corresponding hardware structures and / or software modules for executing each function. Those skilled in the art should readily recognize that, in conjunction with the units and algorithm steps of the various examples described in the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0196] In an exemplary embodiment, this application also provides a smoke recognition device. Figure 20 This is a schematic diagram illustrating the composition of the smoke recognition device provided in an embodiment of this application. Figure 20As shown, the device includes: an acquisition module 2001 and a processing module 2002.

[0197] The acquisition module 2001 is used to acquire a first image of the scene to be identified; the first image includes the area to be identified; the area to be identified is the area where fog-like objects appear; and to acquire various meteorological data corresponding to the scene to be identified.

[0198] The processing module 2002 is used to determine whether a fog-like object is smoke based on various meteorological data and a preset neural network judgment model. The neural network judgment model is used to output the identification type based on the image of the fog-like object and various meteorological data. The identification type includes smoke or water mist.

[0199] In some possible embodiments, the acquisition module 2001 is further configured to acquire a first training sample set before acquiring a first image of the scene to be identified; the first training sample set includes multiple first training samples; each first training sample includes various meteorological data and labels when a fog-like object appears; the labels are smoke or water mist; the processing module 2002 is further configured to train a preset neural network model based on the first training sample set to obtain a neural network judgment model.

[0200] In other possible embodiments, the acquisition module 2001 is specifically used to acquire candidate meteorological data corresponding to each of the multiple candidate meteorological factors; acquire the smoke distribution of the smoke event under different values ​​of each candidate meteorological data; acquire the water mist distribution of the water mist event under different values ​​of each candidate meteorological data; determine the target meteorological factor from the multiple candidate meteorological factors based on the smoke distribution and water mist distribution; and determine the multiple meteorological data in the first training sample based on the target meteorological factor to obtain the first training sample set.

[0201] In some other possible embodiments, the target meteorological factors include at least one of the following: time, surface temperature, relative humidity, downward shortwave radiation flux, PM2.5 concentration, surface air pressure, wind direction, and wind speed.

[0202] In some other possible embodiments, before or after the processing module 2002 determines whether the fog-like object is smoke based on various meteorological data and a preset neural network judgment model, the acquisition module 2001 is further used to acquire a second image; the second image is a close-up image of the area to be identified included in the first image, and the second image includes the area to be identified after being centered and magnified; the processing module 2002 is further used to determine whether the fog-like object is smoke based on the proportion of the area to be identified after being centered and magnified to the second image and a first proportion threshold; and / or, if a skyline is identified in the second image, determine whether the fog-like object is smoke based on the proportion of the part of the area to be identified below the skyline to the area to be identified and a second proportion threshold.

[0203] It should be noted that, Figure 20 The module division shown is illustrative and represents only one logical functional division; in actual implementation, other division methods are possible. For example, two or more functions can be integrated into a single processing module. These integrated modules can be implemented in hardware or as software functional units.

[0204] In an exemplary embodiment, this application also provides a readable storage medium including software instructions that, when run on an electronic device, cause the electronic device to perform any of the methods provided in the above embodiments.

[0205] In an exemplary embodiment, this application also provides a computer program product containing computer execution instructions, which, when run on an electronic device, causes the electronic device to perform any of the methods provided in the above embodiments.

[0206] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented using software programs, implementation can be, in whole or in part, in the form of a computer program product. This computer program product includes one or more computer-executable instructions. When these computer-executable instructions are loaded and executed on a computer, all or part of the flow or function according to the embodiments of this application is generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer-executable instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, computer-executable instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium accessible to a computer or a data storage device containing one or more servers, data centers, etc., that can be integrated with the medium. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state disks, SSDs).

[0207] Although this application has been described herein in conjunction with various embodiments, those skilled in the art, by reviewing the accompanying drawings, disclosure, and appended claims, will understand and implement other variations of the disclosed embodiments in carrying out the claimed application. In the claims, the word "comprising" does not exclude other components or steps, and "a" or "an" does not exclude multiple components. A single processor or other unit can implement several functions listed in the claims. While different dependent claims may recite certain measures, this does not mean that these measures cannot be combined to produce good results.

[0208] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made thereto without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely exemplary illustrations of this application as defined by the appended claims, and are considered to cover any and all modifications, variations, combinations, or equivalents within the scope of this application. Clearly, those skilled in the art can make various alterations and modifications to this application without departing from the spirit and scope of this application. Thus, if such modifications and modifications of this application fall within the scope of the claims of this application and their equivalents, this application is also intended to include such modifications and modifications.

[0209] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A smoke detection method, characterized in that, The method includes: Acquire a first image of the scene to be identified; the first image includes a region to be identified; the region to be identified is the area where a foggy object appears; Obtain various meteorological data corresponding to the scene to be identified; Based on the various meteorological data and the preset neural network judgment model, it is determined whether the fog-like object is smoke; the neural network judgment model is used to output the identification type based on the image including the fog-like object and various meteorological data; the identification type includes smoke or water mist; The method for determining whether the fog-like object is before or after smoke, based on the various meteorological data and a preset neural network judgment model, further includes: Acquire a second image, which is a close-up image of the region to be identified included in the first image, and the second image includes the region to be identified after being magnified in the center; Based on the proportion of the centered, magnified region to be identified in the second image and a first proportion threshold, it is determined whether the fog-like object is smoke; and, if a skyline is identified in the second image, it is determined whether the fog-like object is smoke based on the proportion of the portion of the region to be identified below the skyline in the region to be identified and a second proportion threshold.

2. The method according to claim 1, characterized in that, Before acquiring the first image of the scene to be identified, the method further includes: Obtain a first training sample set; the first training sample set includes multiple first training samples; each first training sample includes various meteorological data and labels when a foggy object appears; the labels are smoke or water mist. The preset neural network model is trained based on the first training sample set to obtain the neural network judgment model.

3. The method according to claim 2, characterized in that, The process of obtaining the first training sample set includes: Obtain candidate meteorological data for each of the various candidate meteorological factors; Obtain the smoke distribution of the smoke event under different values ​​of each candidate meteorological data; Obtain the water mist distribution of water mist events under different values ​​of each candidate meteorological data; Based on the distribution of smoke and water mist, a target meteorological factor is determined from the multiple candidate meteorological factors; Based on the target meteorological factors, various meteorological data in the first training sample are determined to obtain the first training sample set.

4. The method according to claim 3, characterized in that, The target meteorological factors include at least one of the following: time, surface temperature, relative humidity, downward shortwave radiation flux, PM2.5 concentration, surface air pressure, wind direction, and wind speed.

5. A smoke detection device, characterized in that, include: Acquisition module and processing module; The acquisition module is used to acquire a first image of the scene to be identified; the first image includes a region to be identified; the region to be identified is a region where a fog-like object appears; and to acquire various meteorological data corresponding to the scene to be identified. The processing module is used to determine whether the fog-like object is smoke based on the various meteorological data and a preset neural network judgment model; the neural network judgment model is used to output a recognition type based on the image including the fog-like object and various meteorological data; the recognition type includes smoke or water mist. Before or after the processing module determines whether the fog-like object is smoke based on the various meteorological data and a preset neural network judgment model, the acquisition module is further configured to acquire a second image; the second image is a close-up image of the area to be identified included in the first image, and the second image includes the area to be identified after being centered and magnified; the processing module is further configured to determine whether the fog-like object is smoke based on the proportion of the centered and magnified area to be identified in the second image and a first proportion threshold; and, if a skyline is identified in the second image, determine whether the fog-like object is smoke based on the proportion of the portion of the area to be identified below the skyline to the area to be identified and a second proportion threshold.

6. The apparatus according to claim 5, characterized in that, The acquisition module is further configured to acquire a first training sample set before acquiring the first image of the scene to be identified; the first training sample set includes multiple first training samples; each first training sample includes multiple meteorological data and labels when a fog-like object appears; the labels are smoke or water mist; the processing module is further configured to train a preset neural network model based on the first training sample set to obtain the neural network judgment model; And / or, The acquisition module is specifically used to acquire candidate meteorological data corresponding to each of the multiple candidate meteorological factors; acquire the smoke distribution of a smoke event under different values ​​of each candidate meteorological data; acquire the water mist distribution of a water mist event under different values ​​of each candidate meteorological data; determine a target meteorological factor from the multiple candidate meteorological factors based on the smoke distribution and the water mist distribution; and determine multiple meteorological data in the first training sample based on the target meteorological factor to obtain the first training sample set. And / or, The target meteorological factors include at least one of the following: time, surface temperature, relative humidity, downward shortwave radiation flux, PM2.5 concentration, surface air pressure, wind direction, and wind speed.

7. An electronic device, characterized in that, include: Processor and memory; The memory stores instructions that the processor can execute; When the processor is configured to execute the instructions, the electronic device performs the method as described in any one of claims 1-4.

8. A readable storage medium, characterized in that, The readable storage medium includes: software instructions; When the software instructions are executed in an electronic device, the electronic device causes the electronic device to perform the method as described in any one of claims 1-4.

9. A smoke detection system, characterized in that, include: Image acquisition device and electronic device as described in claim 7.

Images