Heat source monitoring method and device, electronic device and storage medium

Abstract

The application relates to a heat source monitoring method and device, an electronic device and a storage medium, wherein the heat source monitoring method comprises the following steps: acquiring a visible light image and an infrared image photographed for a target area; performing feature extraction processing on the visible light image to generate a visible light feature map, and performing feature enhancement processing on the infrared image to generate an attention weight map; performing weighting processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map, and generating a monitoring result indicating whether the target area contains a heat source according to at least the enhanced feature map. Through the application, the problem of low accuracy of firework monitoring is solved, and an accurate and efficient heat source monitoring method is realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of video monitoring, in particular to a heat source monitoring method and device, an electronic device and a storage medium. BACKGROUND

[0002] Due to the suddenness of fire, the randomness of disaster occurrence, and the characteristics of causing huge losses in a short time, once a smoke event occurs, fire fighting measures must be taken at an extremely fast speed. The timeliness and accuracy of fire fighting are mostly determined by whether the fire is discovered in time, whether the analysis is accurate and reasonable, and whether the decision-making measures are appropriate. With the popularization of security devices and intelligent functions, more and more intelligent demands are used for smoke event early warning to reduce life and property losses.

[0003] In the related art, the smoke monitoring method includes video monitoring, ground patrol and satellite remote sensing. The video monitoring is to acquire images by using a long-focus lens in cooperation with a low-illumination camera, and to identify smoke by using images. However, due to the complexity and diversity of the appearance of smoke, and the high similarity between clouds, fog and night lights and the appearance of smoke in appearance features, interference is brought to the identification of smoke hazards, resulting in high false negative rate and high false positive rate, and thus the accuracy of smoke monitoring is low.

[0004] At present, there is no effective solution to the problem of low accuracy of smoke monitoring in the related art. SUMMARY

[0005] Embodiments of the present application provide a heat source monitoring method, device, electronic device and storage medium to at least solve the problem of low accuracy of smoke monitoring in the related art.

[0006] In a first aspect, the embodiments of the present application provide a heat source monitoring method, which comprises:

[0007] obtaining a visible light image and an infrared image photographed for a target area; performing feature extraction processing on the visible light image to generate a visible light feature map, and performing feature enhancement processing on the infrared image to generate an attention weight map;

[0008] performing weighted processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map, and generating a monitoring result indicating whether the target area contains a heat source according to at least the enhanced feature map.

[0009] In some embodiments, the heat source includes smoke, and the generating of the monitoring result indicating whether the target area contains a heat source according to at least the enhanced feature map comprises:

[0010] detecting the pyrotechnic target in the enhanced feature map, and obtaining a pyrotechnic target region in the visible light image containing the pyrotechnic target to be identified if the pyrotechnic target to be identified is detected;

[0011] calculating a region overlap degree between the pyrotechnic target region and the exhaust target region, and generating the monitoring result according to the region overlap degree.

[0012] In some embodiments, the generating the monitoring result according to the region overlap degree comprises:

[0013] performing multi-class scene segmentation processing on the visible light image to obtain at least two scene classification results in the visible light image;

[0014] if the region overlap degree is less than a preset overlap threshold, generating the monitoring result based on the scene classification result, and sending alarm level information corresponding to the scene classification result to an alarm application program or a terminal device.

[0015] In some embodiments, the scene classification result comprises a sky region and a non-sky region, and the generating the monitoring result based on the scene classification result if the region overlap degree is less than a preset overlap threshold comprises:

[0016] determining the sky region and the non-sky region in the visible light image according to the scene classification result, and detecting whether the pyrotechnic target to be identified is located in the sky region or the non-sky region if the region overlap degree is less than the preset overlap threshold.

[0017] if the pyrotechnic target to be identified is located in the sky region, generating a first monitoring result indicating that the pyrotechnic target to be identified is a non-pyrotechnic target, and if the pyrotechnic target to be identified is located in the non-sky region, generating a second monitoring result indicating that the pyrotechnic target to be identified is a pyrotechnic target.

[0018] In some embodiments, the performing multi-class scene segmentation processing on the visible light image to obtain at least two scene classification results in the visible light image comprises:

[0019] performing multi-class scene segmentation processing on the visible light image using a semantic segmentation model to obtain the at least two scene classification results, wherein the semantic segmentation model is generated according to a training classification image containing a preset classification scene, and the preset classification scene corresponds to the scene classification result.

[0020] In some embodiments, the generating the monitoring result according to the degree of overlap of the region comprises:

[0021] In a case where it is detected that the degree of overlap of the region is greater than or equal to a preset overlap threshold, a first monitoring result indicating that the to-be-identified pyrotechnic target is a non-pyrotechnic target is generated, and tail gas feature information corresponding to the tail gas target is acquired;

[0022] A preset abnormal tail gas feature is acquired, and in a case where it is detected that the tail gas feature information matches the abnormal tail gas feature, it is determined that the tail gas target emits abnormally.

[0023] In some embodiments, the feature extraction processing on the visible light image generates a visible light feature map, and the feature enhancement processing on the infrared image generates an attention weight map, which comprises:

[0024] The visible light image and the infrared image are both input into a trained target detection model, the feature extraction module in the target detection model is used to perform feature extraction processing on the visible light image to generate the visible light feature map, and the feature enhancement module in the target detection model is used to perform feature enhancement processing on the infrared image to generate the attention weight map.

[0025] In some embodiments, before the acquiring the visible light image and the infrared image for the target region, the method further comprises:

[0026] Acquiring a to-be-trained visible light image and a to-be-trained infrared image containing a preset heat source target;

[0027] The to-be-trained visible light image is input into a main network in a preset neural network model, and the to-be-trained visible light image is trained by a first convolutional layer of the main network to obtain a to-be-trained visible light feature map; wherein the feature extraction module at least includes the first convolutional layer;

[0028] The to-be-trained infrared image is input into an auxiliary network in the neural network model, and the to-be-trained infrared image is sequentially subjected to convolution processing and normalization processing by a second convolutional layer and an activation function of the auxiliary network to obtain a to-be-trained attention weight map; wherein the feature enhancement module at least includes the second convolutional layer and the activation function;

[0029] The to-be-trained attention weight map and the to-be-trained visible light feature map are subjected to matrix multiplication processing by a third convolutional layer in the main network according to first pixel coordinates of the to-be-trained visible light image and second pixel coordinates of the to-be-trained infrared image, and a to-be-trained enhanced feature map is generated;

[0030] The fourth convolutional layer in the main network is used for convolution feature extraction processing on the to-be-trained enhanced feature map, and a detection frame result containing the preset heat source target is output; and the neural network model is trained according to the preset heat source target and the detection frame result until the neural network model converges and the target detection model is obtained.

[0031] In some embodiments, the weight value assigned to the first hotspot region in the attention weight map is greater than the weight value assigned to the second hotspot region; and the temperature of the first hotspot region is higher than the temperature of the second hotspot region.

[0032] In a second aspect, the embodiments of the present application provide a heat source monitoring device, the device comprising: an acquisition module and a generation module;

[0033] The acquisition module is configured to acquire a visible light image and an infrared image captured for a target region, perform feature extraction processing on the visible light image to generate a visible light feature map, and perform feature enhancement processing on the infrared image to generate an attention weight map.

[0034] The generation module is configured to perform weighted processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map, and generate a monitoring result indicating whether the target region contains a heat source according to at least the enhanced feature map.

[0035] In a third aspect, the embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the heat source monitoring method of the first aspect.

[0036] In a fourth aspect, the embodiments of the present application provide a storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the heat source monitoring method of the first aspect.

[0037] Compared with the related art, the heat source monitoring method, device, electronic device and storage medium provided by the embodiments of the present application acquire a visible light image and an infrared image captured for a target region, perform feature extraction processing on the visible light image to generate a visible light feature map, perform feature enhancement processing on the infrared image to generate an attention weight map, perform weighted processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map, and generate a monitoring result indicating whether the target region contains a heat source according to at least the enhanced feature map, thereby solving the problem of low accuracy of fireworks monitoring and implementing an accurate and efficient heat source monitoring method.

[0038] The details of one or more embodiments of the present application are set forth in the accompanying drawings and the description below, so that other features, objects and advantages of the present application become more apparent. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This is an application environment diagram of a heat source monitoring method according to an embodiment of this application;

[0041] Figure 2 This is a flowchart of a heat source monitoring method according to an embodiment of this application;

[0042] Figure 3 This is a schematic diagram of a training method for an object detection model according to an embodiment of this application;

[0043] Figure 4 This is a flowchart of a heat source monitoring method according to a preferred embodiment of this application;

[0044] Figure 5 This is a structural block diagram of a heat source monitoring device according to an embodiment of this application;

[0045] Figure 6 This is a structural block diagram of a fireworks monitoring system according to an embodiment of this application;

[0046] Figure 7 This is a structural diagram of the internal structure of a computer device according to an embodiment of this application. Detailed Implementation

[0047] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application. Furthermore, it is understood that although the efforts made in such a development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, modifications to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0048] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0049] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application means two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The terms “first,” “second,” “third,” etc., used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0050] The heat source monitoring method provided in this application embodiment can be applied to, for example, Figure 1In the application environment shown, the image acquisition device 12 communicates with the control device 14 via a network. The control device 14 acquires visible light and infrared images of the target area through the image acquisition device 12, performs feature extraction processing on the visible light image to generate a visible light feature map, and performs feature enhancement processing on the infrared image to generate an attention weight map. The control device 14 then performs weighted processing on the visible light feature map based on the attention weight map to obtain an enhanced feature map, and generates a monitoring result for the target area based at least on the enhanced feature map. The image acquisition device 12 can be, but is not limited to, various binocular cameras, PTZ cameras, or other devices used for image acquisition. The control device 14 can be, but is not limited to, various server devices, personal computers or laptops, processing chips, or other control devices. The server device can be implemented using a standalone server or a server cluster composed of multiple servers. It is understood that the above application environment can also be configured to include only smart cameras implemented by smart cameras; that is, the heat source monitoring method provided in this application embodiment can also be executed independently on any smart camera, and does not necessarily require two separate devices: a control device and an image acquisition device, thereby effectively reducing the cost of the heat source monitoring method.

[0051] This embodiment provides a method for monitoring heat sources. Figure 2 This is a flowchart of a heat source monitoring method according to an embodiment of this application, such as... Figure 2 As shown, the process includes the following steps:

[0052] Step S220: Acquire visible light and infrared images of the target area.

[0053] The aforementioned target area refers to a pre-defined area where potential heat sources such as smoke or fire need to be monitored; for example, this target area could be a square, tunnel, or highway. Specifically, visible light and infrared imaging devices can be pre-installed at the shooting location for this target area. Taking a section of highway as an example, the visible light and infrared imaging devices can be installed facing the highway section. The visible light imaging device captures a visible light image containing the target area, and the infrared imaging device captures an infrared image containing the target area. It is understood that the visible light and infrared imaging devices can be installed on the same image acquisition device, or they can be on different devices. It should be noted that after acquiring the captured visible light and infrared images, the control device can pre-process image alignment to ensure that the scene position in the visible light image is the same as the scene position in the infrared image, and that the image size and other information of the visible light image are consistent with those of the infrared image, thereby improving the accuracy of subsequent image processing.

[0054] Step S240: Perform feature extraction processing on the visible light image to generate a visible light feature map, and perform feature enhancement processing on the infrared image to generate an attention weight map.

[0055] The feature extraction process described above can be as follows: Feature extraction is performed on the visible light image using methods such as machine learning, pattern recognition, or image processing to extract salient features from the visible light image, and the visible light feature map is generated based on the extracted salient features. The feature enhancement process described above can be as follows: Feature extraction is performed on the infrared image using methods such as machine learning or image processing to extract salient features from the infrared image, and normalization and other processing are performed on the extracted salient features of the infrared image so that the features of each pixel in the infrared image can be assigned corresponding weight values.

[0056] In some embodiments, the weight value assigned to the first hotspot region in the aforementioned attention weight map is greater than the weight value assigned to the second hotspot region; wherein the temperature of the first hotspot region is higher than the temperature of the second hotspot region. Specifically, algorithms such as neural networks can be used to assign corresponding weight values ​​to each pixel in the infrared image according to the temperature information of the infrared image, following a trend from high to low temperature. The resulting attention weight map assigns higher weight values ​​to high hotspot regions, making them the focus of attention. Conversely, low hotspot regions assign lower weight values, so that they do not require much attention in subsequent image processing.

[0057] Step S260: The visible light feature map is weighted according to the attention weight map to obtain an enhanced feature map, and at least a monitoring result indicating whether the target area contains a heat source is generated based on the enhanced feature map.

[0058] The aforementioned heat sources include fireworks, combustibles, or other objects posing safety hazards. Specifically, the control device can perform matrix multiplication between the coordinates of each pixel in the attention weight map and the coordinates of each pixel in the visible light feature map. This allows the weight values ​​from the attention weight map to be superimposed onto the visible light feature map, thereby achieving a weighted operation on the visible light image based on the attention weight map. This enhances the target features in the visible light image and generates the aforementioned enhanced feature map. The control device can then further perform target detection on the enhanced feature map using methods such as neural networks or image processing. Finally, based on whether the target area is detected to contain a heat source, a corresponding monitoring result is generated.

[0059] Through steps S220 to S260, a visible light feature map is generated by extracting features from the visible light image containing the target area, an attention weight map is generated by enhancing the features of the infrared image containing the target area, and the visible light feature map is weighted according to the attention weight map to obtain an enhanced feature map. Finally, a monitoring result is generated based on the enhanced feature map. This achieves focused attention on areas with higher weight values ​​on the attention feature map, thereby enhancing the features of the target under the visible light image, improving the recall rate of heat source target detection, avoiding the phenomenon that similar features such as clouds, fog, and night lights can easily interfere with the identification of heat source hazards, solving the problem of low accuracy in heat source monitoring, and realizing an accurate and efficient heat source monitoring method.

[0060] In some embodiments, the heat source includes fireworks, and the process of generating a monitoring result indicating whether the target area contains a heat source based at least on the enhanced feature map further includes the following steps:

[0061] Step S261: Perform smoke target detection on the enhanced feature map, and if a smoke target to be identified is detected, obtain the smoke target region in the visible light image that contains the smoke target to be identified; perform exhaust target detection on the enhanced feature map, and obtain the exhaust target region in the visible light image.

[0062] Specifically, the enhanced feature map can be detected using machine learning, target recognition, and other methods, and the target fireworks and exhaust gas targets to be identified in the visible light image can be detected respectively, thereby determining the fireworks target area where the target fireworks are located and the exhaust gas target area where the exhaust gas targets are located.

[0063] Step S262: Calculate the degree of overlap between the target area of ​​the fireworks and the target area of ​​the exhaust gas, and generate the monitoring result based on the degree of overlap.

[0064] The degree of overlap between the aforementioned regions can refer to information such as the area, ratio, or difference of overlap between the fireworks target region and the exhaust gas target region. Alternatively, to improve the accuracy of the algorithm, the degree of overlap can also be the intersection-over-union (IoU) ratio between the fireworks target region and the exhaust gas target region. When the control device detects the fireworks target and the exhaust gas target to be identified through target detection, it calculates the degree of overlap between the fireworks target region and the exhaust gas target region. If the control device detects that the degree of overlap is small, for example, less than a certain value, it can be concluded that the fireworks target to be identified does not belong to vehicle exhaust, and a monitoring result indicating the presence of fireworks in the target region can be generated. Conversely, if the control device detects that the degree of overlap is large, for example, greater than or equal to a certain value, it can be concluded that the overlap area between the fireworks target and the exhaust gas target is large, and the fireworks target to be identified is vehicle exhaust, and a monitoring result indicating the absence of fireworks in the target region can be generated.

[0065] Through steps S261 to S262, smoke and fire targets and exhaust gas targets are detected on the enhanced feature map. The degree of overlap between the calculated smoke and fire target areas and exhaust gas target areas is used to determine whether there is a smoke and fire hazard. This effectively avoids interference from vehicle exhaust gas on smoke and fire monitoring in scenarios such as highways, and further improves the accuracy of smoke and fire monitoring.

[0066] In some embodiments, generating the monitoring result based on the degree of overlap of the region further includes the following steps: performing multi-category scene segmentation processing on the visible light image to obtain at least two scene classification results in the visible light image; when the degree of overlap of the region is detected to be less than a preset overlap threshold, generating the monitoring result based on the scene classification result, and sending alarm level information corresponding to the scene classification result to the alarm application or terminal device.

[0067] The scene segmentation process described above can be as follows: The visible light image is segmented according to different scene categories using machine learning or pixel-based clustering methods. This segmentation, in turn, determines the scene classification results based on different scenes such as roads, forests, or skies. The preset overlap threshold can be set by the user, for example, to 0.20. The control device compares the overlap degree of the region with the preset overlap threshold. If the overlap degree is less than the preset threshold, it indicates that the target smoke is not vehicle exhaust, and further detection can be performed. Specifically, the scene classification results can be used to determine the scene category of the target smoke, and a monitoring result can be generated based on this determination, leading to the corresponding alarm level information. For example, if the target smoke is detected in a forest scene, a monitoring result indicating the presence of a target smoke can be generated, along with the highest alarm level information. It should be further noted that the aforementioned control device can be implemented using a terminal device. In this case, the control device can send the alarm level information to its associated alarm application, which then runs the alarm program based on the alarm level information. Alternatively, the control device can also be a server device, processing chip, or other device besides a terminal device. In this case, the control device can send the alarm level information corresponding to the aforementioned scene classification results to the terminal device, and the terminal device can perform corresponding operations such as sending emails and alarms based on the alarm level information.

[0068] Through the above embodiments, multiple scene classification results are obtained by performing multi-category scene segmentation on visible light images, avoiding interference from other targets such as clouds, fog, and nighttime lights in complex scenes, thereby further improving the accuracy of smoke and fire monitoring. Furthermore, by setting corresponding alarm levels based on scene classification results, users can perform corresponding protection and rescue operations according to the urgency of the event, which can effectively improve the safety of heat source monitoring methods.

[0069] In some embodiments, the scene classification result includes sky areas and non-sky areas; when the overlap of the detected areas is less than a preset overlap threshold, generating the monitoring result based on the scene classification result further includes the following steps:

[0070] Step S263: Determine the sky region and non-sky region in the visible light image based on the scene classification result; if the overlap of the detected regions is less than the preset overlap threshold, detect whether the target to be identified is located in the sky region or the non-sky region; if the target to be identified is located in the sky region, generate a first monitoring result indicating that the target to be identified is a non-smoke target; if the target to be identified is located in the non-sky region, generate a second monitoring result indicating that the target to be identified is a smoke target.

[0071] The control device can segment the visible light image into sky and non-sky regions by performing scene segmentation processing, and record the location information of each region. If the overlap between the smoke target region and the exhaust target region is less than a preset overlap threshold (meaning the smoke target is not vehicle exhaust), the control device can further detect the smoke target to eliminate interference from clouds and fog. Specifically, the control device can determine whether the detected smoke target is located in the sky or non-sky region based on the location of the sky and non-sky regions segmented according to the scene classification results. If the smoke target is located in the sky region, it indicates that the smoke target may be a misidentified cloud or fog, and a first monitoring result indicating that the smoke target is not a smoke target can be generated; or if the smoke target is located in the non-sky region, interference from clouds and fog can be eliminated, and a second monitoring result indicating that the smoke target is a smoke target can be generated.

[0072] Through the above step S263, the visible light image is divided into sky region and non-sky region by scene segmentation method, and it is determined whether the target of the firework to be identified is located in the sky region. This avoids the interference of targets such as clouds and fog in the sky region on the detection of firework, thereby further improving the accuracy of the heat source monitoring method.

[0073] In some embodiments, the above-mentioned multi-category scene segmentation processing of the visible light image to obtain at least two scene classification results in the visible light image further includes the following steps: performing multi-category scene segmentation processing on the visible light image using a semantic segmentation model to obtain the at least two scene classification results; wherein, the semantic segmentation model is generated based on training classification images containing preset classification scenes, and the preset classification scenes correspond to the scene classification results. In this embodiment, machine learning methods can be used to perform image scene segmentation processing on the visible light image. Specifically, the control device can acquire training classification images containing preset classification scenes input by the user; the training classification images can be multiple; the preset classification scenes can be set by the user in advance according to actual needs, for example, the captured training classification information can be pre-divided into areas such as forest, highway, and sky, and labeled accordingly. The control device uses the labeled preset classification scene information to annotate the training classification images to obtain training data; wherein, the training data can also be divided into training sets and test sets, which are used as input images for a preset neural network model. Then, the control device inputs the processed training classification images into the neural network model for training to generate the above-mentioned semantic segmentation model. This semantic segmentation model can be used to perform scene segmentation processing on the aforementioned visible light images to be identified. The processing method is similar to the training method described above, and will not be repeated here.

[0074] Through the above embodiments, a semantic segmentation model is generated by training a training classification image containing a preset classification scene, and the scene segmentation process is performed on the visible light image by the semantic segmentation model, thereby improving the accuracy of scene segmentation and further improving the accuracy of the heat source monitoring method.

[0075] In some embodiments, generating the monitoring result based on the degree of overlap of the regions further includes the following steps: when the degree of overlap of the regions is detected to be greater than or equal to a preset overlap threshold, generating a first monitoring result indicating that the target of the fireworks to be identified is a non-firework target, and acquiring exhaust gas characteristic information corresponding to the exhaust gas target; acquiring preset abnormal exhaust gas characteristics, and determining that the exhaust gas target emission is abnormal when the exhaust gas characteristic information matches the abnormal exhaust gas characteristics. The control device can compare the degree of overlap between the fireworks target region and the exhaust gas region with the preset overlap threshold. If the degree of overlap of the regions is detected to be greater than or equal to the preset overlap threshold, it indicates that the overlap between the target of the fireworks to be identified and the detected exhaust gas target is too high, and thus the target of the fireworks to be identified can be determined to be an exhaust gas target, i.e., generating a first monitoring result indicating that the target of the fireworks to be identified is a non-firework target. Simultaneously, the control device can further detect the exhaust gas characteristics such as color and volume of the aforementioned exhaust gas targets, and match this information with preset abnormal exhaust gas characteristics to achieve classification processing for the exhaust gas targets. These abnormal exhaust gas characteristics can be preset by the user to indicate the characteristic information corresponding to exhaust gas targets in an abnormal state. For example, the abnormal exhaust gas characteristic can be preset to black; that is, if an exhaust gas target with a black color is detected, it is in an abnormal state, and at this time, it can be determined that the exhaust gas target is emitting abnormally, thereby executing the corresponding alarm processing.

[0076] Through the above embodiments, when the target of the firework is detected to be vehicle exhaust, further detection is performed on whether the vehicle exhaust emissions are abnormal, so that an automatic alarm can be triggered in a timely manner when abnormal vehicle exhaust emissions are detected, thereby effectively improving the intelligence level of heat source monitoring and further improving the safety of heat source monitoring methods.

[0077] In some embodiments, the above-mentioned process of extracting features from the visible light image to generate a visible light feature map and performing feature enhancement processing on the infrared image to generate an attention weight map further includes the following steps: inputting both the visible light image and the infrared image into a fully trained target detection model, using the feature extraction module in the target detection model to extract features from the visible light image to generate the visible light feature map, and using the feature enhancement module in the target detection model to enhance features from the infrared image to generate the attention weight map.

[0078] In some embodiments, a method for training an object detection model is provided. Figure 3 This is a schematic diagram of a training method for an object detection model according to an embodiment of this application, as shown below. Figure 3 As shown, before acquiring the visible light and infrared images of the target area, the heat source monitoring method further includes the following steps:

[0079] Step S211: Obtain a visible light image and an infrared image to be trained that contain a preset heat source target.

[0080] The control device collects and acquires aligned visible light and infrared images of the target area, and uses these two images as a pair of input images to establish a training sample set for generating the target detection model.

[0081] Step S212: Input the visible light image to be trained into the main network of the preset neural network model, and train the visible light image to be trained through the first convolutional layer of the main network to obtain the visible light feature map to be trained; wherein, the feature extraction module includes at least the first convolutional layer.

[0082] Specifically, please refer to Figure 3 The aforementioned control device inputs the visible light image and the infrared image to be trained into the aforementioned preset neural network model; wherein, the neural network model includes a main network and an auxiliary network, and the visible light image to be trained is trained through the first convolutional layer in the feature extraction module of the main network, thereby extracting a visible light feature map containing significant feature information.

[0083] Step S213: The infrared image to be trained is input into the auxiliary network of the neural network model, and the infrared image to be trained is sequentially processed by the second convolutional layer and the activation function of the auxiliary network to perform convolution and normalization processing to obtain the attention weight map to be trained; wherein, the feature enhancement module includes at least the second convolutional layer and the activation function.

[0084] In this process, after the infrared image to be trained is input into the neural network model through the control device, the infrared image to be trained can be processed by the second convolutional layer and activation function in the feature enhancement model of the auxiliary network, such as feature extraction, normalization, and sigmoid operation, to generate an attention weight map. Each pixel in the weight map can be autonomously assigned a weight according to the temperature from high to low or from low to high, that is, the higher the temperature, the higher the weight value assigned to the region.

[0085] Step S214: Through the third convolutional layer in the main network, based on the first pixel coordinates of the visible light image to be trained and the second pixel coordinates of the infrared image to be trained, the attention weight map to be trained and the visible light feature map to be trained are multiplied by matrix, and an enhanced feature map to be trained is generated.

[0086] Specifically, the third convolutional layer in the main network performs weighted processing on the attention weight map and the visible light feature map to be trained, resulting in the enhanced feature map to be trained.Figure 3 An example is given below, showing that this weighted processing can be calculated using matrix multiplication, as shown in Formulas 1 and 2 below:

[0087]

[0088] C = A T ×B Formula 2

[0089] In the above formula, matrix A represents the set of pixels in the attention weight map to be trained; each element a in matrix A... ij The matrix A is used to represent the coordinates of each pixel in the attention weight map; where n and m are positive integers greater than 1, and n and m can be the same or different; and 0 ≤ i ≤ n, 0 ≤ j ≤ m. Matrix B is used to represent the set of pixels in the above-mentioned visible light feature map to be trained. Matrix C is used to represent the set of pixels in the above-mentioned enhanced feature map to be trained; as shown in Formulas 1 and 2, matrix C can be obtained by multiplying the transpose of A by matrix B. Then... Figure 3 The values ​​of each matrix in the example are input into Formula 2 above to calculate the result. Figure 3 The feature map to be trained in the image.

[0090] Step S215: The fourth convolutional layer in the main network is used to perform convolutional feature extraction on the feature map to be trained, and the detection box result containing the preset heat source target is output; the neural network model is trained according to the preset heat source target and the detection box result until the neural network model converges and the target detection model is obtained.

[0091] It is understandable that after the target detection model is trained through steps S211 to S215, similar steps can be used to use the target detection model to detect targets in the visible light image and infrared image to obtain the heat source targets included in the target area.

[0092] Through the above steps S211 to S215, the construction method of the dual-input channel target detection model is realized, which enables accurate identification of visible light images and infrared images through the same target detection model, thereby further improving the efficiency and accuracy of the heat source monitoring method.

[0093] The embodiments of this application will be described in detail below with reference to a practical application scenario, taking the detection of smoke and fire targets in a highway environment as an example. Figure 4 This is a flowchart of a heat source monitoring method according to a preferred embodiment of this application, such as... Figure 4 As shown, the process includes the following steps:

[0094] Step S401: Acquire video surveillance footage taken towards the highway; wherein the video surveillance footage includes a visible light image and an infrared image; perform image alignment processing on the visible light image and the infrared image in advance to ensure that the scene position and image size are consistent between the visible light image and the infrared image.

[0095] Step S402: Construct a semantic segmentation model based on deep learning, train the semantic segmentation model, select the best training result to generate an optimized semantic segmentation model; input a visible light image, use the semantic segmentation model to segment the visible light image into a scene, divide the monitoring screen in the visible light image into a sky area and a non-sky area, and record their positions.

[0096] Step S403: Construct a deep learning-based object detection model. This object detection model contains two input channels. The main network is an object detection network, and the input is a visible light image. The main network's feature extraction module generates a visible light feature map. The auxiliary network is a feature enhancement module, and the input is an infrared image. The feature enhancement module normalizes and performs a sigmoid operation on the infrared image to detect high hotspots and generate an attention weight map. The weight values ​​of high hotspot areas are higher, making high hotspot areas the focus of attention.

[0097] Step S404: The attention weight map and the visible feature map are weighted to enhance the target features in the visible light image and generate an enhanced feature map.

[0098] Step S405: Using the target detection module described above, convolutional feature extraction and localization are performed on the enhanced feature map to detect the firework target and the car tail target to be identified, and the corresponding positions are recorded. For the detected car tail target area, the exhaust gas area is divided from it, the position of the exhaust gas target area is recorded and the corresponding position mark is stored.

[0099] Step S406: Calculate the IoU between the fireworks target area and the exhaust gas target area, and determine whether the IoU is less than the preset overlap threshold.

[0100] Step S407: If the judgment result of step S406 is yes, then proceed to step S408; if the judgment result of step S406 is no, then determine that the target of the firework to be identified is automobile exhaust, and then classify the color of the automobile exhaust. If the color of the automobile exhaust is black or other non-compliant color, it indicates that the automobile exhaust emissions are not up to standard and the system alarms. If the color is normal, the system does not alarm.

[0101] Step S408: Based on the positions of the sky area and non-sky area divided by scene segmentation in step S402, determine whether the target to be identified for fireworks detected in step S405 is located in the sky area. If it is located in the sky area, determine that the target to be identified for fireworks is a non-smoke target and the system does not alarm; otherwise, the system alarms.

[0102] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0103] This embodiment also provides a heat source monitoring device, which is used to implement the above embodiments and preferred embodiments, and will not be repeated as described thereon. As used below, the terms "module," "unit," "subunit," etc., can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.

[0104] Figure 5 This is a structural block diagram of a heat source monitoring device according to an embodiment of this application, such as... Figure 5 As shown, the device includes: an acquisition module 52 and a generation module 54; the acquisition module 52 is used to acquire visible light images and infrared images of a target area, perform feature extraction processing on the visible light images to generate a visible light feature map, and perform feature enhancement processing on the infrared images to generate an attention weight map; the generation module 54 is used to perform weighted processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map, and at least generate a monitoring result indicating whether the target area contains a heat source according to the enhanced feature map.

[0105] Through the above embodiments, the acquisition module 52 performs feature extraction processing on the visible light image containing the target area to generate a visible light feature map, performs feature enhancement processing on the infrared image containing the target area to generate an attention weight map, and the generation module 54 performs weighted processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map. Finally, the monitoring result is generated based on the enhanced feature map, thereby realizing the focus on areas with higher weight values ​​on the attention feature map, thereby enhancing the features of the target of interest under the visible light image, improving the recall rate of heat source target detection, avoiding the phenomenon that similar features such as clouds, fog, and night lights can easily interfere with the identification of heat source hazards, solving the problem of low accuracy in heat source monitoring, and realizing an accurate and efficient heat source monitoring device.

[0106] In some embodiments, the heat source monitoring device further includes a training module; the training module is used to acquire a visible light image and an infrared image containing a preset heat source target to be trained; the training module inputs the visible light image to be trained into the main network of a preset neural network model, and trains the visible light image to be trained through the first convolutional layer of the main network to obtain a visible light feature map to be trained; wherein, the feature extraction module includes at least the first convolutional layer; the training module inputs the infrared image to be trained into the auxiliary network of the neural network model, and sequentially performs convolution processing and normalization processing on the infrared image to be trained through the second convolutional layer and activation function of the auxiliary network to obtain an attention to be trained. The weight map; wherein, the feature enhancement module includes at least the second convolutional layer and the activation function; the training module, through the third convolutional layer in the main network, performs matrix multiplication on the attention weight map and the visible light feature map to be trained based on the first pixel coordinates of the visible light image to be trained and the second pixel coordinates of the infrared image to be trained, and generates an enhanced feature map to be trained; the training module, through the fourth convolutional layer in the main network, performs convolutional feature extraction on the enhanced feature map to be trained, and outputs a detection box result containing the preset heat source target; the neural network model is trained based on the preset heat source target and the detection box result until the neural network model converges and the target detection model is obtained.

[0107] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0108] This embodiment also provides a smoke monitoring system. Figure 6 This is a structural block diagram of a fireworks monitoring system according to an embodiment of this application, such as... Figure 6 As shown, the system includes: an image acquisition device 12 and a control device 14; the image acquisition device 12 is used to acquire visible light images and infrared images of a target area, and send the visible light images and the infrared images to the control device 14; the control device 14 is used to perform feature extraction processing on the visible light images to generate a visible light feature map, and perform feature enhancement processing on the infrared images to generate an attention weight map; the control device 14 performs weighted processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map, and at least generates a monitoring result indicating whether the target area contains a heat source based on the enhanced feature map.

[0109] Through the above embodiments, the control device 14 performs feature extraction processing on the visible light image containing the target area to generate a visible light feature map, performs feature enhancement processing on the infrared image containing the target area to generate an attention weight map, and performs weighted processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map. Finally, the monitoring result is generated based on the enhanced feature map, thereby realizing the focus on areas with higher weight values ​​on the attention feature map, thereby enhancing the features of the target under the visible light image, improving the recall rate of heat source target detection, avoiding the phenomenon that similar features such as clouds, fog, and night lights can easily interfere with the identification of heat source hazards, solving the problem of low accuracy in heat source monitoring, and realizing an accurate and efficient smoke and fire monitoring system.

[0110] Furthermore, the control device 14 can communicate with the image acquisition device 12 via a transmission device. This transmission device is used to receive or send data via a network. The network includes a wireless network provided by the platform's communication provider. In one embodiment, the transmission device includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another embodiment, the transmission device can be a Radio Frequency (RF) module for wireless communication with the Internet.

[0111] In some embodiments, the above-mentioned fireworks monitoring system further includes a terminal device; the control device 14 is also used to perform multi-category scene segmentation processing on the visible light image to obtain at least two scene classification results in the visible light image; when the control device 14 detects that the degree of overlap in the area is less than a preset overlap threshold, it generates the monitoring result based on the scene classification result and sends the alarm level information corresponding to the scene classification result to the alarm application or the terminal device.

[0112] In some embodiments, a computer device is provided, which may be a server. Figure 7 This is a structural diagram of the internal structure of a computer device according to an embodiment of this application, such as... Figure 7As shown, the computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database stores monitoring results. The network interface communicates with external terminals via a network connection. When the computer program is executed by the processor, it implements the aforementioned heat source monitoring method.

[0113] Those skilled in the art will understand that Figure 7 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0114] This embodiment also provides an electronic device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0115] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0116] Optionally, in this embodiment, the processor can be configured to perform the following steps via a computer program:

[0117] S1, acquire visible light and infrared images of the target area; perform feature extraction processing on the visible light image to generate a visible light feature map, and perform feature enhancement processing on the infrared image to generate an attention weight map.

[0118] S2, the visible light feature map is weighted according to the attention weight map to obtain an enhanced feature map, and at least the enhanced feature map is used to generate a monitoring result indicating whether the target area contains a heat source.

[0119] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0120] Furthermore, in conjunction with the heat source monitoring methods in the above embodiments, this application embodiment can provide a storage medium for implementation. The storage medium stores a computer program; when executed by a processor, the computer program implements any of the heat source monitoring methods in the above embodiments.

[0121] It should be further noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties.

[0122] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0123] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0124] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A method for monitoring a heat source, characterized in that, The method includes: Acquire visible light and infrared images of the target area; The visible light image is subjected to feature extraction processing to generate a visible light feature map, and the infrared image is subjected to feature enhancement processing to generate an attention weight map; The visible light feature map is weighted according to the attention weight map to obtain an enhanced feature map, and a monitoring result indicating whether the target area contains a heat source is generated based at least on the enhanced feature map; The heat source includes fireworks. Fireworks target detection is performed on the enhanced feature map. If a fireworks target to be identified is detected, the fireworks target region containing the fireworks target to be identified is obtained in the visible light image. Exhaust gas target detection is performed on the enhanced feature map, and the exhaust gas target region in the visible light image is obtained; The degree of overlap between the target area of ​​the fireworks and the target area of ​​the exhaust gas is calculated; the monitoring result is generated based on the degree of overlap, including: The visible light image is subjected to multi-category scene segmentation processing to obtain at least two scene classification results in the visible light image; If the degree of overlap of the detected areas is less than a preset overlap threshold, the monitoring result is generated based on the scene classification result, and the alarm level information corresponding to the scene classification result is sent to the alarm application or terminal device. The scene classification result includes sky areas and non-sky areas; generating the monitoring result based on the scene classification result when the degree of overlap of the detected areas is less than a preset overlap threshold includes: Based on the scene classification results, the sky region and non-sky region in the visible light image are determined; if the degree of overlap of the regions is less than the preset overlap threshold, it is detected whether the firework target to be identified is located in the sky region or the non-sky region. If the target fireworks are located in the sky area, a first monitoring result indicating that the target fireworks are not a firework target is generated; if the target fireworks are located in the non-sky area, a second monitoring result indicating that the target fireworks are a firework target is generated.

2. The heat source monitoring method according to claim 1, characterized in that, The multi-category scene segmentation processing of the visible light image to obtain at least two scene classification results in the visible light image includes: The visible light image is segmented into multiple categories using a semantic segmentation model to obtain at least two scene classification results; wherein the semantic segmentation model is generated based on a training image containing a preset classification scene, and the preset classification scene corresponds to the scene classification result.

3. The heat source monitoring method according to claim 1, characterized in that, The step of generating the monitoring result based on the degree of overlap of the regions includes: If the degree of overlap of the detected areas is greater than or equal to a preset overlap threshold, a first monitoring result is generated indicating that the target to be identified as a non-firework target, and exhaust gas characteristic information corresponding to the exhaust gas target is obtained. Obtain preset abnormal exhaust gas characteristics, and determine that the target exhaust gas emission is abnormal when the exhaust gas characteristic information matches the abnormal exhaust gas characteristics.

4. The heat source monitoring method according to claim 1, characterized in that, The step of performing feature extraction processing on the visible light image to generate a visible light feature map, and performing feature enhancement processing on the infrared image to generate an attention weight map, includes: Both the visible light image and the infrared image are input into a fully trained target detection model. The feature extraction module in the target detection model is used to extract features from the visible light image to generate the visible light feature map. The feature enhancement module in the target detection model is used to enhance the features of the infrared image to generate the attention weight map.

5. The heat source monitoring method according to claim 4, characterized in that, Before acquiring the visible light and infrared images of the target area, the method further includes: Acquire a visible light image and an infrared image to be trained that contain a preset heat source target; The visible light image to be trained is input into the main network of a preset neural network model, and the visible light image to be trained is trained through the first convolutional layer of the main network to obtain the visible light feature map to be trained; wherein, the feature extraction module includes at least the first convolutional layer; The infrared image to be trained is input into the auxiliary network of the neural network model, and the infrared image to be trained is sequentially processed by the second convolutional layer and the activation function of the auxiliary network to perform convolution and normalization processing to obtain the attention weight map to be trained; wherein, the feature enhancement module includes at least the second convolutional layer and the activation function; Through the third convolutional layer in the main network, the attention weight map and the visible light feature map to be trained are multiplied by matrix according to the first pixel coordinate of the visible light image to be trained and the second pixel coordinate of the infrared image to be trained, and an enhanced feature map to be trained is generated. The fourth convolutional layer in the main network is used to perform convolutional feature extraction on the feature map to be trained, and outputs a detection box result containing the preset heat source target; the neural network model is trained according to the preset heat source target and the detection box result until the neural network model converges and the target detection model is obtained.

6. The heat source monitoring method according to any one of claims 1 to 5, characterized in that, The weight value assigned to the first hotspot region in the attention weight map is greater than the weight value assigned to the second hotspot region; wherein, the temperature of the first hotspot region is higher than the temperature of the second hotspot region.

7. A heat source monitoring device, characterized in that, The device includes: an acquisition module and a generation module; The acquisition module is used to acquire visible light images and infrared images captured for the target area, perform feature extraction processing on the visible light images to generate a visible light feature map, and perform feature enhancement processing on the infrared images to generate an attention weight map. The generation module is configured to perform weighted processing on the visible light feature map according to the attention weight map to obtain an enhanced feature map, and at least generate a monitoring result indicating whether the target area contains a heat source based on the enhanced feature map; wherein the heat source includes fireworks, the generation module is further configured to perform fireworks target detection on the enhanced feature map, and if a fireworks target to be identified is detected, obtain a fireworks target area in the visible light image containing the fireworks target to be identified; perform exhaust target detection on the enhanced feature map, and obtain an exhaust target area in the visible light image; calculate the degree of regional overlap between the fireworks target area and the exhaust target area; The generation module is also used to generate the monitoring result based on the degree of overlap of the regions, including performing multi-category scene segmentation processing on the visible light image to obtain at least two scene classification results in the visible light image; and generating the monitoring result based on the scene classification results when the degree of overlap of the regions is detected to be less than a preset overlap threshold, and sending alarm level information corresponding to the scene classification results to the alarm application or terminal device. The scene classification result includes a sky region and a non-sky region. The generation module is further configured to generate the monitoring result based on the scene classification result when the degree of overlap of the regions is less than a preset overlap threshold. This includes determining the sky region and the non-sky region in the visible light image based on the scene classification result; detecting whether the target fireworks are located in the sky region or the non-sky region when the degree of overlap of the regions is less than the preset overlap threshold; and generating a first monitoring result indicating that the target fireworks are a non-firework target when the target fireworks are located in the sky region, and generating a second monitoring result indicating that the target fireworks are a firework target when the target fireworks are located in the non-sky region.

8. An electronic device comprising a memory and a processor, characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to perform the heat source monitoring method according to any one of claims 1 to 6.

9. A storage medium, characterized in that, The storage medium stores a computer program, wherein the computer program is configured to execute the heat source monitoring method according to any one of claims 1 to 6 when it runs.

Images