A flame detection method and a flame detection model training method and device

Abstract

The embodiment of the present application provides a kind of flame detection method, flame detection model training method and device, it is related to image processing technical field, comprising: obtaining the image to be detected based on the video frame that potential flame area is detected in the video to be processed is obtained;Multiple associated images located in the video to be processed are obtained from the video to be processed;According to the order of each associated image in the video to be processed, the similarity of each associated image and the image to be detected is calculated in turn, and the similarity sequence corresponding to the image to be detected is obtained;The similarity sequence corresponding to the image to be detected is input into the first flame detection model trained in advance, and the probability that the type label representing potential flame area in the image to be detected is each type label of preset is obtained;If the probability corresponding to the flame type label in each type of preset is maximum, it is determined that the image to be detected contains target flame area, and the accuracy of flame detection can be improved.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and in particular to a flame detection method, a flame detection model training method, and an apparatus. Background Technology

[0002] In related technologies, when detecting a fire in a target scene, target video of the target scene is acquired. Since the flames are bright during a fire, for each video frame in the target video, the presence of a bright image region within that frame is detected. If a bright image region is detected in that video frame, the similarity between bright image regions in adjacent video frames is calculated. Because the shape of a flame changes as it burns, a low similarity between bright image regions in adjacent video frames indicates the presence of a flame region in the target video. This leads to the determination that a fire has occurred in the target scene.

[0003] However, video flame detection algorithms are poorly resistant to interference from bright objects or targets of similar color, making them prone to false alarms. For example, the brightness of light areas in video frames is also high, and the shape of light areas in adjacent video frames may change for flickering lights. Detecting flames based on the similarity of brighter image areas in adjacent video frames may lead to misidentifying light areas as flame areas. Therefore, the accuracy of flame region detection in related technologies is relatively low. Summary of the Invention

[0004] The purpose of this invention is to provide a flame detection method, a flame detection model training method, and an apparatus to improve the accuracy of flame detection. The specific technical solution is as follows:

[0005] Firstly, to achieve the above objectives, embodiments of the present invention provide a flame detection method, the method comprising: acquiring an image to be detected; wherein the image to be detected is obtained based on video frames containing potential flame regions detected in a video to be processed; acquiring multiple associated images located after the image to be detected from the video to be processed; calculating the similarity between each associated image and the image to be detected in sequence according to the order of each associated image in the video to be processed, thereby obtaining a similarity sequence corresponding to the image to be detected; inputting the similarity sequence corresponding to the image to be detected into a pre-trained first flame detection model to obtain the probability that the type label representing the potential flame region in the image to be detected is a preset type label; wherein the first flame detection model is trained based on the similarity sequence corresponding to a first sample image and the target type label of the first sample image; if the probability corresponding to the flame type label among the preset types is the highest, it is determined that the image to be detected contains a target flame region.

[0006] Optionally, the step of obtaining the image to be detected includes: obtaining a video to be processed; for each video frame in the video to be processed, if a potential flame region is detected in the video frame based on the second flame detection model, obtaining the image to be detected based on the potential flame region in the video frame; wherein, the second flame detection model is trained based on a second sample image containing a flame region and the position of the flame region in the second sample image;

[0007] The step of obtaining multiple associated images located after the image to be detected from the video to be processed includes: determining multiple consecutive video frames in the video to be processed that are located after the video frame to which the image to be detected belongs; and extracting image regions at the same positions as the image to be detected from the multiple consecutive video frames to obtain multiple associated images of the image to be detected.

[0008] Optionally, for each video frame in the video to be processed, if a potential flame region is detected in the video frame based on the second flame detection model, and an image to be detected is obtained based on the potential flame region in the video frame, the process includes: for each video frame in the video to be processed, detecting the video frame based on the second flame detection model to obtain the potential flame region in the video frame; expanding the boundary of the potential flame region in the video frame according to preset width and height adjustment parameters to obtain the adjusted potential flame region, which is used as the image to be detected.

[0009] Secondly, to achieve the above objectives, embodiments of the present invention provide a flame detection model training method for generating a first flame detection model as described in any of the first aspects above. The method includes: acquiring a first sample image and a preset target type label for the first sample image; wherein the first sample image is obtained based on video frames in a sample video that detect potential flame regions; the target type label is either a type label indicating that the first sample image contains a flame region, or a type label indicating that the first sample image does not contain a flame region; acquiring multiple associated images located after the first sample image from the sample video; calculating the similarity between each associated image and the first sample image according to the order of the associated images in the sample video, to obtain a similarity sequence corresponding to the first sample image; inputting the similarity sequence corresponding to the first sample image into a first flame detection model with an initial structure to obtain a predicted type label for the potential flame region in the first sample image; and adjusting the model parameters of the first flame detection model with the initial structure based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image to obtain a trained first flame detection model.

[0010] Optionally, obtaining the first sample image and the preset target type label of the first sample image includes: obtaining a sample video; for each video frame in the sample video, if a potential flame region is detected in the video frame based on the second flame detection model, obtaining a first sample image based on the potential flame region in the video frame; wherein, the second flame detection model is trained based on a second sample image containing a flame region and the position of the flame region in the second sample image; obtaining the preset target type label of the first sample image;

[0011] The step of obtaining multiple associated images located after the first sample image from the sample video includes: for each first sample image, determining multiple consecutive video frames located after the video frame to which the first sample image belongs from the sample video; and extracting image regions at the same position as the first sample image from the multiple consecutive video frames to obtain multiple associated images of the first sample image.

[0012] Optionally, for each video frame in the sample video, if a potential flame region is detected in the video frame based on the second flame detection model, and a first sample image is obtained based on the potential flame region in the video frame, the process includes: for each video frame in the sample video, detecting the potential flame region in the video frame based on the second flame detection model; expanding the boundary of the potential flame region in the video frame according to preset width and height adjustment parameters to obtain an adjusted potential flame region, which is then used as the first sample image.

[0013] Optionally, after obtaining the preset target type label of the first sample image, the method further includes: dividing the first sample images with the same target type label into an image group; storing the first sample images belonging to the same image group and the associated images of the first sample images to the corresponding designated storage location;

[0014] Before calculating the similarity between each associated image and the first sample image according to the order of the associated images in the sample video to obtain the similarity sequence corresponding to the first sample image, the method further includes: obtaining the first sample image and the associated images of the first sample image from the specified storage location.

[0015] Optionally, the first flame detection model is a multilayer perceptron;

[0016] The step of inputting the similarity sequence corresponding to the first sample image into the first flame detection model of the initial structure to obtain a predicted type label indicating whether the first sample image contains a flame region includes: inputting the similarity sequence corresponding to the first sample image into the first hidden layer of the first flame detection model of the initial structure to obtain the first temporal feature of the first sample image output by the first hidden layer; inputting the first temporal feature of the first sample image into the second hidden layer of the first flame detection model of the initial structure to obtain the second temporal feature of the first sample image output by the second hidden layer; inputting the second temporal feature of the first sample image into the output layer of the first flame detection model of the initial structure to obtain the probability that the type label of the potential flame region in the first sample image is a preset type label; and determining the type label with the highest probability from the preset type labels as the predicted type label of the potential flame region in the first sample image.

[0017] Optionally, the step of calculating the similarity between each associated image and the first sample image according to the chronological order of the associated images in the sample video to obtain the similarity sequence corresponding to the first sample image includes: inputting the first sample image into a pre-trained feature extraction model to obtain a sample feature vector representing the image features of the first sample image; for each associated image, inputting the associated image into the feature extraction model to obtain an associated feature vector representing the image features of the associated image; and calculating the similarity between the associated feature vector of each associated image and the sample feature vector according to the chronological order of the associated images in the sample video to obtain the similarity sequence corresponding to the first sample image.

[0018] Optionally, adjusting the model parameters of the first flame detection model with the initial structure based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image to obtain a trained first flame detection model includes: calculating the function value of a preset loss function based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image; wherein the function value of the loss function represents the difference between the target type label and the predicted type label; and adjusting the model parameters of the first flame detection model with the initial structure based on the calculated function value of the loss function until a preset convergence condition is reached to obtain a trained first flame detection model.

[0019] Thirdly, to achieve the above objectives, embodiments of the present invention provide a flame detection device, the device comprising: a target image acquisition module, configured to acquire a target image; wherein the target image is obtained based on video frames containing potential flame regions detected in a video to be processed; an associated image acquisition module, configured to acquire multiple associated images located after the target image from the video to be processed; a similarity sequence determination module, configured to calculate the similarity between each associated image and the target image in sequence according to the order of each associated image in the video to be processed, thereby obtaining a similarity sequence corresponding to the target image; a probability prediction module, configured to input the similarity sequence corresponding to the target image into a pre-trained first flame detection model, thereby obtaining the probability that the type label representing the potential flame region in the target image is a preset type label; wherein the first flame detection model is trained based on the similarity sequence corresponding to a first sample image and the target type label of the first sample image; and a target flame region detection module, configured to determine that the target image contains a target flame region if the probability corresponding to the flame type label among the preset types is the highest.

[0020] Optionally, the image acquisition module is specifically used to acquire the video to be processed; for each video frame in the video to be processed, if a potential flame region is detected in the video frame based on the second flame detection model, an image to be detected is obtained based on the potential flame region in the video frame; wherein, the second flame detection model is trained based on a second sample image containing a flame region and the position of the flame region in the second sample image;

[0021] The associated image acquisition module is specifically used to determine a number of consecutive video frames in the video to be processed that are located after the video frame to which the image to be detected belongs; and to extract image regions at the same positions as the image to be detected from the number of consecutive video frames to obtain a number of associated images of the image to be detected.

[0022] Optionally, the image acquisition module is specifically used to detect each video frame in the video to be processed based on the second flame detection model to obtain the potential flame region in the video frame; and to expand the boundary of the potential flame region in the video frame according to preset width and height adjustment parameters to obtain the adjusted potential flame region as the image to be detected.

[0023] Fourthly, to achieve the above objectives, embodiments of the present invention provide a flame detection model training apparatus for generating a first flame detection model as described in any of the third aspects above, the apparatus comprising:

[0024] A first sample image acquisition module is used to acquire a first sample image and a preset target type label for the first sample image; wherein, the first sample image is obtained based on a video frame containing a potential flame region detected in a sample video; the target type label is either a type label indicating that the first sample image contains a flame region, or a type label indicating that the first sample image does not contain a flame region; an associated image determination module is used to acquire multiple associated images located after the first sample image from the sample video; a similarity sequence determination module is used to calculate the similarity between each associated image and the first sample image according to the order of each associated image in the sample video, thereby obtaining a similarity sequence corresponding to the first sample image; a predicted type label determination module is used to input the similarity sequence corresponding to the first sample image into a first flame detection model of an initial structure, thereby obtaining a predicted type label for the potential flame region in the first sample image; a model parameter adjustment module is used to adjust the model parameters of the first flame detection model of the initial structure based on the target type label of the first sample image and the predicted type label for the potential flame region in the first sample image, thereby obtaining a trained first flame detection model.

[0025] Optionally, the first sample image acquisition module is specifically used to acquire sample videos; for each video frame in the sample video, if a potential flame region is detected in the video frame based on the second flame detection model, a first sample image is obtained based on the potential flame region in the video frame; wherein, the second flame detection model is trained based on the second sample image containing the flame region and the position of the flame region in the second sample image; and a preset target type label for the first sample image is acquired.

[0026] The associated image determination module is specifically used to, for each first sample image, determine, from the sample video, a plurality of consecutive video frames located after the video frame to which the first sample image belongs; and extract, from the plurality of consecutive video frames, an image region at the same position as the first sample image, to obtain a plurality of associated images of the first sample image.

[0027] Optionally, the first sample image acquisition module is specifically used to detect each video frame in the sample video based on the second flame detection model to obtain the potential flame region in the video frame; and to expand the boundary of the potential flame region in the video frame according to preset width and height adjustment parameters to obtain the adjusted potential flame region as the first sample image.

[0028] Optionally, the device further includes: an image group division module, configured to divide first sample images with the same target type label into an image group after the first sample image acquisition module acquires the preset target type label of the first sample image; and an image storage module, configured to store the first sample images belonging to the same image group and the associated images of the first sample images to the corresponding designated storage location.

[0029] The device further includes: an associated image acquisition module, configured to acquire the first sample image and its associated images from the specified storage location before the similarity sequence determination module performs the calculation of the similarity between each associated image and the first sample image according to the order of each associated image in the sample video, and obtains the similarity sequence corresponding to the first sample image.

[0030] Optionally, the first flame detection model is a multilayer perceptron;

[0031] The prediction type label determination module is specifically used to input the similarity sequence corresponding to the first sample image into the first hidden layer of the first flame detection model of the initial structure to obtain the first temporal feature of the first sample image output by the first hidden layer; input the first temporal feature of the first sample image into the second hidden layer of the first flame detection model of the initial structure to obtain the second temporal feature of the first sample image output by the second hidden layer; input the second temporal feature of the first sample image into the output layer of the first flame detection model of the initial structure to obtain the probability that the type label of the potential flame region in the first sample image is a preset type label; and determine the type label with the highest probability from the preset type labels as the prediction type label of the potential flame region in the first sample image.

[0032] Optionally, the similarity sequence determination module is specifically used to input the first sample image into a pre-trained feature extraction model to obtain a sample feature vector representing the image features of the first sample image; for each associated image, input the associated image into the feature extraction model to obtain an associated feature vector representing the image features of the associated image; and calculate the similarity between the associated feature vector of each associated image and the sample feature vector in sequence according to the order of each associated image in the sample video to obtain the similarity sequence corresponding to the first sample image.

[0033] Optionally, the model parameter adjustment module is specifically used to calculate the function value of a preset loss function based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image; wherein, the function value of the loss function represents the difference between the target type label and the predicted type label; based on the calculated function value of the loss function, the model parameters of the first flame detection model of the initial structure are adjusted until a preset convergence condition is reached to obtain the trained first flame detection model.

[0034] This invention also provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus; the memory is used to store computer programs; and the processor, when executing the program stored in the memory, implements the flame detection method steps described in any of the first aspects above, or the flame detection model training method steps described in any of the second aspects above.

[0035] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the flame detection method steps described in any of the first aspects above, or the flame detection model training method steps described in any of the second aspects above.

[0036] This invention also provides a computer program product containing instructions that, when run on a computer, cause the computer to execute the flame detection method described in any of the first aspects above, or the flame detection model training method described in any of the second aspects above.

[0037] Beneficial effects of the embodiments of the present invention:

[0038] This invention provides a flame detection method, which involves acquiring a target image, obtained from video frames containing potential flame regions detected in a video to be processed; acquiring multiple associated images following the target image from the video to be processed; calculating the similarity between each associated image and the target image according to their order in the video to be processed, thus obtaining a similarity sequence corresponding to the target image; inputting the similarity sequence corresponding to the target image into a pre-trained first flame detection model to obtain the probability that the type label representing the potential flame region in the target image is a preset type label; the first flame detection model is trained based on the similarity sequence corresponding to a first sample image and the target type label of the first sample image; if the probability corresponding to the flame type label among the preset types is the highest, it is determined that the target image contains a target flame region.

[0039] Based on the above processing, the similarity sequence corresponding to the image to be detected includes the similarity between the image to be detected and multiple related images in the video following the image to be detected. Therefore, the similarity sequence corresponding to the image to be detected can represent the temporal features of the potential flame region in the image to be detected. Since the shape changes of a burning flame are irregular, the temporal features of the flame region in the image are also irregular. However, the shape changes of other brightness regions (e.g., flickering light regions) are regular, so the temporal features of other brightness regions in the image are regular and exhibit periodic changes. Therefore, the temporal features of the flame region in the image differ from those of other brightness regions. Consequently, flame detection based on the first flame detection model and the similarity sequence corresponding to the image to be detected can avoid misidentifying light regions as flame regions, reducing the false alarm rate of flame detection and improving the accuracy of flame detection.

[0040] Of course, implementing any product or method of the present invention does not necessarily require achieving all of the advantages described above at the same time. Attached Figure Description

[0041] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings.

[0042] Figure 1 A flowchart of a flame detection method provided in an embodiment of the present invention;

[0043] Figure 2 A flowchart of another flame detection method provided in an embodiment of the present invention;

[0044] Figure 3 A schematic diagram illustrating the principle of flame detection provided in an embodiment of the present invention;

[0045] Figure 4 A schematic diagram of the first flame detection model provided in an embodiment of the present invention;

[0046] Figure 5 A flowchart of a flame detection model training method provided in an embodiment of the present invention;

[0047] Figure 6 A flowchart of another flame detection model training method provided in an embodiment of the present invention;

[0048] Figure 7a A graph showing the first type of flame region and the corresponding similarity sequence provided in the embodiments of the present invention;

[0049] Figure 7bA graph showing the second type of flame region and the corresponding similarity sequence provided in the embodiments of the present invention;

[0050] Figure 7c A graph showing the third type of flame region and its corresponding similarity sequence provided in the embodiments of the present invention;

[0051] Figure 7d A graph showing the fourth type of flame region and its corresponding similarity sequence provided in this embodiment of the invention;

[0052] Figure 7e A graph showing the fifth type of flame region and its corresponding similarity sequence provided in this embodiment of the invention;

[0053] Figure 7f A graph showing the sixth type of flame region and its corresponding similarity sequence provided in this embodiment of the invention;

[0054] Figure 7g A graph showing the seventh type of flame region and its corresponding similarity sequence provided in this embodiment of the invention;

[0055] Figure 7h A graph showing the eighth type of flame region and its corresponding similarity sequence provided in this embodiment of the invention;

[0056] Figure 7i A graph showing the ninth type of flame region and its corresponding similarity sequence provided in this embodiment of the invention;

[0057] Figure 8a A graph showing the first type of light area and the corresponding similarity sequence provided in the embodiments of the present invention;

[0058] Figure 8b A graph showing the second type of lighting area and the corresponding similarity sequence provided in an embodiment of the present invention;

[0059] Figure 8c A graph showing the third type of lighting region and its corresponding similarity sequence provided in an embodiment of the present invention;

[0060] Figure 8d A graph showing the fourth type of lighting region and its corresponding similarity sequence provided in this embodiment of the invention;

[0061] Figure 8e A graph showing the fifth type of lighting region and its corresponding similarity sequence provided in this embodiment of the invention;

[0062] Figure 8f A graph showing the sixth type of lighting region and its corresponding similarity sequence provided in this embodiment of the invention;

[0063] Figure 8gA graph showing the seventh type of lighting region and its corresponding similarity sequence provided in this embodiment of the invention;

[0064] Figure 8h A graph showing the eighth type of lighting region and its corresponding similarity sequence provided in this embodiment of the invention;

[0065] Figure 8i A graph showing the ninth type of lighting region and its corresponding similarity sequence provided in this embodiment of the invention;

[0066] Figure 9a A graph showing the first type of other target region and the corresponding similarity sequence provided in the embodiments of the present invention;

[0067] Figure 9b A graph showing a second type of target region and its corresponding similarity sequence provided in an embodiment of the present invention;

[0068] Figure 9c A graph showing a third type of target region and its corresponding similarity sequence provided in this embodiment of the invention;

[0069] Figure 9d A graph showing a fourth type of target region and its corresponding similarity sequence provided in this embodiment of the invention;

[0070] Figure 9e A graph showing the fifth type of target region and its corresponding similarity sequence provided in this embodiment of the invention;

[0071] Figure 10 The first sample image and associated image provided in the embodiments of the present invention;

[0072] Figure 11 A structural diagram of a flame detection device provided in an embodiment of the present invention.

[0073] Figure 12 This is a structural diagram of a flame detection model training device provided in an embodiment of the present invention.

[0074] Figure 13 This is a structural diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0075] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art based on the present invention are within the scope of protection of the present invention.

[0076] In related technologies, video flame detection algorithms have poor anti-interference capabilities for bright objects or targets with similar colors, which easily leads to false alarms and results in low accuracy in flame area detection.

[0077] To address the aforementioned problems, this invention provides a flame detection method applied to an electronic device, which can be a terminal or a server. The electronic device acquires a target image based on video frames containing flame regions detected in a video to be processed; it then acquires multiple associated images following the target image from the video to be processed; and, according to the order of these associated images in the video to be processed, it sequentially calculates the similarity between each associated image and the target image to obtain a similarity sequence corresponding to the target image. Furthermore, the similarity sequence corresponding to the target image is input into a pre-trained first flame detection model to obtain the probability that the type label representing a potential flame region in the target image matches a preset type label. If the probability corresponding to the flame type label among the preset types is the highest, it is determined that the target image contains a target flame region, thus improving the accuracy of flame detection. Subsequently, when it is determined that the target image contains a target flame region, an alarm message can be output.

[0078] See Figure 1 , Figure 1 A flowchart of a flame detection method provided in an embodiment of the present invention, the method comprising the following steps:

[0079] S101: Acquire the image to be detected.

[0080] The image to be detected is obtained based on video frames containing potential flame areas detected in the video to be processed.

[0081] S102: Obtain multiple associated images located after the image to be detected from the video to be processed.

[0082] S103: Calculate the similarity between each associated image and the image to be detected in the order of their positions in the video to be processed, and obtain the similarity sequence corresponding to the image to be detected.

[0083] S104: Input the similarity sequence corresponding to the image to be detected into the pre-trained first flame detection model to obtain the probability that the type label of the potential flame region in the image to be detected is a preset type label.

[0084] The first flame detection model is trained based on the similarity sequence corresponding to the first sample image and the target type label of the first sample image.

[0085] S105: If the probability of the flame type label in the preset types is the highest, it is determined that the image to be detected contains the target flame region.

[0086] Based on the flame detection method provided in this embodiment of the invention, the similarity sequence corresponding to the image to be detected includes the similarity between the image to be detected and multiple related images located after the image to be detected in the video to be processed. Therefore, the similarity sequence corresponding to the image to be detected can represent the temporal characteristics of the potential flame region in the image to be detected. Since the shape changes during flame combustion are irregular, the temporal characteristics of the flame region in the image are also irregular. However, the shape changes of other brightness regions (e.g., flashing light regions) are regular, so the temporal characteristics of the flame region in the image are regular and exhibit periodic changes. Therefore, the temporal characteristics of the flame region in the image differ from those of other brightness regions. Consequently, by performing flame detection based on the first flame detection model and the similarity sequence corresponding to the image to be detected, it is possible to avoid identifying light regions as flame regions, reduce the false alarm rate of flame detection, and improve the accuracy of flame region detection.

[0087] For step S101, the video to be processed is the video of the target scene that needs to be detected for fire. The video frames in the video to be processed that contain potential flame areas are obtained by flame detection based on the second flame detection model.

[0088] In some embodiments, Figure 1 Based on this, see Figure 2 Step S101 may include the following steps:

[0089] S1011: Obtain the video to be processed.

[0090] S1012: For each video frame in the video to be processed, if a potential flame region is detected in the video frame based on the second flame detection model, the image to be detected is obtained based on the potential flame region in the video frame.

[0091] The second flame detection model is trained based on a second sample image containing a flame region and the location of the flame region within that second sample image. This second flame detection model can be a target detection model, such as Mask-RCNN (Mask Region-based Convolutional Neural Network) or RCNN (Region-based Convolutional Neural Networks).

[0092] The electronic device acquires a second sample image containing the flame region, and the location of the flame region in the second sample image (which can be called the target location). The electronic device inputs the second sample image into the second flame detection model with the initial structure, and obtains the location of the flame region in the second sample image output by the second flame detection model with the initial structure (which can be called the predicted location). Based on the target location and the predicted location, the model parameters of the second flame detection model with the initial structure are adjusted to obtain the trained second flame detection model.

[0093] After acquiring the video to be processed, for each video frame in the video to be processed, flame detection is performed on the video frame based on the second flame detection model to obtain the location of the potential flame region in the video frame output by the second flame detection model.

[0094] In one implementation, for each video frame in the video to be processed, if a potential flame area is detected in the video, then the video frame is determined to be the image to be detected.

[0095] In another implementation, the electronic device can directly extract the potential flame region from the video frame according to the position of the potential flame region output by the second flame detection model to obtain the image to be detected. Alternatively, step S1012 may include the following steps: for each video frame in the video to be processed, detect the video frame based on the second flame detection model to obtain the potential flame region in the video frame; expand the boundary of the potential flame region in the video frame according to preset width and height adjustment parameters to obtain the adjusted potential flame region as the image to be detected.

[0096] The preset width and height adjustment parameters are the same as those used to adjust the potential flame region in the sample video. The preset width and height adjustment parameters include: a preset multiple corresponding to the width of the potential flame region and a preset multiple corresponding to the height of the potential flame region. The preset multiple corresponding to the width of the potential flame region can be the same as the preset multiple corresponding to the height of the potential flame region. For example, the preset multiple can range from (2, 2.5). The preset multiple corresponding to the width of the potential flame region can also be different from the preset multiple corresponding to the height of the potential flame region.

[0097] For each video frame in the video to be processed, after identifying the potential flame region in that frame, the electronic device can move the boundary along the width direction of the potential flame region by a certain number of pixels, so that the width of the adjusted potential flame region is a preset multiple of the width of the original potential flame region. It can also move the boundary along the height direction of the potential flame region by a certain number of pixels, so that the height of the adjusted potential flame region is a preset multiple of the height of the original potential flame region, thus obtaining the adjusted potential flame region. Then, the adjusted potential flame region is extracted from the video frame to obtain the image to be detected.

[0098] Alternatively, after identifying the potential flame region in the video frame, if the boundaries of the potential flame region are not equal, the electronic device can shift the other boundaries by a certain number of pixels according to the longest boundary of the potential flame region, making the boundaries of the potential flame region equal, thus obtaining a square potential flame region. Then, the electronic device can shift the boundary along the width direction of the square potential flame region by a certain number of pixels, so that the width of the adjusted potential flame region is a preset multiple of the width of the original potential flame region. And it can shift the boundary along the height direction of the square potential flame region by a certain number of pixels, so that the height of the adjusted potential flame region is a preset multiple of the height of the original potential flame region, thus obtaining the adjusted potential flame region. Finally, the adjusted potential flame region is extracted from the video frame to obtain the image to be detected.

[0099] Based on the above processing, the extracted image to be detected contains background information and the foreground information is not too small, which can improve the accuracy of image features extracted from the subsequent image to be detected, and further improve the accuracy of flame detection.

[0100] Regarding step S102, since the second flame detection model has low accuracy in detecting flames in each video frame of the video to be processed—meaning that the potential flame area detected by the second flame detection model may not actually be a flame area, for example, misdetecting a light area as a flame area—the electronic device can perform flame detection again on the image to be detected based on video frames containing potential flame areas detected in the video to be processed.

[0101] Because the shape changes irregularly during flame combustion, the temporal characteristics of the flame region in the image are also irregular. For example, 50 video frames are extracted from the video to be processed; see [link to relevant documentation]. Figure 3 , Figure 3 Only three frames of the image are shown in the image. Figure 3The first row of images, from left to right, shows multiple video frames extracted from the video to be processed, all of which contain flame regions. The second row of images shows multiple flame regions extracted from these video frames. The flame region extracted from the first video frame is the image to be detected, while the flame regions extracted from the other video frames are associated images.

[0102] Then, using a shapenet (i.e., the feature extraction model in subsequent embodiments), features are extracted from multiple flame regions to obtain feature vectors representing the image features of each flame region, resulting in 50 feature vectors. The cosine distance (i.e., similarity) between the feature vector of the flame region extracted from the second video frame (i.e., the first associated image) and the feature vector of the flame region extracted from the first video frame (i.e., the image to be detected) is calculated. The cosine distance between the feature vector of the flame region extracted from the third video frame (i.e., the second associated image) and the feature vector of the flame region extracted from the first video frame is calculated. The cosine distance between the feature vector of the flame region extracted from the fourth video frame (i.e., the third associated image) and the feature vector of the flame region extracted from the first video frame is calculated, and so on, until the cosine distance between the feature vector of the flame region extracted from the 50th video frame and the feature vector of the flame region extracted from the first video frame is calculated, thus obtaining the similarity sequence corresponding to the image to be detected.

[0103] Figure 3 The image shown in row 4 is a curve graph of the similarity sequence corresponding to the image to be detected. In this curve graph, the horizontal axis represents the sequence number of the associated image, and the vertical axis represents the similarity between the associated image and the image to be detected. It can be seen from this curve graph that the temporal characteristics of the flame region are irregular.

[0104] While the shape changes of other brightness regions (e.g., flickering light regions) are regular, the temporal characteristics of these other brightness regions in the image are regular and exhibit periodic variations. The temporal characteristics of flame regions in the image differ from those of other brightness regions. To obtain the temporal characteristics of potential flame regions in the image to be detected, the electronic device acquires multiple associated images located after the image to be detected from the video to be processed.

[0105] In one implementation, when the image to be detected is a video frame in a video to be processed, the electronic device can determine multiple consecutive video frames located after the image to be detected from the video to be processed, and obtain the associated image of the image to be detected.

[0106] In another implementation, Figure 1 Based on this, see Figure 2Step S102 may include the following steps:

[0107] S1021: Determine multiple consecutive video frames in the video to be processed that are located after the video frame to which the image to be detected belongs.

[0108] S1022: Extract the image region at the same position as the image to be detected from the multiple consecutive video frames to obtain multiple associated images of the image to be detected.

[0109] The image to be detected is obtained from video frames containing potential flame regions within the video to be processed. The electronic device identifies multiple consecutive video frames in the video to be processed that are located after the video frame to which the image to be detected belongs. Then, the electronic device can extract image regions at the same locations as the image to be detected from these identified consecutive video frames, thus obtaining multiple associated images of the image to be detected.

[0110] The number of consecutive video frames is the same as the number of consecutive video frames determined from the sample videos when training the first flame detection model. For example, the number of consecutive video frames can be 25, or it can be 50. This allows the associated images of the image to be detected to occupy less memory on the electronic device, while ensuring that effective temporal features can be extracted subsequently.

[0111] Based on the above processing, the associated images of the image to be detected can be determined. Subsequently, based on the associated images of the image to be detected, the temporal features of the potential flame regions in the image to be detected can be determined. Then, flame detection can be performed based on the temporal features of the potential flame regions in the image to be detected, which can improve the accuracy of flame detection.

[0112] In step S103, the electronic device inputs the image to be detected into a pre-trained feature extraction model to obtain a feature vector (which can be called the first feature vector) representing the image features of the image to be detected. For each associated image of the image to be detected, the associated image is input into the feature extraction model to obtain a feature vector (which can be called the second feature vector) representing the image features of the associated image. Then, according to the order of the associated images in the video to be processed, based on a preset similarity algorithm, the similarity between the second feature vector of each associated image and the first feature vector of the image to be detected is calculated sequentially to obtain a similarity sequence corresponding to the image to be detected.

[0113] The feature extraction model is used to extract the shape features of potential flame regions in the image to be detected. The feature extraction model can be a CNN (Convolutional Neural Network), or it can be MobileNetv2 (a lightweight convolutional neural network), etc. The preset similarity algorithm can be cosine distance, or it can be Euclidean distance, etc., and no specific limitation is made in this embodiment of the invention.

[0114] Regarding steps S104 and S105, in some embodiments, the first flame detection model is an MLP (Multilayer Perceptron). The first flame detection model includes an input layer, a first hidden layer, a second hidden layer, and an output layer.

[0115] For example, see Figure 4 The input signal is the similarity sequence corresponding to the image to be detected. The electronic device inputs the similarity sequence of the image to be detected into the First hidden layer via the Input layer. The First hidden layer extracts features from the similarity sequence, obtaining the temporal features of the image to be detected output by the First hidden layer. The temporal features of the image to be detected output by the First hidden layer are then input into the Second hidden layer. The Second hidden layer extracts features from the temporal features of the image to be detected output by the Second hidden layer, obtaining the temporal features of the image to be detected output by the Second hidden layer. The temporal features of the image to be detected output by the Second hidden layer are then input into the Output layer. The Output layer normalizes the temporal features of the image to be detected output by the Output layer, obtaining the Output signal. The Output signal represents the probability that the type label of the potential flame region in the image to be detected corresponds to a preset type label.

[0116] If the probability corresponding to the flame type label is greater than the probabilities corresponding to other type labels (meaning the probability of the flame type label is the highest), then the image to be detected is determined to contain a target flame region. If the probability corresponding to the flame type label is not greater than the probabilities corresponding to other type labels (meaning the probability of the flame type label is not the highest), then the image to be detected is determined not to contain a target flame region.

[0117] For example, the preset labels include: flame type label, light type label, and other type labels. Correspondingly, the first flame detection model outputs a vector containing three elements. For example, the vector output by the first flame detection model might be [0.6, 0.1, 0.3]. If the probability corresponding to the flame type label is the highest, then the image to be detected is determined to contain a target flame region. If the vector output by the first flame detection model is [0.2, 0.7, 0.1], and the probability corresponding to the light type label is the highest, then the image to be detected is determined to contain a light region, meaning the image to be detected does not contain a target flame region.

[0118] Furthermore, when the image to be detected contains a target flame area, the electronic device can output an alarm message indicating that a fire has occurred in the target scene, thereby improving the safety of the target scene.

[0119] See Figure 5 , Figure 5 A flowchart of a flame detection model training method provided in an embodiment of the present invention is provided. This method is used to generate any of the first flame detection models described in the foregoing embodiments. The method may include the following steps:

[0120] S501: Obtain the first sample image and the target type label of the preset first sample image.

[0121] The first sample image is obtained based on video frames in the sample video that detect potential flame regions; the target type label is either a type label indicating that the first sample image contains a flame region, or a type label indicating that the first sample image does not contain a flame region.

[0122] S502: Obtain multiple associated images located after the first sample image from the sample video.

[0123] S503: Calculate the similarity between each associated image and the first sample image in the order of their appearance in the sample video, and obtain the similarity sequence corresponding to the first sample image.

[0124] S504: Input the similarity sequence corresponding to the first sample image into the first flame detection model of the initial structure to obtain the predicted type label of the potential flame region in the first sample image.

[0125] S505: Based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image, the model parameters of the first flame detection model with the initial structure are adjusted to obtain the trained first flame detection model.

[0126] Based on the flame detection model training method provided in this embodiment of the invention, a first flame detection model is trained based on the similarity sequence and target type label of a first sample image. Subsequently, flame detection is performed based on the similarity sequence corresponding to the trained first flame detection model and the image to be detected. This avoids misidentifying light areas as flame areas, reduces the false alarm rate of flame detection, and improves the accuracy of flame detection.

[0127] For step S501, the sample videos include: sample videos containing flame regions, sample videos containing light regions, and sample videos containing other target regions that are easily misdetected as flame regions. The video frames in the sample videos that detect potential flame regions are obtained through flame detection based on the second flame detection model.

[0128] The target type label of the first sample image is pre-labeled by a technician. If the first sample image contains a flame region, then the target type label of the first sample image is a flame type label, for example, 1 represents the flame type label. If the first sample image contains a light region, then the target type label of the first sample image is a light type label, for example, 2 represents the light type label. If the first sample image contains other target regions that are easily misidentified as flame regions, then the target type label of the first sample image is an other type label, for example, 3 represents the other type label.

[0129] In some embodiments, Figure 5 Based on this, see Figure 6 Step S501 may include the following steps:

[0130] S5011: Obtain sample video.

[0131] S5012: For each video frame in the sample video, if a potential flame region is detected in the video frame based on the second flame detection model, a first sample image is obtained based on the potential flame region in the video frame.

[0132] The second flame detection model is trained based on a second sample image containing a flame region and the position of the flame region in the second sample image.

[0133] S5013: Obtain the target type label of the preset first sample image.

[0134] After acquiring the sample video, for each video frame in the sample video, flame detection is performed on that video frame based on the second flame detection model to obtain the location of the potential flame region in that video frame, as output by the second flame detection model. The second flame detection model can be referred to the relevant description in the foregoing embodiments.

[0135] In one implementation, for each video frame in a sample video, if a potential flame region is detected in the video, then the video frame is determined to be the image to be detected.

[0136] In another implementation, the electronic device can directly extract the potential flame region from the video frame according to the location of the potential flame region output by the second flame detection model to obtain the first sample image. Alternatively, step S5012 may include the following steps: for each video frame in the sample video, detect the potential flame region in the video frame based on the second flame detection model; expand the boundary of the potential flame region in the video frame according to preset width and height adjustment parameters to obtain the adjusted potential flame region, which serves as the first sample image. The preset width and height adjustment parameters can be found in the description of the foregoing embodiments.

[0137] For each video frame in the sample video, after identifying the potential flame region within that frame, the electronic device can move the boundary along the width direction of the potential flame region by a certain number of pixels, such that the width of the adjusted potential flame region is a preset multiple of the width of the original potential flame region. It can also move the boundary along the height direction of the potential flame region by a certain number of pixels, such that the height of the adjusted potential flame region is a preset multiple of the height of the original potential flame region, thus obtaining the adjusted potential flame region. Furthermore, the adjusted potential flame region is extracted from the video frame to obtain the first sample image.

[0138] Alternatively, after identifying the potential flame region in the video frame, if the boundaries of the potential flame region are not equal, the electronic device can shift the other boundaries by a certain number of pixels according to the longest boundary of the potential flame region, making the boundaries of the potential flame region equal, thus obtaining a square potential flame region. Then, the electronic device can shift the boundary along the width direction of the square potential flame region by a certain number of pixels, so that the width of the adjusted potential flame region is a preset multiple of the width of the original potential flame region. And it can shift the boundary along the height direction of the square potential flame region by a certain number of pixels, so that the height of the adjusted potential flame region is a preset multiple of the height of the original potential flame region, thus obtaining the adjusted potential flame region. Then, the adjusted potential flame region is extracted from the video frame to obtain the first sample image.

[0139] Based on the above processing, the extracted first sample image contains background information and the foreground information is not too small, which can improve the accuracy of the image features of the subsequently extracted first sample image and further improve the accuracy of the trained first flame detection model.

[0140] Regarding step S502, since the shape changes during flame combustion are irregular, the temporal characteristics of the flame region in the image are also irregular. For example, see... Figures 7a to 7i , Figures 7a to 7i The left side of the graph shows the first sample image containing the flame region, and the right side shows a curve graph of the similarity sequence corresponding to the first sample image. In this curve graph, the horizontal axis represents the sequence number of the associated image, and the vertical axis represents the similarity between the associated image and the first sample image. Figures 7a to 7i The curves in the graph show that the temporal characteristics of the flame region are irregular.

[0141] If the shape changes of other brightness areas (e.g., areas of flickering light) are regular, then the temporal characteristics of these other brightness areas in the image are regular and exhibit periodic variations. For example, see [link to example]. Figures 8a to 8i , Figures 8a to 8i The left side of the image shows the first sample image containing the illuminated area, and the right side shows a curve graph of the similarity sequence corresponding to the first sample image. In this curve graph, the horizontal axis represents the sequence number of the associated image, and the vertical axis represents the similarity between the associated image and the first sample image. Figures 8a to 8i The curves in the graph show that the temporal characteristics of the light area are regular and exhibit periodic changes.

[0142] See Figures 9a to 9e , Figures 9a to 9e The left side of the image shows the first sample image containing other target regions that are easily misidentified as flame areas. The right side is a curve graph of the similarity sequence corresponding to the first sample image. In this curve graph, the horizontal axis represents the sequence number of the associated image, and the vertical axis represents the similarity between the associated image and the first sample image. Figures 9a to 9e The curve in the graph can be seen Figures 9a to 9e The temporal characteristics are regular and differ from the temporal characteristics of the flame region.

[0143] Therefore, the temporal characteristics of the flame region in the image differ from those of other brightness regions. To obtain the temporal characteristics of the potential flame region in the first sample image, the electronic device can acquire multiple associated images located after the first sample image from the sample video.

[0144] In one implementation, when the first sample image is a video frame in a sample video, the electronic device can determine multiple consecutive video frames located after the first sample image from the sample video to obtain the associated image of the first sample image.

[0145] In another implementation, Figure 5 Based on this, see Figure 6 Step S502 may include the following steps:

[0146] S5021: For each first sample image, determine from the sample video a plurality of consecutive video frames located after the video frame to which the first sample image belongs.

[0147] S5022: Extract the image region at the same position as the first sample image from the multiple consecutive video frames to obtain multiple associated images of the first sample image.

[0148] The first sample image is obtained from video frames containing potential flame regions in the sample video. The electronic device determines a series of consecutive video frames in the sample video that follow the first sample image. Then, the electronic device can extract image regions at the same locations as the first sample image from these determined consecutive video frames to obtain multiple associated images of the first sample image. The number of consecutive video frames can be found in the description of the foregoing embodiments.

[0149] Based on the above processing, the associated images of the first sample image can be determined. Subsequently, based on the associated images of the first sample image, the temporal features of the potential flame region in the first sample image are determined. Then, the first flame detection model is trained based on the temporal features of the potential flame region in the first sample image, which can improve the accuracy of the trained first flame detection model.

[0150] In some embodiments, step S503 may include the following steps: inputting a first sample image into a pre-trained feature extraction model to obtain a sample feature vector representing the image features of the first sample image; for each associated image, inputting the associated image into the feature extraction model to obtain an associated feature vector representing the image features of the associated image; and calculating the similarity between the associated feature vector and the sample feature vector of each associated image in the order of their positions in the sample video to obtain a similarity sequence corresponding to the first sample image.

[0151] After extracting the sample feature vector representing the image features of the first sample image and the associated feature vector representing the image features of the associated images, the electronic device calculates the similarity between the associated feature vector of each associated image and the sample feature vector of the first sample image according to the order of the associated images in the sample video, based on a preset similarity algorithm, to obtain the similarity sequence corresponding to the first sample image. The preset similarity algorithm and feature extraction model can be found in the foregoing description of the embodiments.

[0152] In some embodiments, after step S5013, the method may further include the following steps: dividing first sample images with the same target type label into an image group; storing the first sample images belonging to the same image group and the associated images of the first sample images to the corresponding designated storage location.

[0153] Accordingly, before step S503, the method may further include the following steps: obtaining a first sample image from a specified storage location, and an associated image of the first sample image.

[0154] After acquiring the first sample image and the target type label, the electronic device can group the first sample images with the same target type label into one image group. For example, if the target type labels include: flame type label, light type label, and other type labels, then the first sample images with the flame type label are grouped into one image group, the first sample images with the light type label are grouped into another image group, and the first sample images with the other type labels are grouped into another image group, resulting in three image groups.

[0155] The first sample image belonging to the same image group and its associated image are stored in the corresponding designated storage location. For example, target type labels include: flame type label, light type label, and other type labels. After dividing the first sample images into three image groups according to the target type labels, the first sample images in the three image groups are stored in different directories of the electronic device. For example, directory 1 stores the first sample image and associated image of the flame type label, see [link to documentation]. Figure 10 , Figure 10 The first sample image and associated image of the flame type label stored in directory 1. Figure 10 The image in the first row and first column is the first sample image, and the other images are associated images of the first sample image. Directory 2 stores the first sample image and associated images of the light type labels. Directory 3 stores the first sample image and associated images of other type labels.

[0156] Furthermore, when training the first flame detection model, the electronic device can obtain the first sample image and the associated image of the first sample image from a designated storage location.

[0157] The following example illustrates how to obtain the similarity sequence corresponding to the first sample image.

[0158] (1) Obtain video files of multiple sample videos. Open the video file of each sample video, extract video frames (denoted as video frame f) from the sample video, and denote the frame number of video frame f as fidx.

[0159] (2) Use image detector D (i.e., the second flame detection model) to perform flame detection on video frame f, output the bounding boxes of potential flame regions in video frame f, and extract the potential flame regions in video frame f according to the output bounding boxes. Multiple potential flame regions corresponding to video frame f are obtained, and these multiple potential flame regions constitute a list R.

[0160] (3) For each potential flame region (denoted as potential flame region r) in list R, perform the following operation:

[0161] A. Expand the potential flame area r according to the specified width and height expansion ratio (i.e., the preset width and height adjustment parameters) to obtain the expanded rectangle er (i.e. the first sample image).

[0162] B. Starting from video frame f with sequence number fidx (including the fidx frame), extract the image region at the same position as the extended rectangle er in N consecutive video frames to form multiple sub-images (i.e., associated images). These multiple sub-images form a sub-image sequence C in the order of their positions in the sample video. Add the N sub-images in the sub-image sequence C to the training sub-image set S.

[0163] C. Store the N subgraphs in the subgraph sequence C to different directories, and set the type label of the directory to the type label of the subgraph sequence C.

[0164] (4) For a training subgraph set S containing multiple subgraph sequences C, when N is 50, each subgraph sequence C contains 50 subgraphs, denoted as c1, c2, ..., c50. Perform the following operations on each subgraph sequence C:

[0165] A. Use a shape feature network (i.e., feature extraction model) to extract the shape features of all images in the sub-image sequence C, and denote them as feat(feature)1, feat2, ..., feat50 respectively.

[0166] B. Calculate the cosine distance between feat2 and feat1, the cosine distance between feat3 and feat1, and so on, until the cosine distance between feat50 and feat1 is calculated. Add the calculated cosine distances to the similarity queue l in order.

[0167] C. Determine the target type label of the similarity queue l as the type label of the directory to which the corresponding subgraph sequence C belongs. That is, the flame type label, the light type label, and other type labels.

[0168] D. Add the similarity queue l to the dataset L. The data in dataset L is in the form of (cosine_dis_list, cls). cosine_dis_list is a one-dimensional array, which is the similarity queue l, and cls is the type label of this one-dimensional array.

[0169] Regarding steps S504 and S505, in some embodiments, the first flame detection model is a multilayer perceptron. The first flame detection model includes an input layer, a first hidden layer, a second hidden layer, and an output layer.

[0170] The similarity sequence corresponding to the first sample image is input into the first hidden layer of the initial structure of the first flame detection model. Feature extraction is performed on the similarity sequence of the first sample image through the first hidden layer, yielding the first temporal feature of the first sample image output by the first hidden layer. The first temporal feature of the first sample image is then input into the second hidden layer of the initial structure of the first flame detection model. Feature extraction is performed on the first temporal feature through the second hidden layer, yielding the second temporal feature of the first sample image output by the second hidden layer. The second temporal feature of the first sample image is then input into the output layer of the initial structure of the first flame detection model. The output layer normalizes the second temporal feature to obtain the probability that the type label of the potential flame region in the first sample image corresponds to each preset type label. From the preset type labels, the type label with the highest probability is determined as the predicted type label of the potential flame region in the first sample image.

[0171] For example, the preset labels include: flame type label, light type label, and other type labels. Correspondingly, the first flame detection model outputs a vector containing three elements. If the vector output by the first flame detection model is [0.7, 0.2, 0.1], then the predicted type label for the potential flame region in the first sample image is determined to be the flame type label. If the vector output by the first flame detection model is [0.2, 0.7, 0.1], then the predicted type label for the potential flame region in the first sample image is determined to be the light type label. If the vector output by the first flame detection model is [0.1, 0.1, 0.8], then the predicted type label for the potential flame region in the first sample image is determined to be other type labels.

[0172] If the predicted type label of the potential flame region in the first sample image is a flame type label, it means that the first sample image contains a flame region. If the predicted type label of the potential flame region in the first sample image is a light type label or another type label, it means that the first sample image does not contain a flame region, that is, the potential flame region in the first sample image is another target region that is falsely detected by the second flame detection model.

[0173] Based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image, the function value of a preset loss function is calculated. The calculated function value represents the difference between the target type label and the predicted type label. Then, based on the calculated function value of the loss function, the model parameters of the initial structure of the first flame detection model are adjusted until the preset convergence condition is reached, resulting in a trained first flame detection model.

[0174] The preset convergence condition can be: the first flame detection model with the initial structure is trained a preset number of times. Alternatively, the preset convergence condition can also be: the function value of the loss function calculated for a consecutive preset number of times is less than a preset threshold. A larger function value indicates a greater difference between the target type label and the predicted type label. A smaller function value indicates a smaller difference between the target type label and the predicted type label. If the function value of the loss function calculated for a consecutive preset number of times is less than the preset threshold, it indicates that the predicted type label output by the first flame detection model for a consecutive preset number of times has a small difference from the target type label. Therefore, the accuracy of the first flame detection model is high, and the first flame detection model trained for the preset number of times can be considered a well-trained first flame detection model.

[0175] The flame detection model training method provided in this invention extracts high-frequency temporal features of the second-order statistics of the flame region based on its shape characteristics, and trains a multilayer perceptron based on these high-frequency temporal features. Subsequently, flame detection based on the trained multilayer perceptron can distinguish between flame regions, light regions, and regions containing other interfering targets, thus improving the accuracy of flame detection.

[0176] and Figure 1 For the corresponding method implementation examples, see [link to relevant documentation]. Figure 11 , Figure 11 This is a structural diagram of a flame detection device provided in an embodiment of the present invention. The device includes: a target image acquisition module 1101, used to acquire a target image; wherein the target image is obtained based on video frames containing potential flame regions detected in a video to be processed; an associated image acquisition module 1102, used to acquire multiple associated images located after the target image from the video to be processed; a similarity sequence determination module 1103, used to calculate the similarity between each associated image and the target image according to the order of each associated image in the video to be processed, to obtain a similarity sequence corresponding to the target image; a probability prediction module 1104, used to input the similarity sequence corresponding to the target image into a pre-trained first flame detection model to obtain the probability that the type label of the potential flame region in the target image is a preset type label; wherein the first flame detection model is trained based on the similarity sequence corresponding to a first sample image and the target type label of the first sample image; and a target flame region detection module 1105, used to determine that the target image contains a target flame region if the probability corresponding to the preset flame type label is the highest.

[0177] Optionally, the image acquisition module 1101 is specifically used to acquire the video to be processed; for each video frame in the video to be processed, if a potential flame region is detected in the video frame based on the second flame detection model, an image to be detected is obtained based on the potential flame region in the video frame; wherein, the second flame detection model is trained based on a second sample image containing a flame region and the position of the flame region in the second sample image;

[0178] The associated image acquisition module 1102 is specifically used to determine a number of consecutive video frames in the video to be processed that are located after the video frame to which the image to be detected belongs; and to extract image regions at the same positions as the image to be detected from the number of consecutive video frames to obtain a number of associated images of the image to be detected.

[0179] Optionally, the image acquisition module 1101 is specifically used to detect each video frame in the video to be processed based on the second flame detection model to obtain the potential flame region in the video frame; and to expand the boundary of the potential flame region in the video frame according to the preset width and height adjustment parameters to obtain the adjusted potential flame region as the image to be detected.

[0180] Based on the flame detection device provided in this embodiment of the invention, the similarity sequence corresponding to the image to be detected includes the similarity between the image to be detected and multiple related images located after the image to be detected in the video to be processed. Therefore, the similarity sequence corresponding to the image to be detected can represent the temporal characteristics of the potential flame region in the image to be detected. Since the shape changes during flame combustion are irregular, the temporal characteristics of the flame region in the image are also irregular. However, the shape changes of other brightness regions (e.g., flashing light regions) are regular, so the temporal characteristics of the flame region in the image are regular and exhibit periodic changes. Therefore, the temporal characteristics of the flame region in the image differ from those of other brightness regions. Consequently, by performing flame detection based on the first flame detection model and the similarity sequence corresponding to the image to be detected, it is possible to avoid identifying light regions as flame regions, reduce the false alarm rate of flame detection, and improve the accuracy of flame region detection.

[0181] and Figure 5 For the corresponding method implementation examples, see [link to relevant documentation]. Figure 12 , Figure 12 This is a structural diagram of a flame detection model training device provided in an embodiment of the present invention, used to generate a first flame detection model as described in any of the above embodiments. The device includes:

[0182] The first sample image acquisition module 1201 is used to acquire a first sample image and a preset target type label for the first sample image; wherein, the first sample image is obtained based on a video frame containing a potential flame region detected in a sample video; the target type label is either a type label indicating that the first sample image contains a flame region, or a type label indicating that the first sample image does not contain a flame region; the associated image determination module 1202 is used to acquire multiple associated images located after the first sample image from the sample video; the similarity sequence determination module 1203 is used to calculate the similarity between each associated image and the first sample image according to the order of each associated image in the sample video, to obtain a similarity sequence corresponding to the first sample image; the predicted type label determination module 1204 is used to input the similarity sequence corresponding to the first sample image into the first flame detection model of the initial structure to obtain the predicted type label of the potential flame region in the first sample image; the model parameter adjustment module 1205 is used to adjust the model parameters of the first flame detection model of the initial structure based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image, to obtain a trained first flame detection model.

[0183] Optionally, the first sample image acquisition module 1201 is specifically used to acquire sample videos; for each video frame in the sample video, if a potential flame region is detected in the video frame based on the second flame detection model, a first sample image is obtained based on the potential flame region in the video frame; wherein, the second flame detection model is trained based on the second sample image containing the flame region and the position of the flame region in the second sample image; and a preset target type label for the first sample image is acquired;

[0184] The associated image determination module 1202 is specifically used to, for each first sample image, determine, from the sample video, a plurality of consecutive video frames located after the video frame to which the first sample image belongs; and extract, from the plurality of consecutive video frames, an image region at the same position as the first sample image, to obtain a plurality of associated images of the first sample image.

[0185] Optionally, the first sample image acquisition module 1201 is specifically used to detect each video frame in the sample video based on the second flame detection model to obtain the potential flame region in the video frame; and to expand the boundary of the potential flame region in the video frame according to preset width and height adjustment parameters to obtain the adjusted potential flame region as the first sample image.

[0186] Optionally, the device further includes: an image group division module, configured to divide first sample images with the same target type label into an image group after the first sample image acquisition module 1201 acquires the preset target type label of the first sample image; and an image storage module, configured to store the first sample images belonging to the same image group and the associated images of the first sample images to the corresponding designated storage location.

[0187] The device further includes: an associated image acquisition module, configured to acquire the first sample image and its associated images from the specified storage location before the similarity sequence determination module 1203 performs the following steps: calculating the similarity between each associated image and the first sample image according to the order of the associated images in the sample video, and obtaining the similarity sequence corresponding to the first sample image.

[0188] Optionally, the first flame detection model is a multilayer perceptron;

[0189] The prediction type label determination module 1204 is specifically used to input the similarity sequence corresponding to the first sample image into the first hidden layer of the first flame detection model of the initial structure to obtain the first temporal feature of the first sample image output by the first hidden layer; input the first temporal feature of the first sample image into the second hidden layer of the first flame detection model of the initial structure to obtain the second temporal feature of the first sample image output by the second hidden layer; input the second temporal feature of the first sample image into the output layer of the first flame detection model of the initial structure to obtain the probability that the type label of the potential flame region in the first sample image is a preset type label; and determine the type label with the highest probability from the preset type labels as the prediction type label of the potential flame region in the first sample image.

[0190] Optionally, the similarity sequence determination module 1203 is specifically used to input the first sample image into a pre-trained feature extraction model to obtain a sample feature vector representing the image features of the first sample image; for each associated image, input the associated image into the feature extraction model to obtain an associated feature vector representing the image features of the associated image; and calculate the similarity between the associated feature vector of each associated image and the sample feature vector in sequence according to the order of each associated image in the sample video to obtain the similarity sequence corresponding to the first sample image.

[0191] Optionally, the model parameter adjustment module 1205 is specifically used to calculate the function value of a preset loss function based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image; wherein, the function value of the loss function represents the difference between the target type label and the predicted type label; based on the calculated function value of the loss function, the model parameters of the first flame detection model of the initial structure are adjusted until a preset convergence condition is reached to obtain the trained first flame detection model.

[0192] Based on the flame detection model training device provided in this embodiment of the invention, a first flame detection model is trained based on the similarity sequence and target type label of a first sample image. Subsequently, flame detection is performed based on the similarity sequence corresponding to the trained first flame detection model and the image to be detected. This avoids misidentifying light areas as flame areas, reduces the false alarm rate of flame detection, and improves the accuracy of flame area detection.

[0193] This invention also provides an electronic device, such as... Figure 13 As shown, it includes a processor 1301, a communication interface 1302, a memory 1303, and a communication bus 1304. The processor 1301, the communication interface 1302, and the memory 1303 communicate with each other through the communication bus 1304. The memory 1303 is used to store computer programs. When the processor 1301 executes the program stored in the memory 1303, it implements any of the flame detection method steps described in the above embodiments, or any of the flame detection model training method steps described in the above embodiments.

[0194] The communication bus mentioned in the above electronic devices can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used to represent it in the diagram, but this does not mean that there is only one bus or one type of bus. The communication interface is used for communication between the above electronic devices and other devices.

[0195] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0196] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0197] In another embodiment of the present invention, a computer-readable storage medium is also provided, which stores a computer program. When the computer program is executed by a processor, it implements the flame detection method steps described in any of the above embodiments, or the flame detection model training method steps described in any of the above embodiments.

[0198] In another embodiment of the present invention, a computer program product containing instructions is also provided, which, when run on a computer, causes the computer to execute any of the flame detection methods described in the above embodiments, or any of the flame detection model training methods described in the above embodiments.

[0199] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present invention are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another website, computer, server, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid state disk (SSD)).

[0200] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0201] The various embodiments in this specification are described in a related manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the embodiments of apparatus, electronic devices, computer-readable storage media, and computer program products are basically similar to the method embodiments, and therefore the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0202] The above description is merely a preferred embodiment of the present invention and is not intended to limit the scope of protection of the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention are included within the scope of protection of the present invention.

Claims

1. A flame detection method, characterized by, The method comprises: acquiring a to-be-detected image; wherein the to-be-detected image is obtained based on a video frame containing a potential flame region detected in a to-be-processed video; acquiring a plurality of associated images located after the to-be-detected image from the to-be-processed video; in accordance with the order of each associated image in the to-be-processed video, the similarity of each associated image and the to-be-detected image is calculated in turn to obtain a similarity sequence corresponding to the to-be-detected image; inputting the similarity sequence corresponding to the to-be-detected image into a first flame detection model trained in advance to obtain a probability of a type label representing a potential flame region in the to-be-detected image being each type label in a preset type label; wherein the first flame detection model is obtained based on a similarity sequence corresponding to a first sample image and a target type label of the first sample image; if the probability corresponding to a flame type label in each type label is the maximum, it is determined that the to-be-detected image contains a target flame region; the acquiring of the to-be-detected image comprises: acquiring a to-be-processed video; for each video frame in the to-be-processed video, if a potential flame region exists in the video frame based on a second flame detection model, a to-be-detected image is obtained based on the potential flame region in the video frame; wherein the second flame detection model is obtained based on a second sample image containing a flame region and the position of the flame region in the second sample image; the acquiring of the plurality of associated images located after the to-be-detected image from the to-be-processed video comprises: determining a plurality of continuous video frames located after the video frame to which the to-be-detected image belongs in the to-be-processed video; extracting an image region at the same position as the to-be-detected image from the plurality of continuous video frames to obtain a plurality of associated images of the to-be-detected image.

2. The method of claim 1, wherein, the acquiring of the to-be-detected image based on the potential flame region in the video frame comprises: detecting each video frame in the to-be-processed video based on the second flame detection model to obtain a potential flame region in the video frame; expanding the boundary of the potential flame region in the video frame according to a preset width-height adjustment parameter to obtain an adjusted potential flame region as a to-be-detected image.

3. A flame detection model training method, characterized by, a method for generating the first flame detection model of any one of claims 1 to 2, the method comprising: acquiring a first sample image and a target type label of the first sample image; wherein the first sample image is obtained based on a video frame containing a potential flame region detected in a sample video; the target type label is a type label representing that the first sample image contains a flame region, or a type label representing that the first sample image does not contain a flame region; acquiring a plurality of associated images located after the first sample image from the sample video; sequentially calculate the similarity of each associated image and the first sample image according to the sequence of the associated images in the sample video, to obtain a similarity sequence corresponding to the first sample image; input the similarity sequence corresponding to the first sample image into an initial structure of the first flame detection model, to obtain a predicted type label of the potential flame region in the first sample image; adjust model parameters of the initial structure of the first flame detection model based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image, to obtain a trained first flame detection model.

4. The method of claim 3, wherein, The method further comprises: obtaining a sample video; for each video frame in the sample video, if a potential flame region exists in the video frame based on a second flame detection model, obtaining a first sample image based on the potential flame region in the video frame; wherein the second flame detection model is trained based on a second sample image containing a flame region and a position of the flame region in the second sample image; obtaining a preset target type label of the first sample image; The method further comprises: for each first sample image, determining a plurality of continuous video frames after a video frame to which the first sample image belongs from the sample video; extracting an image region at the same position as the first sample image from the plurality of continuous video frames, to obtain a plurality of associated images of the first sample image.

5. The method of claim 4, wherein, The method further comprises: for each video frame in the sample video, detecting the video frame based on a second flame detection model, to obtain a potential flame region in the video frame; extending the boundary of the potential flame region in the video frame according to a preset width-height adjustment parameter, to obtain an adjusted potential flame region as a first sample image.

6. The method of claim 4, wherein, After obtaining the preset target type label of the first sample image, the method further comprises: dividing first sample images with the same target type label into an image group; storing first sample images belonging to the same image group and associated images of the first sample images to a corresponding designated storage location; Before sequentially calculating the similarity of each associated image and the first sample image according to the sequence of the associated images in the sample video, to obtain a similarity sequence corresponding to the first sample image, the method further comprises: obtaining the first sample image and the associated images of the first sample image from the designated storage location.

7. The method of claim 3, wherein, The first flame detection model is a multi-layer perception machine; The similarity sequence corresponding to the first sample image is input into an initial structure of a first flame detection model to obtain a predicted type label representing whether the first sample image contains a flame region, including: The similarity sequence corresponding to the first sample image is input into a first hidden layer of the initial structure of the first flame detection model to obtain first time domain features of the first sample image output by the first hidden layer; The first time domain features of the first sample image are input into a second hidden layer of the initial structure of the first flame detection model to obtain second time domain features of the first sample image output by the second hidden layer; The second time domain features of the first sample image are input into an output layer of the initial structure of the first flame detection model to obtain a probability of each type label of a potential flame region in the first sample image being a preset type label; A type label corresponding to the maximum probability is determined from the preset type labels as the predicted type label of the potential flame region in the first sample image.

8. The method of claim 3, wherein, The similarity sequence corresponding to the first sample image is obtained by sequentially calculating the similarity between each associated image and the first sample image according to the order of the associated images in the sample video, including: The first sample image is input into a pre-trained feature extraction model to obtain a sample feature vector representing the image features of the first sample image; For each associated image, the associated image is input into the feature extraction model to obtain an associated feature vector representing the image features of the associated image; The similarity sequence corresponding to the first sample image is obtained by sequentially calculating the similarity between the associated feature vector of each associated image and the sample feature vector according to the order of the associated images in the sample video.

9. The method of claim 3, wherein, The model parameters of the initial structure of the first flame detection model are adjusted based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image to obtain a trained first flame detection model, including: The function value of a preset loss function is calculated based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image; wherein the function value of the loss function represents the difference between the target type label and the predicted type label; The model parameters of the initial structure of the first flame detection model are adjusted based on the calculated function value of the loss function until a preset convergence condition is reached to obtain the trained first flame detection model.

10. A flame detection apparatus, characterized by, The device includes: A to-be-detected image acquisition module configured to acquire a to-be-detected image; wherein the to-be-detected image is obtained based on a video frame containing a potential flame region detected in a to-be-processed video; An associated image acquisition module configured to acquire a plurality of associated images located after the to-be-detected image from the to-be-processed video; The similarity sequence determination module is configured to sequentially calculate the similarity between each associated image and the to-be-detected image according to the sequence of the associated images in the to-be-processed video, and obtain a similarity sequence corresponding to the to-be-detected image. The probability prediction module is configured to input the similarity sequence corresponding to the to-be-detected image into a pre-trained first flame detection model to obtain a probability of a type label representing a potential flame region in the to-be-detected image being each of preset type labels. The target flame region detection module is configured to determine that the to-be-detected image contains a target flame region if a flame type label in the preset types corresponds to the maximum probability. The to-be-detected image acquisition module is specifically configured to acquire a to-be-processed video, and for each video frame in the to-be-processed video, if a second flame detection model detects that a potential flame region exists in the video frame, obtain a to-be-detected image based on the potential flame region in the video frame, wherein the second flame detection model is trained based on a second sample image containing a flame region and a position of the flame region in the second sample image. The associated image acquisition module is specifically configured to determine a plurality of continuous video frames in the to-be-processed video located after a video frame to which the to-be-detected image belongs, and extract an image region at the same position as the to-be-detected image from the plurality of continuous video frames to obtain a plurality of associated images of the to-be-detected image.

11. The apparatus of claim 10, wherein, The to-be-detected image acquisition module is specifically configured to, for each video frame in the to-be-processed video, detect the video frame based on a second flame detection model to obtain a potential flame region in the video frame, and perform boundary expansion on the potential flame region in the video frame according to a preset width-height adjustment parameter to obtain an adjusted potential flame region as a to-be-detected image. 12.A flame detection model training apparatus, comprising: The apparatus for generating the first flame detection model of any one of claims 10-11 comprises: A first sample image acquisition module is configured to acquire a first sample image and a target type label of the first sample image; wherein the first sample image is obtained based on a video frame containing a potential flame region detected in a sample video; and the target type label is a type label representing that the first sample image contains a flame region, or a type label representing that the first sample image does not contain a flame region. An associated image determination module is configured to acquire a plurality of associated images located after the first sample image from the sample video. A similarity sequence determination module is configured to sequentially calculate the similarity between each associated image and the first sample image according to the sequence of the associated images in the sample video, and obtain a similarity sequence corresponding to the first sample image. A predicted type label determination module is configured to input the similarity sequence corresponding to the first sample image into a first flame detection model of an initial structure to obtain a predicted type label of a potential flame region in the first sample image. The model parameter adjustment module is configured to adjust model parameters of the initial first flame detection model based on the target type label of the first sample image and the predicted type label of the potential flame region in the first sample image, to obtain a trained first flame detection model.

13. The apparatus of claim 12, wherein, The first sample image acquisition module is specifically configured to acquire a sample video; for each video frame in the sample video, if a potential flame region is detected in the video frame based on a second flame detection model, a first sample image is obtained based on the potential flame region in the video frame; the second flame detection model is trained based on a second sample image containing a flame region and a position of the flame region in the second sample image; and a target type label of the first sample image is acquired. The associated image determination module is specifically configured to, for each first sample image, determine a plurality of continuous video frames located after a video frame to which the first sample image belongs from the sample video; and extract an image region at a same position as the first sample image from the plurality of continuous video frames to obtain a plurality of associated images of the first sample image.

14. The apparatus of claim 13, wherein, The first sample image acquisition module is specifically configured to, for each video frame in the sample video, detect the video frame based on the second flame detection model to obtain a potential flame region in the video frame; and perform boundary expansion on the potential flame region in the video frame according to a preset width-height adjustment parameter to obtain an adjusted potential flame region as a first sample image.

15. The apparatus of claim 13, wherein, The device further comprises: An image group division module configured to divide first sample images with the same target type label into an image group after the first sample image acquisition module acquires the target type label of the first sample image. An image storage module configured to store first sample images belonging to the same image group and associated images of the first sample images to corresponding designated storage positions. The device further comprises: An associated image acquisition module configured to acquire the first sample image and the associated images of the first sample image from the designated storage positions before the similarity sequence determination module sequentially calculates the similarity between each associated image and the first sample image according to the order of the associated images in the sample video to obtain the similarity sequence corresponding to the first sample image.

16. The apparatus of claim 12, wherein, The first flame detection model is a multi-layer perception machine. The prediction type label determination module is specifically configured to input the similarity sequence corresponding to the first sample image into a first hidden layer of a first flame detection model of an initial structure to obtain first time domain features of the first sample image output by the first hidden layer; input the first time domain features of the first sample image into a second hidden layer of the first flame detection model of the initial structure to obtain second time domain features of the first sample image output by the second hidden layer; input the second time domain features of the first sample image into an output layer of the first flame detection model of the initial structure to obtain a probability that a type label of a potential flame region in the first sample image is each type label in preset type labels; and determine, from the preset type labels, a type label corresponding to the maximum probability as the prediction type label of the potential flame region in the first sample image.

17. The apparatus of claim 12, wherein, The similarity sequence determination module is specifically configured to input the first sample image into a pre-trained feature extraction model to obtain a sample feature vector representing image features of the first sample image; input each associated image into the feature extraction model to obtain an associated feature vector representing image features of the associated image; and in accordance with an order of each associated image in the sample video, sequentially calculate a similarity between the associated feature vector of each associated image and the sample feature vector to obtain a similarity sequence corresponding to the first sample image.

18. The apparatus of claim 12, wherein, The model parameter adjustment module is specifically configured to calculate a function value of a preset loss function based on the target type label of the first sample image and the prediction type label of the potential flame region in the first sample image, wherein the function value of the loss function represents a difference between the target type label and the prediction type label; and adjust model parameters of the first flame detection model of the initial structure based on the calculated function value of the loss function until a preset convergence condition is reached to obtain a trained first flame detection model.

19. An electronic device, comprising: The apparatus includes a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. The memory is configured to store a computer program. The processor is configured to execute the program stored in the memory to implement the method of claims 1-2 or any one of claims 3-9.

20. A computer-readable storage medium, characterized in that, The computer program is stored in the computer readable storage medium and is executed by the processor to implement the method of claims 1-2 or any one of claims 3-9. The computer program is stored in the computer readable storage medium and is executed by the processor to implement the method of claims 1-2 or any one of claims 3-9.

Images