Flame early warning method, computer equipment and storage medium

Abstract

The invention relates to a flame early warning method, computer equipment and a storage medium. According to the invention, the deep learning model comprising the forward prediction model and the backward prediction model is set, the trained deep learning model is used to output the prediction image, and the prediction image is compared with the real image, so that the real image is judged to deviate from the development direction of a normal flameless condition when the prediction image and the real image have deviation. Therefore, the occurrence of the flame is judged when the real image deviates from the development direction of the normal flame-free condition, early warning can be carried out before the occurrence of the flame is detected in time in the production process of the energy storage equipment, the occurrence rate of the flame is reduced, and the safety is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, in particular to a flame warning method, computer equipment and a storage medium. BACKGROUND

[0002] In real life, fire prevention is an important matter. For example, in the event of a fire in a power storage device, an explosion and other accidents are likely to occur, causing casualties. Therefore, it is particularly important to predict and monitor the fire of the power storage device and timely detect the fire to avoid major accidents. When the fire is detected, real-time warning should be given to inform the personnel producing the power storage device to take necessary measures in time to reduce losses. In the past few years, target detection algorithms based on deep learning have received considerable attention. With the development of artificial intelligence and some target detection algorithms based on deep convolutional neural networks (CNN), some algorithms stand out in the field of computer vision. Currently, target detection algorithms are commonly used to identify whether the anti-static chain is connected. For example, classic target detection networks such as SDD, RCNN, Faster RCNN, and YOLO series. The detection method based on deep learning has good effect in normal server and indoor environment, but in outdoor environment, the detection effect is poor due to the influence of sunlight, light irradiation, and ground and wall reflection, which often leads to false detection. Moreover, these algorithms require high computing power, and the current detection equipment is mostly edge terminal equipment. Due to the limitation of computing power, it is difficult to directly deploy it on edge terminal equipment. Therefore, timely detection of fire in the production process of power storage devices can effectively improve safety. SUMMARY

[0003] Therefore, a flame warning method, computer equipment and storage medium are provided to solve the technical problem that the current power storage device production process cannot timely detect the occurrence of fire and cause high false positive rate.

[0004] In one aspect, a flame warning method is provided, which comprises: obtaining a first time thermal imaging picture of all cameras facing a target detection area of a power storage device, obtaining a corresponding position relationship of all cameras, obtaining a first thermal imaging picture group of a plurality of image acquisition devices facing the position of the target power storage device, and generating a panoramic picture of the target power storage device according to the first thermal imaging picture group and the relative position of the plurality of image acquisition devices; timely collecting a second thermal imaging picture group through the plurality of image acquisition devices to generate a target detection area, and determining a fire score value in the target detection area according to the target detection area and the second thermal imaging picture group; In response to the flame score value being greater than a first threshold value, a thermal imaging video of the target energy storage device is recorded by the plurality of image acquisition devices, and the thermal imaging video is converted into a time-sequenced thermal imaging picture stream; A deep learning model including a forward prediction model and a backward prediction model is set up, and the thermal imaging picture stream is input into the trained deep learning model; The deep learning model is called to determine whether a predicted image exists at the time corresponding to the thermal imaging picture stream, if not, the thermal imaging picture stream at the next time is obtained as the real image at the next time, if so, the thermal imaging picture stream is taken as the real image, and the distance score is determined according to the predicted image and the real image; According to the distance score and the image frame change threshold, it is determined whether an abnormal change occurs, and a warning is given when an abnormal change occurs.

[0005] In one of the embodiments, the first thermal imaging picture group of the plurality of image acquisition devices directed to the position of the target energy storage device is obtained, and the panoramic view of the target energy storage device is generated according to the first thermal imaging picture group and the relative positions of the plurality of image acquisition devices, which includes: All thermal imaging pictures collected by the plurality of image acquisition devices at the same time are subjected to gray scale processing, Gaussian blur processing and local binary feature processing to obtain the first thermal imaging picture group; All thermal imaging pictures in the first thermal imaging picture group are subjected to superimposition processing to generate the panoramic view; The point coordinates of all pixel points in the panoramic view are calculated according to the positional relationship of all cameras and stored in the coordinate database.

[0006] In one of the embodiments, the second thermal imaging picture group is collected by the plurality of image acquisition devices at regular intervals to generate a target detection area, and the flame score value in the target detection area is determined according to the target detection area and the second thermal imaging picture group, which includes: The thermal imaging pictures in the second thermal imaging picture group are preprocessed and subjected to local binary feature processing; The thermal imaging pictures in the second thermal imaging picture group after processing are subjected to feature recognition and output a first feature map; The first feature map is divided into regions, and a target recommended area is output; The target recommended area features in the first feature map are extracted and a second feature map is output; Corresponding the point coordinates of the pixel points in the second feature map to the point coordinates of all pixel points in the panoramic map, a position of the target detection region in the panoramic map is obtained, and the second feature map is divided into a plurality of target segmentation regions according to temperature distribution in the target detection region; In each target segmentation region, flame target recognition is performed, and a target frame, target classification and target score are output; A flame score value of each target segmentation region is obtained, and a flame score value of the target detection region appearing flame is obtained by weighted summation.

[0007] In one embodiment, the deep learning model is called to determine whether a predicted image exists at the time corresponding to the thermographic picture stream, if not, the thermographic picture stream at the next time is obtained as the real image at the next time, if so, the thermographic picture stream is taken as the real image, and the distance score is determined according to the predicted image and the real image, including: A real image sequence and a predicted image sequence are set in the deep learning model, the real image sequence is used to store real images, and the predicted image sequence is used to store predicted images; In response to the deep learning model receiving the thermographic picture stream, the thermographic picture stream is stored as a real image in the real image sequence, the time corresponding to the real image is obtained t, and the real image is represented as f t ; The deep learning model predicts a predicted image t at time t+1 according to the real image f , and stores the predicted image in the predicted image sequence; It is determined whether a predicted image exists in the predicted image sequence at time t; In response to the predicted image sequence not having a predicted image at time t, a thermographic picture stream at time t+1 is obtained as a thermographic picture stream at time t+1; In response to the predicted image sequence having a predicted image at time t, the predicted image at time t is obtained, and the distance score of the predicted image and the real image at time t is calculated by the formula , wherein λ is a weight parameter, represents the two-norm of .

[0008] In one embodiment, the distance score is compared with an image frame change threshold to determine whether abnormal change occurs, and a warning is given when abnormal change occurs, including: The image frame change threshold comprises a second threshold and a third threshold, the second threshold is greater than the third threshold; When the distance score is greater than the second threshold, it is determined that a flame occurs at the current time; When the distance score is between the second threshold and the third threshold, it is determined that a flame will occur in the energy storage device in the future, and a warning is given; When the distance score is less than the third threshold, it is determined that the current time is normal.

[0009] In one embodiment, the step of determining whether an abnormal change occurs according to the comparison between the distance score and the image frame change threshold, and giving a warning when an abnormal change occurs comprises: obtaining a maximum distance score of the predicted image and the real image; When the maximum distance score is greater than the image frame change threshold, it is determined that a flame occurs at the current time, and a warning is given.

[0010] In another aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes the computer program to implement the following steps: obtaining a first time thermal imaging picture of all cameras facing a target detection area where an energy storage device is located, obtaining a corresponding position relationship of all cameras, obtaining a first thermal imaging picture group of a plurality of image acquisition devices facing the target energy storage device, and generating a panoramic view of the target energy storage device according to the first thermal imaging picture group and the relative positions of the plurality of image acquisition devices; timely acquiring a second thermal imaging picture group through the plurality of image acquisition devices to generate a target detection area, and determining a flame score value in the target detection area according to the target detection area and the second thermal imaging picture group; In response to the flame score value being greater than a first threshold, a thermal imaging video of the target energy storage device is recorded through the plurality of image acquisition devices, and the thermal imaging video is converted into a thermal imaging picture stream arranged in time sequence; setting a deep learning model comprising a forward prediction model and a backward prediction model, inputting the thermal imaging picture stream into the trained deep learning model; calling the deep learning model to determine whether a predicted image exists at the time corresponding to the thermal imaging picture stream, if not, obtaining a thermal imaging picture stream at the next time as a real image at the next time, if so, taking the thermal imaging picture stream as a real image, and determining a distance score according to the predicted image and the real image; The distance score is compared with a threshold of image frame change to determine whether an abnormal change occurs, and a warning is given when the abnormal change occurs.

[0011] In one of the embodiments, before the thermal imaging picture stream is input into the trained deep learning model, the method further comprises: The forward prediction model and the backward prediction model are both composed of an encoder and a decoder, the forward prediction model is used to predict future images at future sampling time according to historical images, and the backward prediction model is used to predict historical images at previous time according to future images; The deep learning model is trained by using the collected image frame sequence; During the training of the forward prediction model, one of the collected image frame sequences is input into the forward prediction model, the forward prediction model outputs a prediction image sequence of a plurality of frames at subsequent time, a real image sequence corresponding to the prediction image sequence in the collected image frame sequence is obtained, a loss of the forward prediction model is calculated by using an absolute value loss function, and whether the forward prediction model is trained is determined according to a value of the absolute value loss function; During the training of the backward prediction model, one of the collected image frame sequences is input into the backward prediction model, the backward prediction model outputs a prediction image sequence of a plurality of frames at previous time, a real image sequence corresponding to the prediction image sequence in the collected image frame sequence is obtained, a loss of the backward prediction model is calculated by using a square loss function, and whether the backward prediction model is trained is determined according to a value of the square loss function.

[0012] In one of the embodiments, the training of the deep learning model by using the collected image frame sequence further comprises: The discriminator obtains a prediction image at a target time output by the decoder and a real image at the target time in the collected image frame sequence, compares the prediction image at the target time with the real image at the target time, and records a first value if the prediction image at the target time is the same as the real image at the target time, and records a second value if the prediction image at the target time is different from the real image at the target time; Whether the deep learning model is trained is determined according to a proportion of the first value and the second value.

[0013] In another aspect, a computer readable storage medium is provided, and a computer program is stored on the computer readable storage medium, and the computer program is executed by a processor to implement the following steps: acquire a first time thermal imaging picture of all cameras facing a target detection area where the energy storage device is located, acquire a corresponding position relationship of all cameras, acquire a first thermal imaging picture group of a plurality of image acquisition devices facing a position where the target energy storage device is located, and generate a panoramic view of the target energy storage device according to the first thermal imaging picture group and the relative positions of the plurality of image acquisition devices; acquire a second thermal imaging picture group by the plurality of image acquisition devices in a timely manner to generate a target detection area, and determine a fire score value in the target detection area according to the target detection area and the second thermal imaging picture group; in response to the fire score value being greater than a first threshold value, record a thermal imaging video of the target energy storage device by the plurality of image acquisition devices, and convert the thermal imaging video into a thermal imaging picture stream arranged in time sequence; set a deep learning model including a forward prediction model and a backward prediction model, and input the thermal imaging picture stream into the trained deep learning model; call the deep learning model to determine whether a predicted image exists at a time corresponding to the thermal imaging picture stream, if not, acquire a thermal imaging picture stream at a next time as a real image at the next time, and if so, take the thermal imaging picture stream as a real image, and determine a distance score according to the predicted image and the real image; determine whether an abnormal change occurs according to a comparison between the distance score and an image frame change threshold value, and give a warning when the abnormal change occurs.

[0014] The flame warning method, computer device and storage medium described above, by setting a deep learning model including a forward prediction model and a backward prediction model, outputting a predicted image by using the trained deep learning model, comparing the predicted image with the real image, determining that the real image deviates from the development direction of the normal no-flame situation when the predicted image deviates from the real image, and thus judging that the flame occurs when the real image deviates from the development direction of the normal no-flame situation, can timely detect and give a warning before the flame occurs in the production process of the energy storage device, reduce the occurrence rate of the flame, and improve the safety. BRIEF DESCRIPTION OF DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiment description. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can also be obtained by those skilled in the art without creative labor.

[0016] Figure 1 a schematic diagram of a deep learning model in an embodiment of the present application is an autoencoder scheme; Figure 2 This is a flow chart of a flame warning method in one embodiment of the present application; Figure 3 This is a schematic diagram of a flame prediction method using forward and backward bidirectional prediction in one embodiment of the present application; Figure 4 This is a schematic diagram of a deep learning model in one embodiment of the present application that is combined with a generative adversarial network solution; Figure 5 This is a diagram of the internal structure of a computer device in one embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0018] In this application scenario, the camera records the video of all cameras under the thermal imaging view through the video stream to form an image frame sequence. Let the current time be t, then the image frame sequence is expressed as .

[0019] Then at time t, the detection of the flame is the judgment of f t Whether a flame occurs at a certain moment, the flame is recorded as 1 if it occurs, and 0 if it is normal; the flame warning is to judge whether a flame will occur within a period of time, for example, within the k frame, that is, whether a flame occurs from t+1 to t+k frames. If a flame occurs in the [t+1, t+k] time period, it is recorded as 1, and 0 if it is normal.

[0020] When the sampling frame rate remains stable, the number of frames can be converted into the advance warning time. Therefore, the larger the warning interval k is, the longer the advance warning time is.

[0021] The flame warning method provided in the embodiment of the present application can be applied to Figure 1 In the application environment shown, the input is a sequence of n images before the current moment, and the output is the predicted current frame. By comparing the predicted current frame with the actual current frame, we determine whether the current frame is abnormal. During training, we only use negative samples, which has the advantage of easy access to datasets and no need for extensive annotation.

[0022] like Figure 1 As shown, this is a self-encoder solution, the input is , after a deep learning model consisting of an encoder and a decoder, the output is . n is an integer.

[0023] In the present embodiment, as shown in Figure 2 A flame early warning method is provided, comprising the following steps: Step S1, acquiring a first thermal imaging picture group of a plurality of image acquisition devices facing the position of the target energy storage device, and generating a panoramic view of the target energy storage device according to the first thermal imaging picture group and the relative positions of the plurality of image acquisition devices; Step S2, acquiring a second thermal imaging picture group by the plurality of image acquisition devices at a fixed time to generate a target detection area, and determining a flame score value in the target detection area according to the target detection area and the second thermal imaging picture group; Step S3, in response to the flame score value being greater than a first threshold value, recording a thermal imaging video of the target energy storage device by the plurality of image acquisition devices, and converting the thermal imaging video into a thermal imaging picture stream arranged in time sequence; Step S4, setting a deep learning model including a forward prediction model and a backward prediction model, and inputting the thermal imaging picture stream into the trained deep learning model; Step S5, calling the deep learning model to determine whether there is a predicted image corresponding to the thermal imaging picture stream at the moment, if not, acquiring a thermal imaging picture stream at the next moment as a real image at the next moment, if so, taking the thermal imaging picture stream as a real image, and determining a distance score according to the predicted image and the real image; Step S6, comparing the distance score with an image frame change threshold value to determine whether an abnormal change occurs, and giving an early warning when an abnormal change occurs.

[0024] Specifically, all cameras in the target detection area, such as a production energy storage device factory, can be accessed or accessed, a plurality of first moment thermal imaging pictures at the same moment are intercepted from the monitoring video recorded by all cameras, and the plurality of pictures are preprocessed. Before intercepting the first moment thermal imaging picture, the user needs to arrange the cameras in the target detection area to ensure that each position in the production energy storage device factory is at least in the field of view of two cameras, so that the matching of the overlapping fields of view in different cameras is matched, and preferably, the camera can be a combined camera containing visible light and thermal imaging.

[0025] In the daily monitoring (step S2), the system periodically acquires the flame score value. If this score value is below the first threshold value, it means that there is no obvious flame sign or only normal temperature fluctuations, and there is no need to continuously record high-definition video and perform complex deep learning analysis, which avoids storing, transmitting and calculating a large amount of useless data under normal conditions, greatly saving storage space, computing resources and network bandwidth. Only when the flame score value exceeds the first threshold value (step S3), indicating that there may be a fire risk, will the system start the higher-level and more resource-consuming video recording and deep learning analysis process. Step S2 is a preliminary detection, mainly a periodic "snapshot" detection, which determines a preliminary flame score value through a single or small number of picture groups, which can quickly screen out potential abnormalities. Step S3 is the preparation for deep analysis. When the preliminary detection finds an abnormality (the flame score value is greater than the first threshold value), the system needs more detailed and continuous information to confirm and predict, and the thermal imaging video and the converted time sequence picture stream provide time sequence data of the flame development process.

[0026] The method further includes: acquiring a first thermal imaging picture group of a plurality of image acquisition devices facing a location of the target energy storage device, and generating a panorama of the target energy storage device according to the first thermal imaging picture group and relative positions of the plurality of image acquisition devices. The method further includes: performing grayscale processing, Gaussian blur processing, and local binary feature processing on all thermal imaging pictures collected by the plurality of image acquisition devices at the same time to obtain the first thermal imaging picture group. The method further includes: performing coincidence processing on all thermal imaging pictures in the first thermal imaging picture group and generating the panorama. The method further includes: calculating point coordinates of all pixel points in the panorama according to the corresponding position relationship of all cameras and storing the point coordinates in a coordinate database.

[0027] Specifically, the preprocessing includes modifying the picture into a grayscale picture, performing Gaussian blur processing on the grayscale picture to further reduce noise interference of the picture and reduce calculation error of pixel value coordinates of a highlight part, and finally performing LBP (local binary pattern) feature processing on the processed picture. The first thermal imaging picture after the preprocessing can be processed to generate the panorama by coinciding pixel points at the same position in each picture. The 3D coordinate axis is constructed by the position relationship sent by the user, including the coordinates corresponding to each camera, and the point coordinates of each pixel point in the panorama are calculated by substituting the panorama into the coordinate axis. When generating the panorama, the image position is adjusted according to the shooting angle to correct the specific process of distortion, align the images at the same position, realize image fitting, and generate the panorama after the coincidence processing.

[0028] The second thermal imaging picture group is acquired by the plurality of image acquisition devices at a timing to generate a target detection region, and a flame score value in the target detection region is determined according to the target detection region and the second thermal imaging picture group, and the method comprises the following steps: The thermal imaging pictures in the second thermal imaging picture group are preprocessed and subjected to local binary feature processing. The thermal imaging pictures in the second thermal imaging picture group after processing are subjected to feature recognition and output a first feature map. The first feature map is subjected to region division and outputs a target recommendation region. The target recommendation region features in the first feature map are extracted and output a second feature map. The point coordinates of the pixel points in the second feature map are corresponded to the point coordinates of all pixel points in the panoramic map, the position of the target detection region in the panoramic map is acquired, and the second feature map is divided into a plurality of target segmentation regions according to temperature distribution in the target detection region. Flame target recognition is performed in each target segmentation region, and a target frame, target classification and target score are output. The flame score value of each target segmentation region is acquired, and the flame score value of the target detection region appearing flame is acquired by weighted summation.

[0029] Specifically, the calculation formula of the flame score value of the target detection region appearing flame is FD=(S1×W1)+(S2×W2)+…+(SN×WN), wherein FD is the flame score value of the target detection region appearing flame, S1, S2, …, SN respectively represent the flame score value of the first, second, and Nth target segmentation region, W1, W2, …, WN respectively represent the weight allocated to the first, second, and Nth target segmentation region, and N is the total number of target segmentation regions contained in the target detection region.

[0030] The local binary feature processing manner of the thermal imaging pictures in the second thermal imaging picture group is to set a gray threshold to distinguish hot and cold of the thermal imaging pictures in the second thermal imaging picture group, and the thermal imaging pictures are set as hot images when the gray value is greater than the gray threshold, and set as cold images when the gray value is less than or equal to the gray threshold. The first feature map refers to the global feature image in the thermal imaging picture in the second thermal imaging picture group, and the second feature map refers to the feature image in the target recommendation region in the second thermal imaging picture group.

[0031] Specifically, after generating the panoramic image, the second thermal imaging picture in the second thermal imaging picture group recorded at the same time from the camera is intercepted again, and the picture is also preprocessed, and then the noise points in the picture are taken out, and the size of the picture is scaled to a specific size; preferably, the specific size is set to 640*640, and during the picture size scaling process, the aspect ratio of the picture is not changed, and the maximum side is scaled to 640, and the small side is filled with gray scale. Then input the picture processed by the LBP feature into the ResNet50 network (Residual Neural Network 50, 50-layer residual network); then input the picture into the FPN (feature pyramid networks feature pyramid network) to obtain the extracted first feature map; then input the first feature map into the RPN (Region Proposal Network region proposal network) to obtain the corresponding target recommendation area; then input the first feature map and the target recommendation area into the ROIAlign (Region of Interesting Align interesting region alignment) network to obtain the second feature map of the final size; finally, input the second feature map through the head layer (prediction layer) to obtain the target detection area and segment it into multiple target segmentation areas.

[0032] Specifically, the input target segmentation area passes through the full connection layer to obtain the target frame, target classification and target score, that is, the target in the second thermal imaging picture group in the thermal imaging picture exists, and the judgment and probability of the target.

[0033] In the embodiment, the deep learning model is called to judge whether the thermal imaging picture stream corresponds to a predicted image at the moment, if not, the thermal imaging picture stream at the next moment is obtained as the real image at the next moment, if so, the thermal imaging picture stream is taken as the real image, and the distance score is determined according to the predicted image and the real image, including: The real image sequence and the predicted image sequence are set in the deep learning model, the real image sequence is used to store the real image, and the predicted image sequence is used to store the predicted image; In response to the deep learning model receiving the thermal imaging picture stream, the thermal imaging picture stream is stored as a real image in the real image sequence, the moment t corresponding to the real image is obtained, and the real image is represented as f t ; The deep learning model predicts the predicted image f t at the moment t+1 according to the real image f ; The predicted image f t is stored in the predicted image sequence; determine whether there is a predicted image in the predicted image sequence at time t; in response to there being no predicted image in the predicted image sequence at time t, obtain a thermal image stream at time t+1 as a thermal image stream at time t+1; in response to there being a predicted image in the predicted image sequence at time t, obtain the predicted image at time t calculate a distance score between the predicted image at time t and a real image at time t by a formula wherein λ is a weight parameter, represents a two-norm.

[0034] The greater the distance score value, the less similar the predicted image at the current time is to the real image, and the higher the warning level. The thermal image stream is taken as the real image sequence, and the real image sequence includes real images arranged in time sequence.

[0035] It should be noted that the formula for calculating the distance score is not unique, and any measurement method that can reflect the difference between the predicted image and the real image can be used as an alternative for distance calculation, such as peak signal-to-noise ratio, etc.

[0036] It can be understood that when there is no predicted image in the predicted image sequence at time t, if the distance score is needed, the method further comprises: in response to there being no predicted image in the predicted image sequence at time t, the deep learning model divides the thermal image stream into a previous time image frame sequence and a next time image frame sequence, predicts an expected image frame sequence at the next time as a predicted image according to the previous time image frame sequence, takes the next time multi-frame image frame sequence as a real image, and obtains a distance score by comparing the predicted image with the real image.

[0037] In the present embodiment, the determination of whether an abnormal change occurs according to a comparison between the distance score and an image frame change threshold value, and the giving of a warning when an abnormal change occurs comprises: setting the image frame change threshold value to include a second threshold value and a third threshold value, wherein the second threshold value is greater than the third threshold value; when the distance score is greater than the second threshold value, determining that a flame occurs at the current time; when the distance score is between the second threshold value and the third threshold value, determining that a flame will occur in the energy storage device at a future time, and giving a warning; when the distance score is less than the third threshold value, determining that the current time is normal.

[0038] That is, the second threshold value is θ1 and the third threshold value is θ2, where θ1>θ2; when When θ1< θ < θ2, it is determined that the flame appears at the current time; When θ < θ1, it is determined that the flame will appear in the future time of the energy storage device, and a warning is given; When θ2< θ, it is determined that the current time is normal.

[0039] The selection of θ1and θ2may be determined by the distribution of past cases. The distribution of past cases.

[0040] In other embodiments, the step of determining whether an abnormal change occurs by comparing the distance score with the image frame change threshold value, and giving a warning when an abnormal change occurs comprises: Obtaining the maximum distance score of the predicted image and the real image; When the maximum distance score is greater than the image frame change threshold value, it is determined that the flame appears at the current time, and a warning is given.

[0041] That is, the image frame change threshold value is set to θ, when The current flame phenomenon occurs.

[0042] In this embodiment, the step of determining whether an abnormal change occurs by comparing the distance score with the image frame change threshold value, and giving a warning when an abnormal change occurs comprises: Obtaining the maximum distance score of the predicted image and the real image; When the maximum distance score is greater than the image frame change threshold value, it is determined that the flame appears at the current time, and a warning is given.

[0043] That is, the image frame change threshold value is set to θ, and for the flame warning, the result of the backward prediction model is combined with the similarity function to evaluate, and the maximum distance score is obtained. When the value is greater than θ, it is determined that the flame will appear in the future time [t+1, t+2].

[0044] It is understood that in this embodiment, the deep learning model is configured to include a forward prediction model and a backward prediction model, and a forward and backward bidirectional prediction method is used to determine the presence of flames. The forward prediction model predicts future images at future sampling moments based on historical images, which is a prediction of what will happen. The backward prediction model predicts historical images at the previous moment based on future images. If flames appear in the future images, the future images predicted based on the current normal situation will be significantly different from the actual future images. Similarly, the historical images derived from the predicted future images will also be significantly different from the actual historical images. Therefore, we can infer whether an abnormality will occur in the future by determining whether there is a significant difference between the actual historical images and the reversely predicted historical images.

[0045] like Figure 3 As shown, in this embodiment, before inputting the thermal imaging image stream into the trained deep learning model, the method further includes: The forward prediction model and the backward prediction model are both composed of an encoder and a decoder, the forward prediction model is used to predict a future image at a future sampling moment based on a historical image, and the backward prediction model is used to predict a historical image at a previous moment based on a future image; Training the deep learning model using the acquired image frame sequence; When training the forward prediction model, an image frame sequence from the acquired image frame sequence is input into the forward prediction model, the forward prediction model outputs a predicted image sequence that predicts a number of frames at a subsequent moment, a real image sequence corresponding to the predicted image sequence from the acquired image frame sequence is obtained, a loss of the forward prediction model is calculated using an absolute value loss function, and whether the forward prediction model has completed training is determined based on the value of the absolute value loss function; When training the backward prediction model, an image frame sequence in the acquired image frame sequence is input into the backward prediction model, the backward prediction model outputs a predicted image sequence that predicts several frames at the previous moment, and a real image sequence corresponding to the predicted image sequence in the acquired image frame sequence is obtained. The loss of the backward prediction model is calculated using a square loss function, and whether the backward prediction model has completed training is determined based on the value of the square loss function.

[0046] The absolute value loss function is , y' is the image in the predicted image sequence, y is the image in the real image sequence, i is the pixel subscript in the image, and n is the number of pixels in the image.

[0047] The square loss function is , y' is an image in a predicted image sequence, y is an image in a real image sequence, i is a pixel index in the image, and n is a number of pixels in the image.

[0048] As shown in Figure 4 In this embodiment, the training of the deep learning model using the collected image frame sequence further includes: The discriminator obtains the predicted image at the target time output by the decoder and the real image at the target time in the collected image frame sequence, compares the predicted image at the target time with the real image at the target time, and records a first value if they are the same and a second value if they are different; The proportion of the first value and the second value is used to determine whether the deep learning model is trained.

[0049] It can be understood that the first value is the degree of realism of the generated predicted image, and the determination is whether it is real. The discriminator is to determine whether the generated image (predicted image) and the original image (real image) are not similar, and mainly plays an auxiliary role in training.

[0050] As shown in Figure 4 A discriminator is added on the basis of Figure 1 to improve the quality of the generated image. The difficulty of the discriminator is not whether the flame occurs, but whether the generated image is real. By comparing the predicted image with the real image, it is determined whether the real image deviates from the development direction of the normal no-flame situation, so that the occurrence of the flame is determined when the development direction deviates from the normal no-flame situation.

[0051] In other embodiments, the training of the deep learning model using the collected image frame sequence includes: The collected image frame sequence is divided into a training set and a test set; The deep learning model includes a neural network model or a Bayesian classifier, and the neural network model or the Bayesian classifier is trained through the training set; The accuracy of the neural network model or the Bayesian classifier is verified through the test set; The training of the neural network model or the Bayesian classifier through the training set includes: The weight and bias value parameters of the neural network model or the Bayesian classifier are initialized; The training set is input to the multi-dimensional data input layer of the neural network model or the Bayesian classifier according to the forward propagation algorithm; The training set is feature extracted by the feature extraction layer of the neural network model or the Bayesian classifier; The neural network model or the Bayesian classifier is used for flame target recognition judgment on the training set after feature extraction. The neural network model or the Bayesian classifier is used for flame target recognition judgment on the training set after feature extraction. A loss function is obtained, and the difference between the output flame target recognition result of the neural network model or the Bayesian classifier and the real cell state result is compared through the loss function to obtain a loss value of the loss function. The derivative of the loss function with respect to the weight and bias value parameters of the neural network model or the Bayesian classifier is calculated layer by layer from back to front through the error back propagation algorithm and the chain rule. The derivative of the loss function with respect to the weight and bias value parameters of the neural network model or the Bayesian classifier is calculated layer by layer from back to front through the error back propagation algorithm and the chain rule. The weight and bias value parameters of the neural network model or the Bayesian classifier are optimized to reduce the loss value of the loss function. The accuracy of the neural network model or the Bayesian classifier is verified through the test set, including: The accuracy of the neural network model or the Bayesian classifier is verified through the test set, including:

[0052] As shown in Figure 3 , the forward prediction model inputs an image frame sequence and outputs a predicted image sequence of the next several frames. Taking k=2 and the sequence length as 3 as an example, the input is , and the output is . It should be noted that the output here can be a model that directly predicts multiple time image frames at once, or it can be an image frame predicted at the next time and repeated iteratively, for example, is predicted to obtain , and then is predicted to obtain , until the required future image sequence is obtained.

[0053] Subsequently, in the backward prediction model, the predicted image obtained in the forward prediction model is used as input, that is, the above input , to obtain the estimate of the past frame. For example: …….

[0054] At the same time, in the training stage, we also use the real sequence to perform backward prediction, for example: ……。

[0055] In the training, we use the absolute value loss function L1 and the square loss function MSE to calculate the loss of forward prediction and backward prediction respectively as the supervision information of model training.

[0056] In the inference, we still use the similarity formula in method one to evaluate whether the flame phenomenon occurs or will occur.

[0057] For the flame detection, we use to make a judgment, and for the flame warning, we evaluate through the result of the backward prediction model combined with the similarity function, as follows: ; Set the second threshold as θ1 and the third threshold as θ2, where θ1> θ2; when , it is determined that the flame occurs at the current time; when , it is determined that the flame will occur in the future time of the energy storage device, giving a warning; when , it is determined that the current time is normal.

[0058] In the above flame warning method, by setting a deep learning model including a forward prediction model and a backward prediction model, using the trained deep learning model to output a prediction image, and comparing the prediction image with the real image, when the prediction image deviates from the real image, it is judged that the real image deviates from the development direction of the normal no-flame situation, so that when the real image deviates from the development direction of the normal no-flame situation, it is judged that the flame occurs, which can detect the flame in the production process of the energy storage device and give a warning before the flame occurs, reduce the occurrence rate of the flame, and improve the safety.

[0059] In an embodiment, a computer device, which can be a server, is provided, and the internal structure diagram thereof can be as shown in Figure 5 . The computer device includes a processor, a memory, a network interface and a database connected through a system bus. Among them, the processor of the computer device is used to provide computing and control ability. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The database of the computer device is used to store flame warning data. The network interface of the computer device is used to communicate with the external terminal through the network connection. The computer program is executed by the processor to realize a flame warning method.

[0060] Those skilled in the art can understand, Figure 5The structure shown in the figure is only a block diagram of part of the structure related to the scheme of the present application, and does not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific computer device can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0061] In one embodiment, a computer device is provided, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, the processor implementing the following steps when executing the computer program: obtaining a first time thermal imaging picture of all cameras facing a target detection area where the energy storage device is located, obtaining a corresponding positional relationship of all cameras, obtaining a first thermal imaging picture group of a plurality of image acquisition devices facing the position where the target energy storage device is located, and generating a panoramic view of the target energy storage device according to the first thermal imaging picture group and the relative positions of the plurality of image acquisition devices; collecting a second thermal imaging picture group through the plurality of image acquisition devices at regular time intervals to generate a target detection area, and determining a flame score value in the target detection area according to the target detection area and the second thermal imaging picture group; in response to the flame score value being greater than a first threshold value, recording a thermal imaging video of the target energy storage device through the plurality of image acquisition devices, and converting the thermal imaging video into a thermal imaging picture stream arranged in time sequence; setting a deep learning model including a forward prediction model and a backward prediction model, and inputting the thermal imaging picture stream into the trained deep learning model; calling the deep learning model to determine whether there is a predicted image corresponding to the thermal imaging picture stream at the moment, if not, obtaining the thermal imaging picture stream at the next moment as the real image at the next moment, and if so, taking the thermal imaging picture stream as the real image, and determining a distance score according to the predicted image and the real image; comparing the distance score with an image frame change threshold value to determine whether an abnormal change occurs, and giving a warning when an abnormal change occurs.

[0062] For specific limitations of the steps implemented by the processor when executing the computer program, refer to the limitations of the method for flame warning in the foregoing, which will not be repeated here.

[0063] In one embodiment, a computer readable storage medium is provided, having a computer program stored thereon, the computer program being executable by a processor to implement the following steps: acquire a first thermal imaging picture of all cameras facing a target detection area where the energy storage device is located, acquire a corresponding position relationship of all cameras, acquire a first thermal imaging picture group of a plurality of image acquisition devices facing a position where the target energy storage device is located, and generate a panoramic view of the target energy storage device according to the first thermal imaging picture group and the relative positions of the plurality of image acquisition devices; acquire a second thermal imaging picture group through the plurality of image acquisition devices in a timely manner to generate a target detection area, and determine a fire score value in the target detection area according to the target detection area and the second thermal imaging picture group; in response to the fire score value being greater than a first threshold value, record a thermal imaging video of the target energy storage device through the plurality of image acquisition devices, and convert the thermal imaging video into a thermal imaging picture stream arranged in time sequence; set a deep learning model including a forward prediction model and a backward prediction model, and input the thermal imaging picture stream into the trained deep learning model; invoke the deep learning model to determine whether a predicted image exists at a moment corresponding to the thermal imaging picture stream, if not, acquire a thermal imaging picture stream at a next moment as a real image at the next moment, and if so, acquire the thermal imaging picture stream as a real image, and determine a distance score according to the predicted image and the real image; determine whether an abnormal change occurs according to a comparison between the distance score and an image frame change threshold value, and give a warning when the abnormal change occurs.

[0064] For specific limitations of the computer program when executed by a processor to implement steps, refer to the above limitations of the method for fire warning, which will not be repeated here.

[0065] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer readable storage medium, and when the computer program is executed, the processes of the above-mentioned embodiments of the methods can be included. Any reference to memory, storage, databases, or other media in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), Synchlink DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct RAMbus dynamic RAM (DRDRAM), and Rambus dynamic RAM (RDRAM), etc.

[0066] The technical features of the above embodiments can be combined in any way. In order to make the description simple, not all possible combinations of the technical features in the above embodiments are described, but as long as the combinations of the technical features do not exist, they should be considered as the scope of the present application.

[0067] The above embodiments only express several implementation manners of the present application, and the description is more specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled in the art, without departing from the concept of the present application, some modifications and improvements can be made, which are all within the scope of the present application. Therefore, the scope of the patent protection of the present application should be subject to the appended claims.

Claims

1. A flame early warning method, characterized in that: include: Acquire a first thermal imaging picture group of a plurality of image acquisition devices facing a location of a target energy storage device, and generate a panoramic image of the target energy storage device based on the first thermal imaging picture group and the relative positions of the plurality of image acquisition devices; regularly capturing a second thermal imaging picture group through the plurality of image capturing devices to generate a target detection area, and determining a flame score value within the target detection area based on the target detection area and the second thermal imaging picture group; In response to the flame score being greater than a first threshold, recording a thermal imaging video of the target energy storage device through the multiple image acquisition devices, and converting the thermal imaging video into a thermal imaging picture stream arranged in chronological order; Setting a deep learning model including a forward prediction model and a backward prediction model, and inputting the thermal imaging image stream into the trained deep learning model; Calling the deep learning model to determine whether a predicted image exists at the moment corresponding to the thermal imaging image stream; if not, obtaining the thermal imaging image stream at the next moment as the real image at the next moment; if so, using the thermal imaging image stream as the real image, and determining a distance score based on the predicted image and the real image; The distance score is compared with the image frame change threshold to determine whether an abnormal change occurs, and an early warning is given when an abnormal change occurs.

2. The flame early warning method according to claim 1, characterized in that: The step of acquiring a first thermal imaging picture group of a plurality of image acquisition devices facing the location of the target energy storage device and generating a panoramic image of the target energy storage device according to the first thermal imaging picture group and the relative positions of the plurality of image acquisition devices includes: Performing grayscale processing, Gaussian blur processing, and local binary feature processing on all thermal imaging pictures acquired at the same time by the multiple image acquisition devices to obtain a first thermal imaging picture group; performing overlap processing on all thermal imaging pictures in the first thermal imaging picture group and generating the panoramic picture; The point coordinates of all pixel points in the panoramic image are calculated according to the positional relationship corresponding to all cameras and the point coordinates are stored in a coordinate library.

3. The flame early warning method according to claim 1, characterized in that: The step of periodically acquiring a second thermal imaging picture group through the plurality of image acquisition devices to generate a target detection area, and determining a flame score value within the target detection area based on the target detection area and the second thermal imaging picture group includes: performing preprocessing and local binary feature processing on the thermal imaging images in the second thermal imaging image group; Performing feature recognition on the processed thermal imaging images in the second thermal imaging image group and outputting a first feature map; Divide the first feature map into regions and output target recommended regions; Extracting target recommendation area features from the first feature map and outputting a second feature map; Matching the point coordinates of the pixel points in the second feature map with the point coordinates of all the pixel points in the panoramic image, obtaining the position of the target detection area in the panoramic image, and segmenting the second feature map into a plurality of target segmentation areas according to the temperature distribution within the target detection area; Perform flame target recognition in each target segmentation area and output target frame, target classification and target score; The flame score value of each target segmented area is obtained, and the flame score value of the flame appearing in the target detection area is obtained by weighted summation.

4. The flame early warning method according to claim 1, characterized in that: Calling the deep learning model to determine whether a predicted image exists at a moment corresponding to the thermal imaging image stream, and if not, obtaining the thermal imaging image stream at the next moment as the real image at the next moment; if so, using the thermal imaging image stream as the real image, and determining a distance score based on the predicted image and the real image includes: Setting a real image sequence and a predicted image sequence in the deep learning model, wherein the real image sequence is used to store real images, and the predicted image sequence is used to store predicted images; In response to the deep learning model receiving the thermal imaging picture stream, the thermal imaging picture stream is stored as a real image in the real image sequence, the time t corresponding to the real image is obtained, and the real image is represented as f t ; The deep learning model is based on the real image f t Prediction image at prediction time t+1 , the predicted image Storing in the predicted image sequence; Determining whether a predicted image exists in the predicted image sequence at time t; In response to the absence of a predicted image at time t in the predicted image sequence, acquiring a thermal imaging picture stream at time t+1 as the thermal imaging picture stream at time t+1; In response to the presence of a predicted image at time t in the predicted image sequence, the predicted image at time t is obtained , through the formula Calculate the distance score between the predicted image and the real image at time t, where λ is the weight parameter, express The second norm of .

5. The flame early warning method according to claim 1, characterized in that: The determining whether an abnormal change occurs based on the comparison of the distance score with the image frame change threshold, and giving an early warning when an abnormal change occurs includes: Setting the image frame change threshold to include a second threshold and a third threshold, wherein the second threshold is greater than the third threshold; When the distance score is greater than the second threshold, it is determined that a flame has occurred at the current moment; When the distance score is between the second threshold and the third threshold, it is determined that the energy storage device is about to catch fire at a future moment, and an early warning is issued; When the distance score is smaller than the third threshold, it is determined that the current moment is normal.

6. The flame early warning method according to claim 1, characterized in that: The determining whether an abnormal change occurs based on the comparison of the distance score with the image frame change threshold, and giving an early warning when an abnormal change occurs includes: Obtaining a maximum distance score between the predicted image and the real image; When the maximum value of the distance score is greater than the image frame change threshold, it is determined that a fire has occurred at the current moment and an early warning is given.

7. The flame early warning method according to claim 1, characterized in that: Before inputting the thermal imaging image stream into the trained deep learning model, the method further includes: The forward prediction model and the backward prediction model are both composed of an encoder and a decoder, the forward prediction model is used to predict a future image at a future sampling moment based on a historical image, and the backward prediction model is used to predict a historical image at a previous moment based on a future image; Training the deep learning model using the acquired image frame sequence; When training the forward prediction model, an image frame sequence from the acquired image frame sequence is input into the forward prediction model, the forward prediction model outputs a predicted image sequence that predicts a number of frames at a subsequent moment, a real image sequence corresponding to the predicted image sequence from the acquired image frame sequence is obtained, a loss of the forward prediction model is calculated using an absolute value loss function, and whether the forward prediction model has completed training is determined based on the value of the absolute value loss function; When training the backward prediction model, an image frame sequence in the acquired image frame sequence is input into the backward prediction model, the backward prediction model outputs a predicted image sequence that predicts several frames at the previous moment, and a real image sequence corresponding to the predicted image sequence in the acquired image frame sequence is obtained. The loss of the backward prediction model is calculated using a square loss function, and whether the backward prediction model has completed training is determined based on the value of the square loss function.

8. The flame early warning method according to claim 7, characterized in that: The training of the deep learning model using the acquired image frame sequence further includes: Obtaining, by a discriminator, a predicted image at a target moment output by the decoder and a real image at the target moment in the acquired image frame sequence, comparing the predicted image at the target moment with the real image, and recording a first value if they are the same, and a second value if they are different; Whether the deep learning model has completed training is determined based on the ratio of the first value to the second value.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 8 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images