Event detection method and device based on multi-source sensor fusion and storage medium

Abstract

The invention discloses an event detection method and device based on multi-source sensor fusion and a storage medium. The method comprises the following steps: acquiring a to-be-detected video frame sequence collected by each of a plurality of visual sensors in a reactor test scene; respectively carrying out clustering processing on each to-be-detected video frame sequence to obtain an event image frame sequence; a preset event detection model is obtained, the preset event detection model comprises a classification network and an event detection network, the residual error backbone network comprises different multi-scale convolutional layers, and the aggregation network comprises a plurality of fusion layers; performing defect image classification on target frame images corresponding to different sensors through a classification network, extracting image features of the target frame images under different defect image categories through a residual error backbone network, and fusing image feature vectors under the corresponding defect image categories through fusion layers in an aggregation network; and inputting each image fusion feature vector into an event detection network to determine a detected target event in the target frame image.

Description

Technical Field

[0001] The present invention relates to the field of event detection technology, and in particular to an event detection method, device and storage medium based on multi-source sensor fusion. Background Art

[0002] Event detection is used in many fields, such as reactor testing, pipeline high-temperature fluid transmission, etc., to ensure the effective and orderly progress of the processes in each field.

[0003] Currently, event detection is usually performed based on images collected by a single visible light sensor. However, since visible light sensors are sensitive to light, when images are collected under extreme conditions such as fog, rain, and dense obstacles, the information provided by the images is limited, or even no useful information can be extracted, resulting in low event detection accuracy. Summary of the invention

[0004] The present invention provides an event detection method, device and storage medium based on multi-source sensor fusion, which are mainly capable of improving the detection accuracy of events.

[0005] According to a first aspect of the present invention, there is provided an event detection method based on multi-source sensor fusion, comprising:

[0006] Acquire a sequence of video frames to be detected in a reactor test scene collected by each sensor of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of a visible light sensor, an infrared sensor, a thermal imaging sensor, an ultrasonic sensor, and a depth sensor;

[0007] Performing clustering processing on each frame image in each of the to-be-detected video frame sequences respectively to obtain an event image frame sequence in each of the to-be-detected video frame sequences, wherein each frame image in the event image frame sequence contains any target event to be detected;

[0008] Obtaining a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images for each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images;

[0009] Any frame image in each of the event image frame sequences is used as a target frame image, and the target frame images of corresponding spatiotemporal positions corresponding to different sensors are input into a preset event detection model together. The target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories. The image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers. The image feature vectors under the corresponding defect image categories are fused through each of the fusion layers in the aggregation network to obtain image fusion feature vectors output by each of the fusion layers. Each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.

[0010] Optionally, clustering each frame image in each of the to-be-detected video frame sequences to obtain an event image frame sequence in each of the to-be-detected video frame sequences comprises:

[0011] Among different sensors, a sensor that can capture images of key areas is determined as a reference sensor, an optical flow feature extraction algorithm is used to extract optical flow features of each frame image in a reference video frame sequence corresponding to the reference sensor, and a scale-invariant feature extraction transformation algorithm is used to extract local features of each frame image in the reference video frame sequence;

[0012] horizontally splicing the optical flow features and the local features to obtain a splicing feature vector corresponding to each frame image in the reference video frame sequence;

[0013] Initializing centroid vectors corresponding to different clusters, respectively, wherein the different clusters include event clusters and non-event clusters;

[0014] Calculating the cosine similarity between the splicing feature vector corresponding to each frame image in the reference video frame sequence and the centroid vectors corresponding to the different clusters, and dividing each frame image in the reference video frame sequence into the different clusters based on the cosine similarity corresponding to the different clusters;

[0015] Based on the spliced ​​feature vectors corresponding to the frames of the different clusters, obtaining updated centroid vectors corresponding to the different clusters;

[0016] Based on the updated centroid vector, re-dividing each frame image in the reference video frame sequence into the different clusters until the updated centroid vector does not change, and determining the images finally divided into different clusters as frames of images under different clustering categories, wherein the different clustering categories include event categories and non-event categories, and the event category includes at least one of a steam leakage event category, a leakage event category, a personnel breaking into a sample reactor test scene event category, and a high temperature event category;

[0017] Determine each frame image under the event category as an event image frame sequence, and determine each frame image under the non-event category as a non-event image frame sequence;

[0018] Any sensor except the reference sensor among different sensors is taken as a sensor to be clustered, and the clustering category of each frame image corresponding to the reference sensor is taken as a benchmark, and the images of the same clustering category are clustered for each frame image in the sensor to be clustered, so as to obtain each frame image under different clustering categories in the sensor to be clustered.

[0019] Optionally, after clustering each frame image in each of the to-be-detected video frame sequences to obtain an event image frame sequence in each of the to-be-detected video frame sequences, the method further includes:

[0020] The first image in the event image frame sequence is selected as a reference image, and other frame images in the event image frame sequence except the first image are determined as images to be registered, and target features are extracted from the reference image and the image to be registered respectively, wherein the target features include: at least one of feature points, feature lines, and feature regions;

[0021] Matching the target features in the image to be registered with the target features in the reference image to obtain a feature pair;

[0022] Determining a spatial transformation algorithm from the image to be registered to the reference image based on the feature pair;

[0023] The image to be registered is transformed to the same spatial position as the reference image by using the spatial transformation algorithm, so as to obtain each frame image after registration in each event image frame sequence;

[0024] For each registered frame image in each event image frame sequence, determine a target area having features similar to those of each registered frame image in a preset spatial area corresponding to each registered frame image, so as to obtain a spatial domain similarity group corresponding to each registered frame image;

[0025] Sort the pixel values ​​of the images in the spatial domain similarity group respectively to obtain sorted pixel values;

[0026] Determine an intermediate pixel value among the sorted pixel values, and use the intermediate pixel value to replace each pixel value in a corresponding area in each of the registered frame images, so as to obtain each frame image after filtering in each of the event image frame sequences;

[0027] Based on the brightness information and pixel distribution information in each frame of the filtered image, color correction and contrast enhancement are performed on each frame of the filtered image to obtain a denoised event image frame sequence;

[0028] The taking any frame image in each of the event image frame sequences as the target frame image respectively comprises:

[0029] Any frame image in each of the event image frame sequences after denoising is used as a target frame image.

[0030] Optionally, the different defect image categories include occluded image categories, blurred image categories, and thermal cross image categories; the different multi-scale convolutional layers include an occluded image multi-scale convolutional layer corresponding to the occluded image category, a blurred image multi-scale convolutional layer corresponding to the blurred image category, and a thermal cross image multi-scale convolutional layer corresponding to the thermal cross image category;

[0031] The image features of the target frame images under different defect image categories are extracted by the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each multi-scale convolutional layer, including:

[0032] The target frame image under the occluded image category is input into the occluded image multi-scale convolution layer to obtain multiple occluded image feature vectors of different scales, the target frame image under the blurred image category is input into the blurred image multi-scale convolution layer to obtain multiple blurred image feature vectors of different scales, and the target frame image under the thermal cross image category is input into the thermal cross image multi-scale convolution layer to obtain multiple thermal cross image feature vectors of different scales.

[0033] Optionally, each of the fusion layers includes an occlusion image fusion layer corresponding to the occlusion image multi-scale convolution layer, a fuzzy image fusion layer corresponding to the fuzzy image multi-scale convolution layer, and a thermal cross image fusion layer corresponding to the thermal cross image multi-scale convolution layer;

[0034] The step of fusing the image feature vectors of the corresponding defective image categories through the fusion layers in the aggregation network to obtain the image fusion feature vectors output by the fusion layers includes:

[0035] The multiple occluded image feature vectors of different scales output by the occluded image multi-scale convolution layer are input into the occluded image fusion layer for feature fusion to obtain the occluded image fusion characteristic vector; the multiple blurred image feature vectors of different scales output by the blurred image multi-scale convolution layer are input into the blurred image fusion layer for feature fusion to obtain the blurred image fusion characteristic vector; the multiple thermal cross image feature vectors of different scales output by the thermal cross image multi-scale convolution layer are input into the thermal cross image fusion layer for feature fusion to obtain the thermal cross image fusion characteristic vector.

[0036] Optionally, before obtaining the preset event detection model, the method further includes:

[0037] Construct multiple preset initial event detection models;

[0038] Acquire a sample data set, wherein the sample data set includes sample images in a sample reactor test scene collected by multiple sensors, and actual event detection results corresponding to the sample images, wherein the actual event detection results include event locations and event types, and the event types include steam leakage events, electric leakage events, personnel intrusion into the sample reactor test scene events, and high temperature events;

[0039] Based on the number of models of the preset initial event detection model, the sample data set is divided into a plurality of training data sets and a plurality of test data sets, and each group of training data sets is used to train the corresponding preset initial event detection model to obtain each trained preset initial event detection model, wherein, in the training process of the preset initial event detection model, the sample training images of the sample reactor test scene of the training data set are used as input data, and the actual event detection results corresponding to the sample training images are used as output data;

[0040] Using each group of test data sets to test the corresponding trained preset initial event detection model, to obtain the event detection accuracy corresponding to each trained preset initial event detection model;

[0041] Based on the event detection accuracy, a preset event detection model is determined among the trained preset initial event detection models.

[0042] Optionally, the method further comprises:

[0043] Taking the last event image in each of the to-be-detected video frame sequences as a reference frame image, determining an event feature vector of the target event to be detected in each of the reference frame images;

[0044] Cross-processing each of the event feature vectors to obtain a cross-event feature vector;

[0045] After the to-be-detected video frame sequence is acquired, a real-time target frame image acquired by a reference sensor in each of the sensors is acquired, wherein the real-time target frame image is a next frame image corresponding to the reference frame image, and the reference sensor is a sensor capable of acquiring a complete and clear real-time target frame image in a reactor test scenario;

[0046] Based on the intersection event feature vector, target event detection is performed on the real-time target frame image, and a circumscribed rectangular frame of the target event is drawn in the real-time target frame image;

[0047] Based on the coordinate information of the circumscribed rectangular frame, determining adjustment parameters of the remaining adjustable sensors in each sensor except the reference sensor, wherein the adjustment parameters include an adjustment angle and an adjustment direction;

[0048] Based on the adjustment parameters, the remaining adjustable sensors are adjusted, and the target event is tracked using the reference sensor and the adjusted remaining adjustable sensors.

[0049] According to a second aspect of the present invention, there is provided an event detection device based on multi-source sensor fusion, comprising:

[0050] A first acquisition unit is used to acquire a video frame sequence to be detected in a reactor test scene collected by each sensor of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of a visible light sensor, an infrared sensor, a thermal imaging sensor, an ultrasonic sensor, and a depth sensor;

[0051] A clustering unit, used for performing clustering processing on each frame image in each of the to-be-detected video frame sequences, respectively, to obtain an event image frame sequence in each of the to-be-detected video frame sequences, wherein each frame image in the event image frame sequence contains any target event to be detected;

[0052] a second acquisition unit, configured to acquire a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images for each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images;

[0053] An event detection unit is used to take any frame image in each of the event image frame sequences as a target frame image, and input the target frame images of corresponding spatiotemporal positions corresponding to different sensors into a preset event detection model together; the target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories; the image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers; each image feature vector under the corresponding defect image category is respectively fused through each of the fusion layers in the aggregation network to obtain the image fusion feature vectors output by each of the fusion layers; each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.

[0054] According to a third aspect of the present invention, there is provided a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above event detection method based on multi-source sensor fusion.

[0055] According to a fourth aspect of the present invention, there is provided a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the above event detection method based on multi-source sensor fusion when executing the program.

[0056] According to an event detection method, device and storage medium based on multi-source sensor fusion provided by the present invention, compared with the current method of performing event detection based on images collected by a single visible light sensor, the present invention obtains a video frame sequence to be detected in a reactor test scene collected by each of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors and depth sensors; and clusters each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; and then obtains a preset event detection model, wherein the preset event detection model includes a classification network for defect image classification of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual The backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images; finally, any frame image in each of the event image frame sequences is used as a target frame image, and the target frame images of corresponding spatiotemporal positions corresponding to different sensors are input into a preset event detection model together; the target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories; the image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers; each image feature vector under the corresponding defect image category is fused through each of the fusion layers in the aggregation network to obtain the image fusion feature vectors output by each of the fusion layers; each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.Therefore, by clustering the video frame sequence to be detected in advance, the event frame sequence containing any target event is determined, and then the detected target event is directly determined in the event frame sequence, which can avoid the time wasted in event detection of non-event frame sequences that do not contain target events, so that the present invention can improve the detection efficiency of events. In addition, the present invention adopts a special model architecture, including a classification network suitable for defect image classification, a residual backbone network suitable for targeted feature extraction of different defect images, an aggregation network suitable for targeted fusion of image features of different defect images, and an event detection network, which can realize targeted multi-scale feature extraction and fusion of different defect images collected by multiple sensors, avoiding interference from defects in the image. At the same time, multi-scale feature extraction enables the model to capture detailed information and global information in the image, thereby improving the feature extraction accuracy and feature fusion accuracy of the image, and further improving the detection accuracy of the target event. At the same time, the present invention performs target event detection by performing comprehensive analysis on images collected by multiple visual sensors, which can avoid the problem that the image information collected by a single sensor is limited or even the image features cannot be extracted, so that the present invention can further improve the detection accuracy of the target event. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of this application. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0058] Figure 1 A flow chart of an event detection method based on multi-source sensor fusion provided by an embodiment of the present invention is shown;

[0059] Figure 2 Another flow chart of an event detection method based on multi-source sensor fusion provided by an embodiment of the present invention is shown;

[0060] Figure 3 A schematic diagram of the structure of an event detection device based on multi-source sensor fusion provided by an embodiment of the present invention is shown;

[0061] Figure 4 A schematic diagram of the structure of another event detection device based on multi-source sensor fusion provided by an embodiment of the present invention is shown;

[0062] Figure 5 A schematic diagram of the physical structure of a computer device provided by an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0063] The present invention will be described in detail below with reference to the accompanying drawings and in combination with embodiments. It should be noted that the embodiments and features in the embodiments of the present application can be combined with each other without conflict.

[0064] Currently, the method of detecting events based solely on images collected by a single visible light sensor has low event detection accuracy because the images collected by a single sensor provide limited information and cannot even extract useful information.

[0065] In order to solve the above problems, the embodiments of the present invention provide an event detection method, device and storage medium based on multi-source sensor fusion, such as Figure 1 As shown, the method includes:

[0066] 101. Obtain a sequence of video frames to be detected in a reactor test scene collected by each sensor of multiple visual sensors.

[0067] Among them, the multiple visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, depth sensors, etc.

[0068] According to the embodiment of the present invention, multiple visual sensors such as visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors are pre-installed in the reactor test scene. The multiple visual sensors are used to collect videos of the reactor test process to obtain video frame sequences to be detected in the reactor test scene respectively collected by the multiple visual sensors. The target events in the images are detected by performing a comprehensive analysis on the images in the video frame sequences collected by the multiple visual sensors. The target events are then notified to the staff, thereby correcting events such as steam leaks that occur during the reactor test. The embodiment of the present invention detects target events such as steam leaks and leakage that occur during the reactor test by performing a comprehensive analysis on the images collected by the multiple visual sensors, thereby avoiding the problem of incomplete information in images collected by a single sensor. Thus, the embodiment of the present invention can improve the detection accuracy of events.

[0069] 102. Perform clustering processing on each frame image in each to-be-detected video frame sequence to obtain an event image frame sequence in each to-be-detected video frame sequence.

[0070] Each frame image in the event image frame sequence contains any target event to be detected, and the any target event includes at least one of a steam leakage event, a leakage event, a personnel intrusion event, etc. occurring in the reactor during the test.

[0071] For the embodiment of the present invention, after obtaining the video frame sequence to be detected of the reactor collected by each sensor in the test process of multiple visual sensors, each video frame sequence to be detected includes multiple frames of images, and each frame image includes an image with a target event and an image without a target event. In order to perform event detection only on images with target events, it is necessary to perform clustering processing on each frame image in each video frame sequence to be detected to obtain an event image frame sequence and a non-event image frame sequence with target events corresponding to each sensor. After that, the target event is detected only based on the event image frame sequence, thereby avoiding the time and resources wasted in performing target detection on a non-event image frame sequence without a target event. Therefore, the embodiment of the present invention can improve the event detection efficiency and save event detection resources.

[0072] 103. Obtain a preset event detection model, wherein the preset event detection model includes a classification network for classifying defective images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network includes different multi-scale convolutional layers for extracting image features of different defective images, and the aggregation network includes multiple fusion layers for fusing image features of different defective images.

[0073] For the embodiments of the present invention, by optimizing the model structure, constructing a targeted event detection model and using a specific data set for training, a preset event detection model can be obtained, which significantly improves the detection capability of target events in reactor test scenarios, especially the detection capability of target events in images collected by multiple sensors. The preset event detection model consists of four parts: a classification network, a residual backbone network, an aggregation network and an event detection network. Classification network: Since the reactor test scenario is complex and there are a variety of test devices, and the acquisition effects of different sensors are different, there will be defects such as occlusion, oversaturation, motion blur, noise, and deformation in the images collected by different sensors. Taking steam leakage as an example, the steam may be partially blocked by the pipeline, the sensor platform may shake, causing the steam image to be blurred, and the steam heat transfer may cause the target foreground and background temperatures to be similar, forming image thermal crossover. Based on this, it is necessary to classify the event frame image (taking the target frame image as an example) through a classification network to obtain the target frame images under different defect image categories, and then use the backbone network to classify the target frames under different defect image categories. The targeted feature extraction of the marked frame image and the targeted fusion of the image features under different defect image categories by using the aggregation network can improve the accuracy of image feature extraction and feature fusion through targeted feature extraction and targeted feature fusion; Backbone network: Different multi-scale convolutional layers under different defect image categories are used. These convolutional layers can perform targeted feature extraction on different defect images. At the same time, these convolutional layers can not only effectively extract the features of the image, but also enhance the adaptability of the model to the change of event morphology by introducing deformable convolution, which is especially suitable for the scenario of the constantly changing shape of steam in the reactor test scene. Aggregation network: Contains multiple feature fusion layers, which can perform targeted fusion processing on the image features of different scales of different defect images to capture the details and global information in the image at the same time, which is crucial for event detection based on multi-sensor images. Event detection network: Target event detection is performed by comprehensively analyzing the multiple fused feature vectors output by the aggregation network to ensure accurate recognition of targets at multiple scales. The dataset used to train the model is designed specifically for reactor test scenarios and contains images collected by multiple visual sensors, which helps the model learn how to effectively detect target events in multiple sensor images in reactor test scenarios.

[0074] 104. Any frame image in each event image frame sequence is used as the target frame image, and the target frame images of corresponding spatiotemporal positions corresponding to different sensors are input into the preset event detection model together. The target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories. The image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each multi-scale convolution layer. The image feature vectors under the corresponding defect image category are fused through each fusion layer in the aggregation network to obtain the image fusion feature vectors output by each fusion layer, and each image fusion feature vector is input into the event detection network to determine the detected target event in the target frame image.

[0075] The target frame images at the corresponding time and space positions refer to images collected by different visual sensors at the same time and space. The images collected by different visual sensors at the same time and space are determined as target frame images.

[0076] For the embodiment of the present invention, the target frame images of the corresponding spatiotemporal positions collected by different sensors are input into the trained preset event detection model for image classification, targeted feature extraction, targeted feature fusion, and target event detection. Image classification: The target frame images of the corresponding spatiotemporal positions collected by different sensors are input into the classification network, and the target frame images under different defect image categories can be output through the classification network. Targeted feature extraction: The multi-scale convolutional layers corresponding to different defect image categories in the backbone network are used to perform multi-scale feature extraction on the target frame images under the corresponding defect image categories, and image feature vectors of multiple levels under different defect image categories are obtained. Targeted feature fusion: The fusion layers corresponding to different defect image categories in the backbone network are used to perform fusion processing on the image feature vectors of multiple levels under the corresponding defect image categories, and fusion features with richer information under different defect image categories are obtained. Target event detection: The event detection network is used to comprehensively analyze different fusion features to determine the detected target event in the target frame image, that is, the event detection network is used to comprehensively analyze different fusion features to determine the detected target event in the target frame image corresponding to different sensors. Therefore, the embodiment of the present invention detects target events in the reactor test scenario by comprehensively analyzing images collected by multiple visual sensors, which can avoid the problem that the image information collected by a single sensor is incomplete or useful features cannot be extracted, so that the embodiment of the present invention can improve the accuracy of event detection. Secondly, the event detection model designed by the embodiment of the present invention adds a classification network, a targeted feature extraction network and a targeted feature fusion network. The classification network can accurately classify defect images collected by different sensors, so that the multi-scale convolution layers corresponding to different defect image categories in the backbone network can be used to extract features of the corresponding defect images respectively. For example, there are multi-scale convolution layers in the backbone network that are specifically used for feature extraction of blurred images, and there are multi-scale convolution layers that are specifically used for feature extraction of thermal cross images. By setting targeted multi-scale feature extraction convolution layers for different defect images in the backbone network, the accuracy of feature extraction can be improved. At the same time, a targeted feature fusion network is set in the aggregation network for image features under different defect images, which can improve the accuracy of feature fusion, so that more implicit information can be more accurately extracted in the target frame image, thereby improving the detection accuracy of the target event.

[0077] In another embodiment of the present invention, the feature extraction process of the residual backbone network can be expressed by the following formula:

[0078] R a =δ(f a (F5)

[0079] Among them, f aThe convolutional layer representing attribute a (defect image category a), R a is the image feature output by the a-attribute convolutional layer), C is the number of defined explicit target attributes (the number of defect image categories), H, W are the set image sizes output by the convolutional layer, F5 is the image under the a-attribute, and δ is the activation function.

[0080] According to an event detection method based on multi-source sensor fusion provided by the present invention, compared with the current method of performing event detection based on images collected by a single visible light sensor, the present invention obtains a video frame sequence to be detected in a reactor test scene collected by each of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors; and clusters each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; and then obtains a preset event detection model, wherein the preset event detection model includes a classification network for classifying defective images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network It includes different multi-scale convolution layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images; finally, any frame image in each of the event image frame sequences is used as the target frame image, and the target frame images of the corresponding spatiotemporal positions corresponding to different sensors are input into the preset event detection model together; the target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories; the image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolution layers; each image feature vector under the corresponding defect image category is fused through each of the fusion layers in the aggregation network to obtain the image fusion feature vectors output by each of the fusion layers; each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.Therefore, by clustering the video frame sequence to be detected in advance, the event frame sequence containing any target event is determined, and then the detected target event is directly determined in the event frame sequence, which can avoid the time wasted in event detection of non-event frame sequences that do not contain target events, so that the present invention can improve the detection efficiency of events. In addition, the present invention adopts a special model architecture, including a classification network suitable for defect image classification, a residual backbone network suitable for targeted feature extraction of different defect images, an aggregation network suitable for targeted fusion of image features of different defect images, and an event detection network, which can realize targeted multi-scale feature extraction and fusion of different defect images collected by multiple sensors, avoiding interference from defects in the image. At the same time, multi-scale feature extraction enables the model to capture detailed information and global information in the image, thereby improving the feature extraction accuracy and feature fusion accuracy of the image, and further improving the detection accuracy of the target event. At the same time, the present invention performs target event detection by performing comprehensive analysis on images collected by multiple visual sensors, which can avoid the problem that the image information collected by a single sensor is limited or even the image features cannot be extracted, so that the present invention can further improve the detection accuracy of the target event.

[0081] Further, in order to better illustrate the process of event detection in the reactor test scenario based on multi-source sensor fusion, as a refinement and extension of the above embodiment, the embodiment of the present invention provides another event detection method, device and storage medium based on multi-source sensor fusion, such as Figure 2 As shown, the method includes:

[0082] 201. Obtain a sequence of video frames to be detected in a reactor test scene collected by each sensor of a plurality of visual sensors.

[0083] Among them, the multiple visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors.

[0084] Specifically, each visual sensor is turned on at the same time and can simultaneously collect images of the test process in the reactor test scene in real time. The video acquisition time is set, and when the set time is reached, the video frame sequence to be detected within the preset time period collected by each visual sensor is obtained, and then the target event detection is performed in each frame image in the video frame sequence to be detected.

[0085] 202. Perform clustering processing on each frame image in each to-be-detected video frame sequence to obtain an event image frame sequence in each to-be-detected video frame sequence.

[0086] For the embodiment of the present invention, in order to avoid wasting time on target detection of non-event frame images that do not contain target events, the embodiment of the present invention needs to perform clustering processing on each frame image in each frame sequence of to-be-detected video frames collected by different sensors in advance. Based on this, step 202 includes: among different sensors, determining the sensor that can collect the key area image as the reference sensor, extracting the optical flow features of each frame image in the reference video frame sequence corresponding to the reference sensor by using the optical flow feature extraction algorithm, and extracting the local features of each frame image in the reference video frame sequence by using the scale-invariant feature extraction transformation algorithm; horizontally splicing the optical flow features and the local features to obtain the splicing feature vectors corresponding to each frame image in the reference video frame sequence; initializing the centroid vectors corresponding to different clusters, wherein the different clusters include event clusters and non-event clusters; calculating the cosine similarity between the splicing feature vectors corresponding to each frame image in the reference video frame sequence and the centroid vectors corresponding to the different clusters, and dividing the frame images in the reference video frame sequence into clusters based on the cosine similarity corresponding to the different clusters. into the different clusters; based on the splicing feature vectors corresponding to each frame image in the different clusters, obtaining the updated centroid vectors corresponding to the different clusters; based on the updated centroid vectors, re-dividing each frame image in the reference video frame sequence into the different clusters until the updated centroid vector does not change, and determining the images finally divided into different clusters as each frame image under different clustering categories, wherein the different clustering categories include event categories and non-event categories, and the event categories include at least one of steam leakage event categories, leakage event categories, personnel breaking into sample reactor test scene event categories, and high temperature event categories; determining each frame image under the event category as an event image frame sequence, and determining each frame image under the non-event category as a non-event image frame sequence; taking any sensor other than the reference sensor among different sensors as a sensor to be clustered, and taking the clustering category of each frame image corresponding to the reference sensor as a benchmark, performing image clustering of the same clustering category on each frame image in the sensor to be clustered, so as to obtain each frame image under different clustering categories in the sensor to be clustered. Each frame image in the event image frame sequence contains any target event to be detected.

[0087] Among them, the key area image refers to the area with the highest probability of the target event. Specifically, the visual sensor that can collect the key area image is used as the reference sensor, and each frame image in the reference video frame sequence collected by the reference sensor is clustered in advance. First, the motion vector of the pixel or feature point is estimated by the least squares method in the local area of ​​each frame image through the optical flow feature extraction algorithm, and then the optical flow feature of each frame image is determined according to the motion vector of each pixel. At the same time, the local features of each frame image are extracted by the scale-invariant feature extraction transformation algorithm. The specific extraction process is as follows: Construct the scale space corresponding to each frame image: that is, in order to simulate the effect of human eyes observing objects at different distances, first construct the scale space by Gaussian filtering and image pyramid. The scale space consists of a series of images with different resolutions, which are obtained by Gaussian blurring and downsampling. Detect extreme points: In the scale space, find local extreme points by using the Gaussian difference function. These extreme points are potential feature points that exist at different scales. They usually correspond to stable structures such as corner points and edge points in the image. Key point positioning: The extreme points found need to be further screened and accurately positioned to remove unstable and misdetected points. Key points can be determined by comparing the difference between extreme points and their surrounding pixels, for example, selecting points with larger differences as extreme points. Screening stable points: Use thresholds and other methods to limit the number and quality of key points. For example, select key points greater than the preset threshold as stable points to ensure that the extracted key points have high stability and reliability; Stable point direction assignment: Direction histogram: In order to make the stable point invariant to image angle and rotation, one or more directions need to be assigned to each stable point. This can be achieved by calculating the gradient direction and amplitude of the pixels in the neighborhood of the stable point and creating a direction histogram. Determine the main direction and auxiliary direction: The peak of the histogram is used as the main direction of the stable point, and other directions that are higher than a certain proportion (such as 80%) of the main direction are used as auxiliary directions. Generate key point descriptors: Rotate the coordinate axis: Rotate the coordinate axis to the main direction of the stable point to ensure the rotation invariance of the descriptor. Block statistics: With the stable point as the center, the surrounding area within the preset range of the stable point is divided into several sub-blocks (such as 4x4), and the gradient direction and amplitude are counted in each sub-block to generate a gradient histogram. Form a descriptor: The gradient histograms of all sub-blocks are spliced ​​together to form a high-dimensional vector (such as 128 dimensions) as the descriptor of the stable point. Feature point matching: Calculate the Euclidean distance: For the stable point descriptors in the two images, the similarity is evaluated by calculating the Euclidean distance between them. The smaller the Euclidean distance, the more similar the two key points are. Matching threshold: Set a matching threshold. When the Euclidean distance between the two stable point descriptors is less than the threshold, they are considered to be a pair of matching points.In this way, it is possible to determine each pair of matching points, and then determine the local features of each frame image based on each pair of matching points. The embodiment of the present invention extracts the local features of the image through a scale-invariant feature extraction transformation algorithm. These features have good robustness to image rotation, scale scaling, brightness changes, etc. Then, for each frame of the image, its corresponding optical flow features and local features are horizontally spliced ​​to obtain a spliced ​​feature vector. By splicing two feature vectors, more implicit features in the image can be mined, making the subsequent clustering process more accurate. The clustering process for the image: first, select the centroid vectors corresponding to the initial centroids corresponding to K clusters, where the K clusters include: event clusters and non-event clusters, and the event clusters include: steam leakage event clusters, leakage event clusters, personnel breaking into sample reactor test scene event clusters, and high temperature event clusters. For the spliced ​​feature vectors corresponding to each frame image, calculate the cosine similarity between each spliced ​​feature vector and the K centroid vectors. Further, after calculating the cosine similarity between each frame image and the centroid vector of each cluster, assign each frame image to the cluster corresponding to the centroid vector with the largest cosine similarity. , then for each cluster, recalculate the centroid of each cluster and its corresponding centroid vector, and re-divide each frame image into different clusters, so as to continuously divide each frame image until the position of the centroid does not change, that is, the centroid vector does not change, and finally the images divided into different clusters are determined as images under different clustering themes, thereby obtaining event image frame sequences and non-event image frame sequences, and the event image frame sequences can be further divided into steam leakage event frame image sequences, leakage event frame image sequences, personnel breaking into sample reactor test scene event frame image sequences, and high temperature event frame image sequences.

[0088] Furthermore, any sensor except the reference sensor among different sensors is used as a sensor to be clustered. After clustering each frame image in the reference sensor, each frame image in the remaining sensors to be clustered also needs to be clustered. Specifically, the clustering category of each frame image in the reference sensor can be used as a benchmark to cluster each frame image in other sensors to be clustered. For example, four categories are obtained by clustering each frame image in the reference sensor, including a steam leakage event category, a leakage event category, a personnel intrusion into a sample reactor test scene event category, and a high temperature event category. Then, each frame image in the sensor to be clustered is also clustered according to the above four categories to obtain images under the four categories. In another embodiment of the present invention, the other sensors to be clustered can also be clustered based on the image frame sequence number of different categories in the reference sensor. The image frames in the sensors are clustered. For example, the clustering results of the image frames in the reference sensor are: the 1st to 3rd frames are one category, the 4th to 8th frames are one category, and the 9th to 10th frames are one category. Then, the image frames corresponding to the other sensors to be clustered are also determined as one category for the 1st to 3rd frames, one category for the 4th to 8th frames, and one category for the 9th to 10th frames. Thus, by clustering the remaining sensor images based on the clustering results of the image frames in a certain sensor, it is possible to ensure that the images in the to-be-detected video frame sequences corresponding to different sensors can be unified in terms of time and space positions. After the time and space positions are unified, the detection results of different sensors can be compared and fused in the same framework, thereby eliminating the errors caused by time difference or spatial dislocation and improving the accuracy of target event detection.

[0089] Furthermore, in order to improve the detection accuracy of target events, before performing target event detection on each frame image, it is necessary to perform denoising processing on each frame image in the event image frame sequence. Based on this, the method includes: selecting the first image in the event image frame sequence as the reference image, and determining the other frame images in the event image frame sequence excluding the first image as the images to be registered, respectively extracting target features from the reference image and the image to be registered, wherein the target features include: at least one of feature points, feature lines, and feature areas; matching the target features in the image to be registered with the target features in the reference image to obtain a feature pair; determining a spatial transformation algorithm from the image to be registered to the reference image based on the feature pair; using the spatial transformation algorithm to transform the image to be registered to the same spatial position as the reference image to obtain each registered frame image in the event image frame sequence; for For each registered frame image in each of the event image frame sequences, within the preset spatial area corresponding to each of the registered frame images, a target area with similar features to the registered frame images is determined to obtain a spatial similarity group corresponding to each of the registered frame images; the pixel values ​​of the images in the spatial similarity group are sorted respectively to obtain sorted pixel values; an intermediate pixel value is determined from the sorted pixel values, and each pixel value in the corresponding area of ​​each of the registered frame images is replaced with the intermediate pixel value to obtain each filtered frame image in each of the event image frame sequences; based on the brightness information and pixel distribution information in each of the filtered frame images, color correction and contrast enhancement are performed on each of the filtered frame images to obtain each denoised event image frame sequence; taking any frame image in each of the event image frame sequences as a target frame image includes: taking any frame image in each of the denoised event image frame sequences as a target frame image.

[0090] Specifically, it is necessary to perform inter-frame temporal and spatial filtering on the event frame image sequence to avoid interference from random changes in image feature points. The first frame image in the event frame image sequence is selected as the reference image, and the feature points, lines and regions are used to realize the registration of the remaining images to be registered. Filtering is performed in the temporal and spatial domains of each event frame, and the short-term jitter and long-term changes in the event frame image are filtered out by optimizing the smoothness energy function between the input and output frames. The expression is as follows:

[0091]

[0092] Wherein, E(J) represents the smoothness energy function of the image to be registered, I(x,y,t) is the reference image, J(x,y,t) is the output image (image to be registered), I(x,y,t+1) represents the next frame image corresponding to the reference image, x represents the horizontal coordinate of the pixel in the image, y represents the vertical coordinate of the pixel in the image, t represents the acquisition time corresponding to the image, α represents the weight coefficient of smoothness, the first term ∑|J(x,y,t)-I(x,y,t)| represents the similarity between the input and output frames of the spatiotemporal window, and the second term α∑|J(x,y,t)-I(x,y,t)| represents the smoothness between the output frames. Gradient optimization filtering is performed on the Markov random field. The brightness information and pixel distribution information in the filtered image are used to perform gamma color correction and contrast enhancement on the filtered image in turn to obtain the denoised image. Specifically, first, the first image is selected from the event frame image sequence as the benchmark image (also called the reference image). This is because in many application scenarios, such as video analysis, dynamic image processing, etc., the first image is often regarded as the starting point or standard point for comparison and registration of subsequent images. Image registration: The process of spatially transforming images acquired at different times, different perspectives or different sensors so that the corresponding points in these images are consistent in space. For event frame image registration, the main steps are as follows: Feature extraction: Extract target features such as feature points, lines or regions from the benchmark image and the image to be registered. These features are features that can be easily identified in the image and remain stable between different images. Feature matching: Match the target features in the image to be registered with the target features in the benchmark image, such as finding the corresponding feature pairs based on the similarity of the feature descriptors. Transformation model estimation: Determine the spatial transformation algorithm from the image to be registered to the benchmark image based on the matched feature pairs, such as affine transformation, rigid transformation and non-rigid transformation. Image transformation: Apply the spatial transformation model to transform the image to be registered to the same spatial position as the benchmark image to achieve image registration. Similarity group establishment: Spatial similarity group: In the preset spatial area around the reference image (the preset spatial area is set according to actual needs), find image blocks or target areas with similar features to the reference image to form a spatial similarity group. Temporal similarity group: In the time series, select image frames with similar features and similar time to the reference image to form a temporal similarity group. Filtering: Use the image information in the similarity group to filter the reference image. The filtering method can be selected according to specific needs, such as mean filtering, median filtering, Gaussian filtering, etc., to remove noise, enhance edges or smooth images. If the purpose is to improve image quality or enhance specific features, more advanced filtering techniques can be considered, such as Wiener filtering, Kalman filtering or filtering methods based on deep learning. Color correction and contrast enhancement: Based on the brightness information and pixel distribution information of the image, color correction and contrast enhancement are performed on each frame of the filtered image to obtain a denoised sequence of each event image frame.

[0093] 203. Obtain a preset event detection model, wherein the preset event detection model includes a classification network for classifying defective images for each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network includes different multi-scale convolutional layers for extracting image features of different defective images, and the aggregation network includes multiple fusion layers for fusing image features of different defective images.

[0094] For the embodiment of the present invention, in order to improve the detection accuracy of the preset event detection model for the target event, it is necessary to construct and train the preset event detection model in advance. Based on this, the method includes: constructing multiple preset initial event detection models; obtaining a sample data set, wherein the sample data set includes sample images in a sample reactor test scene collected by multiple sensors, and actual event detection results corresponding to the sample images, wherein the actual event detection results include event locations and event types, and the event types include steam leakage events, leakage events, personnel breaking into sample reactor test scene events, and high temperature events; based on the number of models of the preset initial event detection model, the sample data set is divided into multiple groups of training A data set and multiple test data sets are provided, and each set of training data sets is used to train a corresponding preset initial event detection model to obtain each trained preset initial event detection model, wherein, in the training process of the preset initial event detection model, the sample training images in the sample reactor test scene of the training data set are used as input data, and the actual event detection results corresponding to the sample training images are used as output data; each set of test data sets is used to test the corresponding trained preset initial event detection model to obtain the event detection accuracy corresponding to each trained preset initial event detection model; based on the event detection accuracy, the preset event detection model is determined in each trained preset initial event detection model.

[0095] Specifically, first, multiple preset initial event detection models are constructed. The structure of the initial event detection model includes: classification network, residual backbone network, aggregation network, and event detection network. Secondly, a sample data set is obtained, and it is ensured that the sample data set contains all necessary files, including image files collected by multiple visual sensors (the image files include blurred images, occluded images, and thermal cross images, etc.) and corresponding annotation information, and finally the model is trained and tested. Specifically, the sample data set can be divided first: according to the number of preset initial event detection models, the sample data set is divided into multiple training sets and multiple test sets using random or specific strategies (such as stratified sampling). Then, each group of training sets is used to train the corresponding preset initial event detection model, and indicators such as loss values ​​during the training process are monitored to evaluate the model performance. Model parameters such as learning rate and optimizer are adjusted as needed to optimize the training effect. Finally, each group of test sets is used to test the corresponding trained preset initial event detection model to evaluate the performance of each trained model on unseen data. Calculate and record the accuracy and other indicators on the test set, and finally select the model with the required accuracy and maximum accuracy as the preset event detection model for the final event detection. If the model performance does not meet the requirements, you can return to the training stage for more iterations or adjustments. By training and verifying multiple initial models to select an event detection model, the event detection accuracy of the final event detection model can be higher.

[0096] 204. Any frame image in each event image frame sequence is used as the target frame image, and the target frame images of corresponding spatiotemporal positions corresponding to different sensors are input into the preset event detection model together. The target frame images corresponding to different sensors are classified into defect images through a classification network to obtain target frame images under different defect image categories, wherein the different defect image categories include occluded image categories, blurred image categories, and thermal cross image categories; different multi-scale convolutional layers include occluded image multi-scale convolutional layers corresponding to occluded image categories, blurred image multi-scale convolutional layers corresponding to blurred image categories, and thermal cross image multi-scale convolutional layers corresponding to thermal cross image categories; each fusion layer includes an occluded image fusion layer corresponding to the occluded image multi-scale convolutional layer, a blurred image fusion layer corresponding to the blurred image multi-scale convolutional layer, and a thermal cross image fusion layer corresponding to the thermal cross image multi-scale convolutional layer.

[0097] 205. Input the target frame image under the occluded image category into the occluded image multi-scale convolution layer to obtain multiple occluded image feature vectors of different scales, input the target frame image under the blurred image category into the blurred image multi-scale convolution layer to obtain multiple blurred image feature vectors of different scales, and input the target frame image under the thermal cross image category into the thermal cross image multi-scale convolution layer to obtain multiple thermal cross image feature vectors of different scales.

[0098] 206. Input multiple occluded image feature vectors of different scales output by the occluded image multi-scale convolution layer into the occluded image fusion layer for feature fusion to obtain the occluded image fusion characteristic vector, input multiple blurred image feature vectors of different scales output by the blurred image multi-scale convolution layer into the blurred image fusion layer for feature fusion to obtain the blurred image fusion characteristic vector, input multiple thermal cross image feature vectors of different scales output by the thermal cross image multi-scale convolution layer into the thermal cross image fusion layer for feature fusion to obtain the thermal cross image fusion feature vector.

[0099] 207. Input the occluded image fusion characteristic vector, the blurred image fusion characteristic vector, and the thermal cross image fusion characteristic vector into an event detection network to determine a detected target event in the target frame image.

[0100] Specifically, the target frame images of the corresponding spatiotemporal positions corresponding to different sensors are input together into the classification network in the preset event detection model. Through the classification network, each input image can be classified into defect images to obtain target frame images under defect categories such as occluded image category, blurred image category, thermal cross image category, etc. Then, the target frame images under the occluded image category, the target frame images under the blurred image category, and the target frame images under the thermal cross image category are respectively input into the occluded image multi-scale convolution layer, the blurred image multi-scale convolution layer, and the thermal cross image multi-scale convolution layer in the residual backbone network. The occluded image multi-scale convolution layer can output the occluded image features of different scales corresponding to each image under the occluded category. The blurred image multi-scale convolution layer can output the blurred image features of different scales corresponding to each image under the blurred category. The thermal cross image multi-scale convolution layer can output the thermal cross image features of different scales corresponding to each image under the thermal cross category. Thermal cross image features of different scales, and then the occluded image features of different scales, the blurred image features of different scales, and the thermal cross image features of different scales are respectively input into the occluded image fusion layer, the blurred image fusion layer, and the thermal cross image fusion layer to obtain the occluded image fusion characteristic vector output by the occluded image fusion layer, the blurred image fusion characteristic vector output by the blurred image fusion layer, and the thermal cross image fusion characteristic vector output by the thermal cross image fusion layer. Further, the occluded image fusion characteristic vector, the blurred image fusion characteristic vector, and the thermal cross image fusion characteristic vector are cross-processed at the feature level to further obtain the implicit features between each fusion characteristic vector, and then the cross-processed features are input into the event detection network to determine the detected target event in the target frame image, including the location information and type information of the event, and the type information includes: steam leakage event, leakage event, personnel intrusion event, high temperature event, etc. Therefore, by classifying defective images collected by different sensors and then performing targeted feature extraction and feature fusion on images under different defective image categories, the feature extraction accuracy and feature fusion accuracy can be improved. At the same time, the multi-scale convolution layer is used to process images with different scale changes. The multi-scale convolution layer can process input images or image regions of different sizes. At the same time, the multi-scale convolution layer has different receptive fields and can capture features of different scales in the image, including local details and broader contextual information, thereby further improving the comprehensiveness and accuracy of feature extraction. At the same time, by fusing features at different scales, it is possible to comprehensively utilize information from different levels and extract implicit information at the same time, so as to more accurately understand the image content and thus improve the detection accuracy of events in the image.

[0101] According to another event detection method, device and storage medium based on multi-source sensor fusion provided by the present invention, compared with the current method of performing event detection based on images collected by a single visible light sensor, the present invention obtains a video frame sequence to be detected in a reactor test scene collected by each of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors and depth sensors; and clusters each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; and then obtains a preset event detection model, wherein the preset event detection model includes a classification network for defect image classification of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual The difference backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images; finally, any frame image in each of the event image frame sequences is used as the target frame image, and the target frame images of the corresponding spatiotemporal positions corresponding to different sensors are input into the preset event detection model together; the target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories; the image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers; each image feature vector under the corresponding defect image category is fused through each of the fusion layers in the aggregation network to obtain the image fusion feature vectors output by each of the fusion layers; each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.Therefore, by clustering the video frame sequence to be detected in advance, the event frame sequence containing any target event is determined, and then the detected target event is directly determined in the event frame sequence, which can avoid the time wasted in event detection of non-event frame sequences that do not contain target events, so that the present invention can improve the detection efficiency of events. In addition, the present invention adopts a special model architecture, including a classification network suitable for defect image classification, a residual backbone network suitable for targeted feature extraction of different defect images, an aggregation network suitable for targeted fusion of image features of different defect images, and an event detection network, which can realize targeted multi-scale feature extraction and fusion of different defect images collected by multiple sensors, avoiding interference from defects in the image. At the same time, multi-scale feature extraction enables the model to capture detailed information and global information in the image, thereby improving the feature extraction accuracy and feature fusion accuracy of the image, and further improving the detection accuracy of the target event. At the same time, the present invention performs target event detection by performing comprehensive analysis on images collected by multiple visual sensors, which can avoid the problem that the image information collected by a single sensor is limited or even the image features cannot be extracted, so that the present invention can further improve the detection accuracy of the target event.

[0102] Furthermore, if after target event detection is performed on each frame event image in the video frame sequence to be detected, if the reactor test is still ongoing, the target event in the last frame image in the video frame sequence to be detected can be used as a reference to track the target event of the real-time image in the reactor test scene that is subsequently collected. Based on this, the method includes: taking the last frame event image in each of the video frame sequences to be detected as a reference frame image, determining the event feature vector of the detected target event in each of the reference frame images; cross-processing each of the event feature vectors to obtain a cross-event feature vector; after the video frame sequence to be detected is collected, obtaining the real-time target frame image collected by the reference sensor in each sensor, wherein the real-time target frame image The image is the next frame image corresponding to the reference frame image, and the reference sensor is a sensor that can collect complete and clear real-time target frame images in a reactor test scenario; based on the cross-event feature vector, target event detection is performed on the real-time target frame image, and a circumscribed rectangular frame of the target event is drawn in the real-time target frame image; based on the coordinate information of the circumscribed rectangular frame, the adjustment parameters of the remaining adjustable sensors in each sensor excluding the reference sensor are determined, wherein the adjustment parameters include an adjustment angle and an adjustment direction; based on the adjustment parameters, the remaining adjustable sensors are adjusted, and the target event is tracked using the reference sensor and the adjusted remaining adjustable sensors.

[0103] Specifically, the last event image in each sequence of video frames to be detected is used as a reference image, and the event feature vector of the target event to be detected in each reference image is determined. Then, the event feature vectors in each reference image are cross-processed. For example, the elements at the same position in each event feature vector are multiplied, and a preset transformation coefficient (set according to actual needs) is determined. Finally, the multiplied vector is transformed using the preset transformation coefficient to obtain a cross event feature vector. When the reference sensor that can clearly and accurately capture the target event image collects the real-time target frame image during the reactor test, the event feature vector of the target event to be detected in the reference image can be used to compare with the real target frame image. The target event is detected in the real-time target frame image by similarity matching with the real-time target frame image, and an external rectangular frame is drawn for the target event. According to the coordinate position of the reference center point of the external rectangular frame in the real-time target frame image, at the same time, the verification image collected by other remaining sensors in the same time and space is obtained, and the verification center point position information of the target event is identified in the verification image. The reference position information and the verification position information are compared to determine the adjustment parameters of the remaining sensors, including translation, rotation angle, adjustment direction, etc., and the position of the remaining sensors is corrected according to the adjustment parameters. Finally, the corrected sensors and reference sensors are used to track the target event in the reactor test scenario. Therefore, by correcting the sensors whose image acquisition is unclear or incomplete, the accurate detection and precise tracking of the target event can be determined.

[0104] Further, as Figure 1 In a specific implementation, the present invention provides an event detection device based on multi-source sensor fusion, such as Figure 3 As shown, the device includes: a first acquisition unit 31, a clustering unit 32, a second acquisition unit 33, and an event detection unit 34.

[0105] The first acquisition unit 31 can be used to acquire a video frame sequence to be detected in a reactor test scene collected by each sensor of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of a visible light sensor, an infrared sensor, a thermal imaging sensor, an ultrasonic sensor, and a depth sensor.

[0106] The clustering unit 32 can be used to perform clustering processing on each frame image in each of the video frame sequences to be detected, so as to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected.

[0107] The second acquisition unit 33 can be used to acquire a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images for each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, the residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images.

[0108] The event detection unit 34 can be used to take any frame image in each of the event image frame sequences as the target frame image, and input the target frame images at corresponding spatiotemporal positions corresponding to different sensors into a preset event detection model together; perform defect image classification on the target frame images corresponding to different sensors through the classification network to obtain target frame images under different defect image categories; extract image features of the target frame images under different defect image categories through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers; fuse each image feature vector under the corresponding defect image category through each of the fusion layers in the aggregation network to obtain image fusion feature vectors output by each of the fusion layers; and input each of the image fusion feature vectors into the event detection network to determine the detected target event in the target frame image.

[0109] In a specific application scenario, in order to perform clustering on each frame of image, such as Figure 4 As shown, the clustering unit 32 includes a feature extraction module 321 , a feature splicing module 322 , an initialization module 323 , a division module 324 , a determination module 325 , and a clustering module 326 .

[0110] The feature extraction module 321 can be used to determine the sensor that can capture the key area image as the reference sensor among different sensors, use the optical flow feature extraction algorithm to extract the optical flow features of each frame image in the reference video frame sequence corresponding to the reference sensor, and use the scale-invariant feature extraction transformation algorithm to extract the local features of each frame image in the reference video frame sequence.

[0111] The feature splicing module 322 may be used to horizontally splice the optical flow features and the local features to obtain a splicing feature vector corresponding to each frame image in the reference video frame sequence.

[0112] The initialization module 323 may be used to initialize centroid vectors corresponding to different clusters, wherein the different clusters include event clusters and non-event clusters.

[0113] The division module 324 can be used to calculate the cosine similarity between the spliced ​​feature vector corresponding to each frame image in the reference video frame sequence and the centroid vector corresponding to the different clusters, and divide each frame image in the reference video frame sequence into the different clusters based on the cosine similarity corresponding to the different clusters.

[0114] The determination module 325 may be configured to obtain updated centroid vectors corresponding to the different clusters based on the spliced ​​feature vectors corresponding to the frames of images in the different clusters.

[0115] The division module 324 can be specifically used to re-divide each frame image in the baseline video frame sequence into the different clusters based on the updated centroid vector until the updated centroid vector does not change, and determine the images finally divided into different clusters as frame images under different clustering categories, wherein the different clustering categories include event categories and non-event categories, and the event categories include at least one of steam leakage event categories, leakage event categories, personnel intrusion into sample reactor test scene event categories, and high temperature event categories.

[0116] The determination module 325 may also be used to determine each frame image under the event category as an event image frame sequence, and determine each frame image under the non-event category as a non-event image frame sequence.

[0117] The clustering module 326 can be used to take any sensor except the reference sensor among different sensors as a sensor to be clustered, and based on the clustering category of each frame image corresponding to the reference sensor, perform image clustering of the same clustering category on each frame image in the sensor to be clustered, so as to obtain each frame image under different clustering categories in the sensor to be clustered.

[0118] In a specific application scenario, in order to perform denoising on each frame of image, the device further includes: a denoising unit 35 .

[0119] The denoising unit 35 can be used to select the first image in the event image frame sequence as the reference image, and determine the other frame images in the event image frame sequence excluding the first image as the images to be registered, respectively extract target features from the reference image and the image to be registered, wherein the target features include: at least one of feature points, feature lines, and feature areas; match the target features in the image to be registered with the target features in the reference image to obtain a feature pair; determine the spatial transformation algorithm from the image to be registered to the reference image based on the feature pair; transform the image to be registered to the same spatial position as the reference image using the spatial transformation algorithm to obtain each registered frame image in the event image frame sequence; and for each registered frame in the event image frame sequence For each frame image after registration, in the preset spatial area corresponding to each frame image after registration, determine the target area with similar features to each frame image after registration, so as to obtain the spatial domain similarity group corresponding to each frame image after registration; sort the pixel values ​​of the images in the spatial domain similarity group respectively to obtain the sorted pixel values; determine the intermediate pixel value among the sorted pixel values, and use the intermediate pixel value to replace each pixel value in the corresponding area in each frame image after registration, so as to obtain each frame image after filtering in each event image frame sequence; based on the brightness information and pixel distribution information in each filtered frame image, perform color correction and contrast enhancement on each filtered frame image, so as to obtain each denoised event image frame sequence; take any frame image in each denoised event image frame sequence as the target frame image.

[0120] In a specific application scenario, different defect image categories include occluded image categories, blurred image categories, and thermal cross image categories; different multi-scale convolutional layers include occluded image multi-scale convolutional layers corresponding to the occluded image categories, blurred image multi-scale convolutional layers corresponding to the blurred image categories, and thermal cross image multi-scale convolutional layers corresponding to the thermal cross image categories. Based on this, in order to extract the image feature vectors of each frame image, the event detection unit 34 can be specifically used to input the target frame image under the occluded image category into the occluded image multi-scale convolutional layer to obtain multiple occluded image feature vectors of different scales, input the target frame image under the blurred image category into the blurred image multi-scale convolutional layer to obtain multiple blurred image feature vectors of different scales, and input the target frame image under the thermal cross image category into the thermal cross image multi-scale convolutional layer to obtain multiple thermal cross image feature vectors of different scales.

[0121] In a specific application scenario, in order to fuse the image feature vectors under different defect image categories, the event detection unit 34 can be specifically used to input multiple occluded image feature vectors of different scales output by the occluded image multi-scale convolution layer into the occluded image fusion layer for feature fusion to obtain the occluded image fusion characteristic vector, input multiple blurred image feature vectors of different scales output by the blurred image multi-scale convolution layer into the blurred image fusion layer for feature fusion to obtain the blurred image fusion characteristic vector, and input multiple thermal cross image feature vectors of different scales output by the thermal cross image multi-scale convolution layer into the thermal cross image fusion layer for feature fusion to obtain the thermal cross image fusion characteristic vector.

[0122] In a specific application scenario, in order to construct a preset time detection model, the device further includes: a construction unit 36.

[0123] The construction unit 36 ​​can be specifically used to construct multiple preset initial event detection models; obtain a sample data set, wherein the sample data set includes sample images in a sample reactor test scene collected by multiple sensors, and actual event detection results corresponding to the sample images, wherein the actual event detection results include event locations and event types, and the event types include steam leakage events, leakage events, personnel intrusion into the sample reactor test scene events, and high temperature events; based on the number of models of the preset initial event detection model, divide the sample data set into multiple groups of training data sets and multiple groups of test data sets, and use each group of training data sets to train the corresponding preset initial event detection model to obtain each trained preset initial event detection model, wherein in the training process of the preset initial event detection model, the sample training images in the sample reactor test scene of the training data set are used as input data, and the actual event detection results corresponding to the sample training images are used as output data; use each group of test data sets to test the corresponding trained preset initial event detection model to obtain the event detection accuracy corresponding to each trained preset initial event detection model; based on the event detection accuracy, determine the preset event detection model in each trained preset initial event detection model.

[0124] In a specific application scenario, in order to track the target event in the reactor test scenario in real time, the device also includes a target tracking unit 37 .

[0125] The target tracking unit 37 can be used to use the last frame event image in each of the video frame sequences to be detected as a reference frame image to determine the event feature vector of the detected target event in each of the reference frame images; perform cross processing on each of the event feature vectors to obtain a cross event feature vector; after the video frame sequence to be detected is collected, obtain a real-time target frame image collected by a reference sensor in each of the sensors, wherein the real-time target frame image is the next frame image corresponding to the reference frame image, and the reference sensor is a sensor that can collect complete and clear real-time target frame images in a reactor test scenario; based on the cross event feature vector, perform target event detection on the real-time target frame image, and draw an external rectangular frame of the target event in the real-time target frame image; based on the coordinate information of the external rectangular frame, determine the adjustment parameters of the remaining adjustable sensors in each of the sensors excluding the reference sensor, wherein the adjustment parameters include an adjustment angle and an adjustment direction; based on the adjustment parameters, adjust the remaining adjustable sensors, and track the target event using the reference sensor and the adjusted remaining adjustable sensors.

[0126] It should be noted that for other corresponding descriptions of the functional modules involved in the event detection device based on multi-source sensor fusion provided in the embodiment of the present invention, reference can be made to Figure 1 The corresponding description of the method shown will not be repeated here.

[0127] Based on the above Figure 1The method shown, accordingly, an embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the following steps are implemented: obtaining a video frame sequence to be detected in a reactor test scene collected by each sensor of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of a visible light sensor, an infrared sensor, a thermal imaging sensor, an ultrasonic sensor, and a depth sensor; clustering each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; obtaining a preset event detection model, wherein the preset event detection model includes a classification network for defect image classification of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network includes Different multi-scale convolutional layers for extracting image features from different defect images, the aggregation network includes multiple fusion layers for fusing image features from different defect images; any frame image in each of the event image frame sequences is used as a target frame image, and the target frame images at corresponding spatiotemporal positions corresponding to different sensors are input into a preset event detection model together; defect image classification is performed on the target frame images corresponding to different sensors through the classification network to obtain target frame images under different defect image categories; image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers; each image feature vector under the corresponding defect image category is fused through each of the fusion layers in the aggregation network to obtain image fusion feature vectors output by each of the fusion layers; each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.

[0128] Based on the above Figure 1 The method shown and Figure 3 The embodiment of the device shown in the figure, the embodiment of the present invention also provides a physical structure diagram of a computer device, such as Figure 5As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor, wherein the memory 42 and the processor 41 are both arranged on a bus 43, and the processor 41 implements the following steps when executing the program: obtaining a video frame sequence to be detected in a reactor test scene collected by each of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of a visible light sensor, an infrared sensor, a thermal imaging sensor, an ultrasonic sensor, and a depth sensor; clustering each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; obtaining a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event classification network for detecting target events. The invention relates to a component detection network, wherein the residual backbone network comprises different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network comprises multiple fusion layers for fusing image features of different defect images; any frame images in each of the event image frame sequences are respectively used as target frame images, and the target frame images of corresponding spatiotemporal positions corresponding to different sensors are input into a preset event detection model together; defect image classification is performed on the target frame images corresponding to different sensors through the classification network to obtain target frame images under different defect image categories; image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers; each image feature vector under the corresponding defect image category is respectively fused through each of the fusion layers in the aggregation network to obtain image fusion feature vectors output by each of the fusion layers; each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.

[0129] According to the technical solution of the present invention, the present invention obtains a video frame sequence to be detected in a reactor test scene collected by each sensor of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of a visible light sensor, an infrared sensor, a thermal imaging sensor, an ultrasonic sensor, and a depth sensor; and clusters each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; and then obtains a preset event detection model, wherein the preset event detection model includes a classification network for defect image classification of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network includes different multi-scale convolution networks for extracting image features of different defect images. The accumulation layer and the aggregation network include multiple fusion layers for fusing image features of different defect images; finally, any frame image in each of the event image frame sequences is used as the target frame image, and the target frame images of the corresponding spatiotemporal positions corresponding to different sensors are input into the preset event detection model together; the target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories; the image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolution layers; each image feature vector under the corresponding defect image category is fused through each of the fusion layers in the aggregation network to obtain the image fusion feature vectors output by each of the fusion layers; each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.Therefore, by clustering the video frame sequence to be detected in advance, the event frame sequence containing any target event is determined, and then the detected target event is directly determined in the event frame sequence, which can avoid the time wasted in event detection of non-event frame sequences that do not contain target events, so that the present invention can improve the detection efficiency of events. In addition, the present invention adopts a special model architecture, including a classification network suitable for defect image classification, a residual backbone network suitable for targeted feature extraction of different defect images, an aggregation network suitable for targeted fusion of image features of different defect images, and an event detection network, which can realize targeted multi-scale feature extraction and fusion of different defect images collected by multiple sensors, avoiding interference from defects in the image. At the same time, multi-scale feature extraction enables the model to capture detailed information and global information in the image, thereby improving the feature extraction accuracy and feature fusion accuracy of the image, and further improving the detection accuracy of the target event. At the same time, the present invention performs target event detection by performing comprehensive analysis on images collected by multiple visual sensors, which can avoid the problem that the image information collected by a single sensor is limited or even the image features cannot be extracted, so that the present invention can further improve the detection accuracy of the target event.

[0130] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, and optionally, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0131] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An event detection method based on multi-source sensor fusion, characterized in that: include: Acquire a sequence of video frames to be detected in a reactor test scene collected by each sensor of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of a visible light sensor, an infrared sensor, a thermal imaging sensor, an ultrasonic sensor, and a depth sensor; Performing clustering processing on each frame image in each of the to-be-detected video frame sequences respectively to obtain an event image frame sequence in each of the to-be-detected video frame sequences, wherein each frame image in the event image frame sequence contains any target event to be detected; Obtaining a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images for each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images; Any frame image in each of the event image frame sequences is used as a target frame image, and the target frame images of corresponding spatiotemporal positions corresponding to different sensors are input into a preset event detection model together. The target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories. The image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers. The image feature vectors under the corresponding defect image categories are fused through each of the fusion layers in the aggregation network to obtain image fusion feature vectors output by each of the fusion layers. Each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.

2. The method according to claim 1, characterized in that: The clustering process is performed on each frame image in each of the to-be-detected video frame sequences to obtain an event image frame sequence in each of the to-be-detected video frame sequences, including: Among different sensors, a sensor that can capture images of key areas is determined as a reference sensor, an optical flow feature extraction algorithm is used to extract optical flow features of each frame image in a reference video frame sequence corresponding to the reference sensor, and a scale-invariant feature extraction transformation algorithm is used to extract local features of each frame image in the reference video frame sequence; horizontally splicing the optical flow features and the local features to obtain a splicing feature vector corresponding to each frame image in the reference video frame sequence; Initializing centroid vectors corresponding to different clusters, respectively, wherein the different clusters include event clusters and non-event clusters; Calculating the cosine similarity between the splicing feature vector corresponding to each frame image in the reference video frame sequence and the centroid vectors corresponding to the different clusters, and dividing each frame image in the reference video frame sequence into the different clusters based on the cosine similarity corresponding to the different clusters; Based on the spliced ​​feature vectors corresponding to the frames of the different clusters, obtaining updated centroid vectors corresponding to the different clusters; Based on the updated centroid vector, re-dividing each frame image in the reference video frame sequence into the different clusters until the updated centroid vector does not change, and determining the images finally divided into different clusters as frames of images under different clustering categories, wherein the different clustering categories include event categories and non-event categories, and the event category includes at least one of a steam leakage event category, a leakage event category, a personnel breaking into a sample reactor test scene event category, and a high temperature event category; Determine each frame image under the event category as an event image frame sequence, and determine each frame image under the non-event category as a non-event image frame sequence; Any sensor except the reference sensor among different sensors is taken as a sensor to be clustered, and the clustering category of each frame image corresponding to the reference sensor is taken as a benchmark, and the images of the same clustering category are clustered for each frame image in the sensor to be clustered, so as to obtain each frame image under different clustering categories in the sensor to be clustered.

3. The method according to claim 1, characterized in that After clustering each frame image in each of the to-be-detected video frame sequences to obtain an event image frame sequence in each of the to-be-detected video frame sequences, the method further includes: The first image in the event image frame sequence is selected as a reference image, and other frame images in the event image frame sequence except the first image are determined as images to be registered, and target features are extracted from the reference image and the image to be registered respectively, wherein the target features include: at least one of feature points, feature lines, and feature regions; Matching the target features in the image to be registered with the target features in the reference image to obtain a feature pair; Determining a spatial transformation algorithm from the image to be registered to the reference image based on the feature pair; The image to be registered is transformed to the same spatial position as the reference image by using the spatial transformation algorithm, so as to obtain each frame image after registration in each event image frame sequence; For each registered frame image in each event image frame sequence, determine a target area having features similar to those of each registered frame image in a preset spatial area corresponding to each registered frame image, so as to obtain a spatial domain similarity group corresponding to each registered frame image; Sort the pixel values ​​of the images in the spatial domain similarity group respectively to obtain sorted pixel values; Determine an intermediate pixel value among the sorted pixel values, and use the intermediate pixel value to replace each pixel value in a corresponding area in each of the registered frame images, so as to obtain each frame image after filtering in each of the event image frame sequences; Based on the brightness information and pixel distribution information in each frame of the filtered image, color correction and contrast enhancement are performed on each frame of the filtered image to obtain a denoised event image frame sequence; The taking any frame image in each of the event image frame sequences as the target frame image respectively comprises: Any frame image in each of the event image frame sequences after denoising is used as a target frame image.

4. The method according to claim 1, characterized in that Different defect image categories include occluded image categories, blurred image categories, and thermal cross image categories; different multi-scale convolutional layers include occluded image multi-scale convolutional layers corresponding to the occluded image categories, blurred image multi-scale convolutional layers corresponding to the blurred image categories, and thermal cross image multi-scale convolutional layers corresponding to the thermal cross image categories; The image features of the target frame images under different defect image categories are extracted by the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each multi-scale convolutional layer, including: The target frame image under the occluded image category is input into the occluded image multi-scale convolution layer to obtain multiple occluded image feature vectors of different scales, the target frame image under the blurred image category is input into the blurred image multi-scale convolution layer to obtain multiple blurred image feature vectors of different scales, and the target frame image under the thermal cross image category is input into the thermal cross image multi-scale convolution layer to obtain multiple thermal cross image feature vectors of different scales.

5. The method according to claim 4, characterized in that Each of the fusion layers includes an occlusion image fusion layer corresponding to the occlusion image multi-scale convolution layer, a fuzzy image fusion layer corresponding to the fuzzy image multi-scale convolution layer, and a thermal cross image fusion layer corresponding to the thermal cross image multi-scale convolution layer; The step of fusing the image feature vectors of the corresponding defective image categories through the fusion layers in the aggregation network to obtain the image fusion feature vectors output by the fusion layers includes: The multiple occluded image feature vectors of different scales output by the occluded image multi-scale convolution layer are input into the occluded image fusion layer for feature fusion to obtain the occluded image fusion characteristic vector; the multiple blurred image feature vectors of different scales output by the blurred image multi-scale convolution layer are input into the blurred image fusion layer for feature fusion to obtain the blurred image fusion characteristic vector; the multiple thermal cross image feature vectors of different scales output by the thermal cross image multi-scale convolution layer are input into the thermal cross image fusion layer for feature fusion to obtain the thermal cross image fusion characteristic vector.

6. The method according to claim 1, characterized in that Before obtaining the preset event detection model, the method further includes: Construct multiple preset initial event detection models; Acquire a sample data set, wherein the sample data set includes sample images in a sample reactor test scene collected by multiple sensors, and actual event detection results corresponding to the sample images, wherein the actual event detection results include event locations and event types, and the event types include steam leakage events, electric leakage events, personnel intrusion into the sample reactor test scene events, and high temperature events; Based on the number of models of the preset initial event detection model, the sample data set is divided into a plurality of training data sets and a plurality of test data sets, and each group of training data sets is used to train the corresponding preset initial event detection model to obtain each trained preset initial event detection model, wherein, in the training process of the preset initial event detection model, the sample training images of the sample reactor test scene of the training data set are used as input data, and the actual event detection results corresponding to the sample training images are used as output data; Using each group of test data sets to test the corresponding trained preset initial event detection model, to obtain the event detection accuracy corresponding to each trained preset initial event detection model; Based on the event detection accuracy, a preset event detection model is determined among the trained preset initial event detection models.

7. The method according to claim 1, characterized in that The method further comprises: Taking the last event image in each of the to-be-detected video frame sequences as a reference frame image, determining an event feature vector of the target event to be detected in each of the reference frame images; Cross-processing each of the event feature vectors to obtain a cross-event feature vector; After the to-be-detected video frame sequence is acquired, a real-time target frame image acquired by a reference sensor in each of the sensors is acquired, wherein the real-time target frame image is a next frame image corresponding to the reference frame image, and the reference sensor is a sensor capable of acquiring a complete and clear real-time target frame image in a reactor test scenario; Based on the intersection event feature vector, target event detection is performed on the real-time target frame image, and a circumscribed rectangular frame of the target event is drawn in the real-time target frame image; Based on the coordinate information of the circumscribed rectangular frame, determining adjustment parameters of the remaining adjustable sensors in each sensor except the reference sensor, wherein the adjustment parameters include an adjustment angle and an adjustment direction; Based on the adjustment parameters, the remaining adjustable sensors are adjusted, and the target event is tracked using the reference sensor and the adjusted remaining adjustable sensors.

8. An event detection device based on multi-source sensor fusion, characterized in that: include: A first acquisition unit is used to acquire a video frame sequence to be detected in a reactor test scene collected by each sensor of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of a visible light sensor, an infrared sensor, a thermal imaging sensor, an ultrasonic sensor, and a depth sensor; A clustering unit, used for performing clustering processing on each frame image in each of the to-be-detected video frame sequences, respectively, to obtain an event image frame sequence in each of the to-be-detected video frame sequences, wherein each frame image in the event image frame sequence contains any target event to be detected; a second acquisition unit, configured to acquire a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images for each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images; An event detection unit is used to take any frame image in each of the event image frame sequences as a target frame image, and input the target frame images of corresponding spatiotemporal positions corresponding to different sensors into a preset event detection model together; the target frame images corresponding to different sensors are classified into defect images through the classification network to obtain target frame images under different defect image categories; the image features of the target frame images under different defect image categories are extracted through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers; each image feature vector under the corresponding defect image category is respectively fused through each of the fusion layers in the aggregation network to obtain the image fusion feature vectors output by each of the fusion layers; each of the image fusion feature vectors is input into the event detection network to determine the detected target event in the target frame image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images