Event detection method, device and storage medium based on multi-source sensor fusion

Abstract

This invention discloses an event detection method, apparatus, and storage medium based on multi-source sensor fusion. The method includes: acquiring a sequence of video frames to be detected in a reactor test scenario collected by each of multiple visual sensors; performing clustering processing on each sequence of video frames to be detected to obtain an event image frame sequence; acquiring a preset event detection model, including a classification network, an event detection network, a residual backbone network including different multi-scale convolutional layers, and an aggregation network including multiple fusion layers; classifying defect images of target frame images corresponding to different sensors through the classification network, extracting image features of target frame images under different defect image categories through the residual backbone network, fusing each image feature vector under the corresponding defect image category through each fusion layer in the aggregation network, and inputting each image fusion feature vector into the event detection network to determine the target event to be detected in the target frame image.

Description

Technical Field

[0001] This invention relates to the field of event detection technology, and in particular to an event detection method, apparatus and storage medium based on multi-source sensor fusion. Background Technology

[0002] Event detection has applications in many fields, such as reactor testing and high-temperature fluid transmission in pipelines, to ensure the effective and orderly operation of processes in each field.

[0003] Currently, event detection is typically performed based solely on images captured by a single visible light sensor. However, because visible light sensors are sensitive to illumination, images captured under extreme conditions such as fog, rain, or dense obstacles provide limited information, or even fail to extract useful information, resulting in low accuracy in event detection. Summary of the Invention

[0004] This invention provides an event detection method, device, and storage medium based on multi-source sensor fusion, which mainly improves the accuracy of event detection.

[0005] According to a first aspect of the present invention, an event detection method based on multi-source sensor fusion is provided, comprising:

[0006] Acquire a sequence of video frames to be detected in a reactor test scenario collected by each of multiple visual sensors, wherein the multiple visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors;

[0007] Clustering processing is performed on each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected;

[0008] A preset event detection model is obtained, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images.

[0009] Each frame image in the event image frame sequence is used as a target frame image. Target frame images corresponding to different spatiotemporal locations of different sensors are input into a preset event detection model. The classification network classifies the target frame images corresponding to different sensors to obtain target frame images under different defect image categories. The residual backbone network extracts image features from the target frame images under different defect image categories to obtain multiple image feature vectors at different scales output by each multi-scale convolutional layer under the corresponding defect image category. Each fusion layer in the aggregation network fuses the image feature vectors under the corresponding defect image category to obtain image fusion feature vectors output by each fusion layer. The image fusion feature vectors are input into the event detection network to determine the detected target event in the target frame image.

[0010] Optionally, the step of clustering each frame image in each of the video frame sequences to be detected to obtain the event image frame sequence in each video frame sequence to be detected includes:

[0011] Among different sensors, the sensor that can acquire images of key areas is determined as the benchmark sensor. The optical flow feature extraction algorithm is used to extract the optical flow features of each frame in the benchmark video frame sequence corresponding to the benchmark sensor, and the scale-invariant feature extraction transformation algorithm is used to extract the local features of each frame in the benchmark video frame sequence.

[0012] The optical flow features and the local features are horizontally stitched together to obtain the stitched feature vectors corresponding to each frame in the reference video frame sequence.

[0013] Initialize the centroid vectors corresponding to different clusters, wherein the different clusters include event clusters and non-event clusters;

[0014] Calculate the cosine similarity between the stitched feature vector corresponding to each frame image in the reference video frame sequence and the centroid vector corresponding to the different clusters, and classify each frame image in the reference video frame sequence into the different clusters based on the cosine similarity corresponding to the different clusters;

[0015] Based on the stitched feature vectors corresponding to each frame of the images in the different clusters, the updated centroid vectors corresponding to the different clusters are obtained;

[0016] Based on the updated centroid vector, each frame image in the reference video frame sequence is reclassified into different clusters until the updated centroid vector does not change. The images finally classified into different clusters are determined as frame images under different cluster categories. The different cluster categories include event categories and non-event categories. Event categories include at least one of the following: steam leak event category, electric leakage event category, personnel intrusion into the sample reactor test scenario event category, and high temperature event category.

[0017] Each frame image under the event category is determined as the event image frame sequence, and each frame image under the non-event category is determined as the non-event image frame sequence;

[0018] Any sensor other than the reference sensor among different sensors is taken as the sensor to be clustered. Based on the clustering category of each frame image corresponding to the reference sensor, the frames of each frame image in the sensor to be clustered are clustered into the same clustering category to obtain the frames of each frame image under different clustering categories in the sensor to be clustered.

[0019] Optionally, after performing clustering processing on each frame image in each of the video frame sequences to be detected to obtain the event image frame sequences in each video frame sequence to be detected, the method further includes:

[0020] The first image in the event image frame sequence is selected as the reference image, and the other frames in the event image frame sequence excluding the first image are determined as images to be registered. Target features are extracted from the reference image and the images to be registered, respectively. The target features include at least one of feature points, feature lines, and feature regions.

[0021] The target features in the image to be registered are matched with the target features in the reference image to obtain feature pairs;

[0022] Based on the feature pairs, a spatial transformation algorithm is determined from the image to be registered to the reference image;

[0023] The spatial transformation algorithm is used to transform the image to be registered to the same spatial position as the reference image, so as to obtain the registered frame images in each event image frame sequence;

[0024] For each registered frame image in each event image frame sequence, within a preset spatial region corresponding to each registered frame image, a target region with similar features to each registered frame image is determined to obtain a spatial similarity group corresponding to each registered frame image.

[0025] The pixel values ​​of the images in the spatial similarity group are sorted to obtain the sorted pixel values;

[0026] The middle pixel value is determined from the sorted pixel values, and the middle pixel value is used to replace each pixel value in the corresponding region of each registered frame image to obtain each filtered frame image in each event image frame sequence.

[0027] Based on the brightness information and pixel distribution information in each of the filtered frames, color correction and contrast enhancement are performed on each of the filtered frames to obtain a denoised sequence of event image frames.

[0028] The step of taking any frame image from each of the event image frame sequences as the target frame image includes:

[0029] Each frame image in the denoised event image frame sequence is taken as the target frame image.

[0030] Optionally, different defect image categories include occluded image categories, blurred image categories, and hot-crossing image categories; different multi-scale convolutional layers include occluded image multi-scale convolutional layers corresponding to the occluded image categories, blurred image multi-scale convolutional layers corresponding to the blurred image categories, and hot-crossing image multi-scale convolutional layers corresponding to the hot-crossing image categories;

[0031] The step of extracting image features from target frame images under different defect image categories through the residual backbone network to obtain multiple image feature vectors at different scales corresponding to the defect image category output by each of the multi-scale convolutional layers includes:

[0032] The target frame image under the occluded image category is input into the occluded image multi-scale convolutional layer to obtain multiple occluded image feature vectors at different scales. The target frame image under the blurred image category is input into the blurred image multi-scale convolutional layer to obtain multiple blurred image feature vectors at different scales. The target frame image under the hot cross image category is input into the hot cross image multi-scale convolutional layer to obtain multiple hot cross image feature vectors at different scales.

[0033] Optionally, each of the fusion layers includes an occlusion image fusion layer corresponding to the occlusion image multi-scale convolutional layer, a blurred image fusion layer corresponding to the blurred image multi-scale convolutional layer, and a hot cross image fusion layer corresponding to the hot cross image multi-scale convolutional layer.

[0034] The process involves fusing image feature vectors under the corresponding defect image category through each fusion layer in the aggregation network to obtain image fusion feature vectors output by each fusion layer, including:

[0035] Multiple occluded image feature vectors of different scales output from the multi-scale convolutional layer of the occluded image are input into the occluded image fusion layer for feature fusion to obtain an occluded image fusion feature vector. Multiple blurred image feature vectors of different scales output from the multi-scale convolutional layer of the blurred image are input into the blurred image fusion layer for feature fusion to obtain a blurred image fusion feature vector. Multiple hot-crossing image feature vectors of different scales output from the multi-scale convolutional layer of the hot-crossing image are input into the hot-crossing image fusion layer for feature fusion to obtain a hot-crossing image fusion feature vector.

[0036] Optionally, before obtaining the preset event detection model, the method further includes:

[0037] Construct multiple pre-defined initial event detection models;

[0038] Obtain a sample dataset, which includes sample images of a sample reactor test scenario collected by multiple sensors, and actual event detection results corresponding to the sample images. The actual event detection results include event location and event type, and the event type includes steam leak event, electric leakage event, personnel intrusion into the sample reactor test scenario event, and high temperature event.

[0039] Based on the number of models in the preset initial event detection model, the sample dataset is divided into multiple training datasets and multiple test datasets. Each training dataset is used to train the corresponding preset initial event detection model to obtain each trained preset initial event detection model. In the training process of the preset initial event detection model, the sample training images of the sample reactor test scenario in the training dataset are used as input data, and the actual event detection results corresponding to the sample training images are used as output data.

[0040] The corresponding pre-trained initial event detection models were tested using various test datasets to obtain the event detection accuracy of each pre-trained initial event detection model.

[0041] Based on the event detection accuracy, a preset event detection model is determined from each trained preset initial event detection model.

[0042] Optionally, the method further includes:

[0043] The last frame event image in each of the video frame sequences to be detected is used as a reference frame image to determine the event feature vector of the target event to be detected in each of the reference frame images;

[0044] The event feature vectors are cross-processed to obtain cross-event feature vectors;

[0045] After the video frame sequence to be detected is acquired, the real-time target frame image acquired by the reference sensor in each sensor is obtained, wherein the real-time target frame image is the next frame image corresponding to the reference frame image, and the reference sensor is a sensor that can acquire a complete and clear real-time target frame image under the reactor test scenario;

[0046] Based on the cross-event feature vector, target event detection is performed on the real-time target frame image, and the bounding rectangle of the target event is drawn in the real-time target frame image;

[0047] Based on the coordinate information of the circumscribed rectangle, the adjustment parameters of the remaining adjustable sensors in each sensor, excluding the reference sensor, are determined, wherein the adjustment parameters include the adjustment angle and the adjustment direction;

[0048] Based on the adjustment parameters, the remaining adjustable sensors are adjusted, and the target event is tracked using the reference sensor and the adjusted remaining adjustable sensors.

[0049] According to a second aspect of the present invention, an event detection device based on multi-source sensor fusion is provided, comprising:

[0050] The first acquisition unit is used to acquire the sequence of video frames to be detected in the reactor test scenario collected by each of the multiple visual sensors, wherein the multiple visual sensors include at least two of the following: visible light sensor, infrared sensor, thermal imaging sensor, ultrasonic sensor, and depth sensor.

[0051] A clustering unit is used to perform clustering processing on each frame image in each of the video frame sequences to be detected, so as to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected.

[0052] The second acquisition unit is used to acquire a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images.

[0053] The event detection unit is used to take any frame image in each event image frame sequence as the target frame image, input the target frame images corresponding to the corresponding spatiotemporal locations of different sensors into a preset event detection model, classify the target frame images corresponding to different sensors into defect images through the classification network, obtain target frame images under different defect image categories, extract the image features of the target frame images under different defect image categories through the residual backbone network, obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers, fuse each image feature vector under the corresponding defect image category through each of the fusion layers in the aggregation network, obtain the image fusion feature vector output by each of the fusion layers, and input each of the image fusion feature vectors into the event detection network to determine the detected target event in the target frame image.

[0054] According to a third aspect of the present invention, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the above-described event detection method based on multi-source sensor fusion.

[0055] According to a fourth aspect of the present invention, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the above-described event detection method based on multi-source sensor fusion.

[0056] According to the present invention, an event detection method, apparatus, and storage medium based on multi-source sensor fusion, compared with the current method of event detection based solely on images acquired by a single visible light sensor, the present invention acquires a sequence of video frames to be detected in a reactor test scenario acquired by each of multiple visual sensors, wherein the multiple visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors; and performs clustering processing on each frame image in each of the video frame sequences to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; then, a preset event detection model is obtained, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network is used to classify defect images of each sensor image, the aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The backbone network includes multi-scale convolutional layers for extracting image features from different defect images, and the aggregation network includes multiple fusion layers for fusing image features from different defect images. Finally, any frame image from each event image frame sequence is used as the target frame image. Target frame images from different sensors at corresponding spatiotemporal locations are input into a preset event detection model. The classification network classifies the target frame images from different sensors to obtain target frame images under different defect image categories. The residual backbone network extracts image features from the target frame images under different defect image categories, obtaining multiple image feature vectors at different scales output by each multi-scale convolutional layer for the corresponding defect image category. Each fusion layer in the aggregation network fuses these image feature vectors for the corresponding defect image category, obtaining image fusion feature vectors output by each fusion layer. These image fusion feature vectors are then input into the event detection network to determine the detected target event in the target frame image.Therefore, by pre-clustering the video frame sequence to be detected to determine the event frame sequence containing any target event, and then directly identifying the target event in the event frame sequence, the time wasted on event detection of non-event frame sequences that do not contain the target event can be avoided. Thus, this invention can improve the event detection efficiency. In addition, this invention adopts a specially designed model architecture, including a classification network suitable for defect image classification, a residual backbone network suitable for targeted feature extraction of different defect images, an aggregation network suitable for targeted fusion of image features of different defect images, and an event detection network. This enables targeted multi-scale feature extraction and fusion of different defect images collected by multiple sensors, avoiding interference from defects in the image. At the same time, multi-scale feature extraction enables the model to capture detailed and global information in the image, thereby improving the image feature extraction accuracy and feature fusion accuracy, and thus improving the target event detection accuracy. Meanwhile, this invention performs target event detection by comprehensively analyzing images collected by multiple visual sensors, avoiding the problem of limited image information or even inability to extract image features from images collected by a single sensor. Thus, this invention can further improve the target event detection accuracy. Attached Figure Description

[0057] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0058] Figure 1 A flowchart of an event detection method based on multi-source sensor fusion provided by an embodiment of the present invention is shown;

[0059] Figure 2 A flowchart of another event detection method based on multi-source sensor fusion provided by an embodiment of the present invention is shown;

[0060] Figure 3 A schematic diagram of the structure of an event detection device based on multi-source sensor fusion provided in an embodiment of the present invention is shown.

[0061] Figure 4 This diagram illustrates the structure of another event detection device based on multi-source sensor fusion provided in an embodiment of the present invention.

[0062] Figure 5 A schematic diagram of the physical structure of a computer device provided in an embodiment of the present invention is shown. Detailed Implementation

[0063] The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in the present application can be combined with each other.

[0064] Currently, the method of event detection based solely on images acquired by a single visible light sensor has low accuracy because the images acquired by a single sensor provide limited information, or even fail to extract useful information.

[0065] To address the aforementioned problems, embodiments of the present invention provide an event detection method, apparatus, and storage medium based on multi-source sensor fusion, such as... Figure 1 As shown, the method includes:

[0066] 101. Obtain the sequence of video frames to be detected in the reactor test scenario collected by each of the multiple vision sensors.

[0067] Among them, multiple visual sensors include at least two of the following: visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, depth sensors, etc.

[0068] In this embodiment of the invention, multiple visual sensors, including visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors, are pre-installed in the reactor test scenario. These multiple visual sensors are used to acquire videos of the reactor test process, resulting in a sequence of video frames to be detected in the reactor test scenario, each acquired by a different visual sensor. By comprehensively analyzing the images in the video frame sequences acquired by the multiple visual sensors, target events in the images are detected. The target events are then communicated to the personnel, thereby enabling the correction of events such as steam leaks that occur during the reactor test. This embodiment of the invention detects target events such as steam leaks and electrical leaks during the reactor test by comprehensively analyzing images acquired by multiple visual sensors, avoiding the problem of incomplete information in images acquired by a single sensor. Therefore, this embodiment of the invention can improve the accuracy of event detection.

[0069] 102. Cluster the images of each frame in each video frame sequence to be detected to obtain the event image frame sequence in each video frame sequence to be detected.

[0070] The event image frame sequence contains any target event to be detected in each frame. The target event includes at least one of the following events during the reactor test: steam leak, electrical leakage, or personnel intrusion.

[0071] In this embodiment of the invention, after acquiring the sequence of video frames to be detected from the reactor during the test process collected by each of the multiple visual sensors, each sequence of video frames to be detected contains multiple frames of images. Each frame of images contains images with target events and images without target events. In order to perform event detection only on the images with target events, it is necessary to perform clustering processing on each frame of the sequence of video frames to be detected to obtain the event image frame sequence with target events and the non-event image frame sequence corresponding to each sensor. Then, the target event is detected only based on the event image frame sequence, thereby avoiding the time and resources wasted on target detection of the non-event image frame sequence without target events. Thus, this embodiment of the invention can improve the event detection efficiency and save event detection resources.

[0072] 103. Obtain a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images.

[0073] In this embodiment of the invention, by optimizing the model structure, constructing a targeted event detection model, and training it using a specific dataset, a preset event detection model can be obtained. This significantly improves the detection capability of target events in reactor test scenarios, especially in images acquired by multiple sensors. The preset event detection model consists of four parts: a classification network, a residual backbone network, an aggregation network, and an event detection network. Classification network: Due to the complexity of reactor test scenarios and the presence of various test devices, coupled with the different acquisition effects of different sensors, images acquired by different sensors may contain defects such as occlusion, oversaturation, motion blur, noise, and deformation. Taking steam leakage as an example, steam may be partially obscured by pipes, the sensor platform may shake, causing steam image blurring, and steam heat transfer may cause similar temperatures between the foreground and background, resulting in image thermal crossover. Therefore, a classification network is needed to classify event frame images (taking target frame images as an example) to obtain target frame images under different defect image categories. Then, a backbone network is used to classify the target frame images under different defect image categories. The system employs targeted feature extraction from standard images and a convergence network to fuse image features across different defect image categories. This targeted feature extraction and fusion improve the accuracy of both. The backbone network utilizes multi-scale convolutional layers for different defect image categories. These layers extract features from various defect images and enhance the model's adaptability to changing event shapes through deformable convolutions, making it particularly suitable for scenarios with constantly changing shapes, such as steam, in reactor experiments. The convergence network contains multiple feature fusion layers that fuse image features at different scales for different defect images, capturing both detailed and global information—crucial for event detection based on multi-sensor images. The event detection network detects target events by comprehensively analyzing the multiple fused feature vectors output by the convergence network, ensuring accurate target identification at multiple scales. The dataset used to train the model is specifically designed for reactor test scenarios and contains images acquired by multiple visual sensors, which helps the model learn how to effectively detect target events in multiple sensor images under reactor test scenarios.

[0074] 104. Take any frame image in each event image frame sequence as the target frame image. Input the target frame images corresponding to the corresponding spatiotemporal locations of different sensors into the preset event detection model. Classify the target frame images corresponding to different sensors into defect images through the classification network to obtain target frame images under different defect image categories. Extract the image features of the target frame images under different defect image categories through the residual backbone network to obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each multi-scale convolutional layer. Fuse the image feature vectors under the corresponding defect image categories through each fusion layer in the aggregation network to obtain the image fusion feature vectors output by each fusion layer. Input the image fusion feature vectors into the event detection network to determine the detected target event in the target frame image.

[0075] The target frame image at the corresponding spatiotemporal location refers to images acquired by different visual sensors at the same time and in the same spatiotemporal space. Images acquired by different visual sensors at the same time and in the same spatiotemporal space are identified as the target frame images.

[0076] In this embodiment of the invention, target frame images at corresponding spatiotemporal locations acquired by different sensors are input into a pre-trained preset event detection model for image classification, targeted feature extraction, targeted feature fusion, and target event detection. Image classification: Target frame images at corresponding spatiotemporal locations acquired by different sensors are input into a classification network, which outputs target frame images under different defect image categories. Targeted feature extraction: Multi-scale feature extraction is performed on the target frame images under corresponding defect image categories using multi-scale convolutional layers corresponding to different defect image categories in the backbone network, resulting in image feature vectors at multiple levels under different defect image categories. Targeted feature fusion: The image feature vectors at multiple levels under corresponding defect image categories are fused using fusion layers corresponding to different defect image categories in the backbone network, resulting in fused features with richer information under different defect image categories. Target event detection: The event detection network comprehensively analyzes different fused features to determine the target event to be detected in the target frame image. That is, by comprehensively analyzing different fused features through the event detection network, the target event to be detected can be determined in the target frame images corresponding to different sensors. Therefore, this embodiment of the invention detects target events in reactor test scenarios by comprehensively analyzing images acquired by multiple visual sensors. This avoids the problem of incomplete information or inability to extract useful features from images acquired by a single sensor, thus improving the accuracy of event detection. Furthermore, the event detection model designed in this embodiment incorporates a classification network, a targeted feature extraction network, and a targeted feature fusion network. The classification network accurately classifies defect images from different sensors, allowing for feature extraction from corresponding defect images using multi-scale convolutional layers in the backbone network corresponding to different defect image categories. For example, the backbone network includes multi-scale convolutional layers specifically for feature extraction from blurred images and those specifically for feature extraction from thermally crossed images. By setting targeted multi-scale feature extraction convolutional layers for different defect images in the backbone network, the accuracy of feature extraction is improved. Simultaneously, a targeted feature fusion network is set in the aggregation network for image features under different defect images, improving the accuracy of feature fusion. This allows for more accurate extraction of implicit information from the target frame image, thereby improving the accuracy of target event detection.

[0077] In another embodiment of the present invention, the feature extraction process of the residual backbone network can be represented by the following formula:

[0078] R a =δ(f a (F5))

[0079] Among them, f aR represents the convolutional layer representing attribute a (defect image category a). a δ is the image feature output by the convolutional layer with attribute a), C is the number of defined explicit target attributes (the number of defect image categories), H and W are the set image sizes of the convolutional layer output, F5 is the image under attribute a, and δ is the activation function.

[0080] According to the present invention, an event detection method based on multi-source sensor fusion, compared with the current method of event detection based solely on images acquired by a single visible light sensor, is provided. This invention acquires a sequence of video frames to be detected from a reactor test scenario acquired by each of multiple visual sensors, including at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors. Each frame in each of the video frame sequences to be detected is then clustered to obtain an event image frame sequence, wherein each frame in the event image frame sequence contains any target event to be detected. A preset event detection model is then obtained, comprising a classification network for classifying defect images from each sensor image, a residual backbone network for extracting features from each sensor image, an aggregation network for fusing features from each sensor image, and an event detection network for detecting the target event. The residual backbone network... The system includes multi-scale convolutional layers for extracting image features from different defect images, and an aggregation network comprising multiple fusion layers for fusing image features from different defect images. Finally, any frame image from each event image frame sequence is used as the target frame image. Target frame images corresponding to different spatiotemporal locations from different sensors are input into a preset event detection model. The classification network classifies the target frame images corresponding to different sensors to obtain target frame images under different defect image categories. The residual backbone network extracts image features from the target frame images under different defect image categories, obtaining multiple image feature vectors at different scales output by each of the multi-scale convolutional layers for the corresponding defect image category. Each fusion layer in the aggregation network fuses the image feature vectors for the corresponding defect image category, obtaining image fusion feature vectors output by each fusion layer. These image fusion feature vectors are then input into the event detection network to determine the detected target event in the target frame image.Therefore, by pre-clustering the video frame sequence to be detected to determine the event frame sequence containing any target event, and then directly identifying the target event in the event frame sequence, the time wasted on event detection of non-event frame sequences that do not contain the target event can be avoided. Thus, this invention can improve the event detection efficiency. In addition, this invention adopts a specially designed model architecture, including a classification network suitable for defect image classification, a residual backbone network suitable for targeted feature extraction of different defect images, an aggregation network suitable for targeted fusion of image features of different defect images, and an event detection network. This enables targeted multi-scale feature extraction and fusion of different defect images collected by multiple sensors, avoiding interference from defects in the image. At the same time, multi-scale feature extraction enables the model to capture detailed and global information in the image, thereby improving the image feature extraction accuracy and feature fusion accuracy, and thus improving the target event detection accuracy. Meanwhile, this invention performs target event detection by comprehensively analyzing images collected by multiple visual sensors, avoiding the problem of limited image information or even inability to extract image features from images collected by a single sensor. Thus, this invention can further improve the target event detection accuracy.

[0081] Furthermore, to better illustrate the event detection process in the reactor test scenario based on multi-source sensor fusion, as a refinement and extension of the above embodiments, this invention provides another event detection method, apparatus, and storage medium based on multi-source sensor fusion, such as... Figure 2 As shown, the method includes:

[0082] 201. Obtain the sequence of video frames to be detected in the reactor test scenario collected by each of the multiple vision sensors.

[0083] Among them, multiple visual sensors include at least two of the following: visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors.

[0084] Specifically, each visual sensor is activated simultaneously and can simultaneously acquire images of the experimental process in the reactor test scenario in real time. A video acquisition time is set; when the set time is reached, the sequence of video frames to be detected within a preset time period acquired by each visual sensor is obtained. Then, target event detection is performed on each frame in the sequence of video frames to be detected.

[0085] 202. Cluster the images of each frame in each video frame sequence to be detected to obtain the event image frame sequence in each video frame sequence to be detected.

[0086] In this embodiment of the invention, to avoid wasting time on target detection of non-event frame images that do not contain target events, this embodiment of the invention requires pre-clustering of each frame image in the video frame sequence to be detected acquired by different sensors. Based on this, step 202 includes: determining the sensor capable of acquiring key region images as the reference sensor among different sensors; extracting optical flow features of each frame image in the reference video frame sequence corresponding to the reference sensor using an optical flow feature extraction algorithm; and extracting local features of each frame image in the reference video frame sequence using a scale-invariant feature extraction transformation algorithm; horizontally concatenating the optical flow features and the local features to obtain the concatenated feature vector corresponding to each frame image in the reference video frame sequence; initializing the centroid vectors corresponding to different clusters, wherein the different clusters include event clusters and non-event clusters; calculating the cosine similarity between the concatenated feature vectors corresponding to each frame image in the reference video frame sequence and the centroid vectors corresponding to the different clusters; and based on the cosine similarity corresponding to the different clusters, dividing each frame image in the reference video frame sequence into clusters. The images are assigned to different clusters; based on the stitched feature vectors corresponding to each frame in each cluster, the updated centroid vectors corresponding to each cluster are obtained; based on the updated centroid vectors, each frame in the reference video frame sequence is reassigned to the different clusters until the updated centroid vectors do not change, and the images finally assigned to different clusters are determined as frames under different cluster categories, wherein the different cluster categories include event categories and non-event categories, and the event categories include at least one of steam leak event category, electric leakage event category, personnel intrusion into sample reactor test scenario event category, and high temperature event category; each frame under the event category is determined as an event image frame sequence, and each frame under the non-event category is determined as a non-event image frame sequence; any sensor other than the reference sensor among the different sensors is taken as the sensor to be clustered, and based on the cluster category of each frame corresponding to the reference sensor, each frame in the sensor to be clustered is clustered into the same cluster category to obtain each frame in the sensor to be clustered into different cluster categories. Each frame in the event image frame sequence contains any target event to be detected.

[0087] In this context, the key region image refers to the area where the target event has the highest probability of occurrence. Specifically, using a visual sensor capable of acquiring key region images as the benchmark sensor, the images in the benchmark video frame sequence acquired by the benchmark sensor are pre-clustered. First, the least squares method is applied to estimate the motion vectors of pixels or feature points in the local regions of each frame image using an optical flow feature extraction algorithm. Then, the optical flow features of each frame image are determined based on the motion vectors of each pixel. Simultaneously, the local features of each frame image are extracted using a scale-invariant feature extraction transformation algorithm. The specific extraction process is as follows: Constructing the scale space corresponding to each frame image: In order to simulate the effect of the human eye observing an object at different distances, the scale space is first constructed using Gaussian filtering and image pyramids. The scale space consists of a series of images with different resolutions, which are obtained through Gaussian blurring and downsampling. Detecting extreme points: In the scale space, local extreme points are found using the difference of Gaussian function. These extreme points are potential feature points that exist at different scales, and they usually correspond to stable structures such as corner points and edge points in the image. Keypoint Localization: The found extreme points need further filtering and precise localization to remove unstable and falsely detected points. Keypoints can be determined by comparing the differences between extreme points and their surrounding pixels; for example, points with larger differences are selected as extreme points. Stable Point Filtering: Methods such as thresholding are used to limit the number and quality of keypoints. For example, keypoints with values ​​greater than a preset threshold are selected as stable points to ensure high stability and reliability of the extracted keypoints. Stable Point Orientation Assignment: Orientation Histogram: To ensure the stability of the keypoints is invariant to image angles and rotations, one or more orientations need to be assigned to each stable point. This can be achieved by calculating the gradient direction and magnitude of the pixels in the stable point's neighborhood and creating an orientation histogram. Determining Primary and Secondary Orientations: The peak value of the histogram is used as the primary orientation of the stable point, and other orientations higher than the primary orientation by a certain percentage (e.g., 80%) are used as secondary orientations. Generating Keypoint Descriptors: Rotating Coordinate Axes: The coordinate axes are rotated to the primary orientation of the stable point to ensure the rotation invariance of the descriptors. Block Statistics: Centered on the stable point, the surrounding area within a preset range is divided into several sub-blocks (e.g., 4x4). Gradient direction and magnitude are calculated within each sub-block to generate a gradient histogram. Descriptor Formation: The gradient histograms of all sub-blocks are concatenated to form a high-dimensional vector (e.g., 128-dimensional), which serves as the descriptor for the stable point. Feature Point Matching: Euclidean Distance Calculation: For stable point descriptors in two images, the similarity is evaluated by calculating the Euclidean distance between them. A smaller Euclidean distance indicates greater similarity between the two keypoints. Matching Threshold: A matching threshold is set. When the Euclidean distance between two stable point descriptors is less than this threshold, they are considered a matching pair.This allows for the determination of matching points. Based on these matching points, local features of each frame of the image are then determined. In this embodiment of the invention, local features of the image are extracted using a scale-invariant feature extraction transformation algorithm. These features are robust to image rotation, scaling, brightness changes, etc. Then, for each frame of the image, its corresponding optical flow features and local features are horizontally concatenated to obtain a concatenated feature vector. By concatenating the two feature vectors, more latent features in the image can be mined, making the subsequent clustering process more accurate. For the image clustering process: First, select the centroid vectors corresponding to the initial centroids of K clusters. These K clusters include event clusters and non-event clusters. Event clusters include: steam leak event clusters, electrical leakage event clusters, personnel intrusion into the sample reactor test scenario event clusters, and high-temperature event clusters. For the stitched feature vectors corresponding to each frame of the image, calculate the cosine similarity between each stitched feature vector and the K centroid vectors. Further, after calculating the cosine similarity between each frame of the image and the centroid vectors of each cluster, assign each frame of the image to the cluster corresponding to the centroid vector with the highest cosine similarity. Then, for each cluster, the centroid and its corresponding centroid vector are recalculated, and each frame image is reclassified into different clusters. This process is repeated until the position of the centroid does not change, i.e., the centroid vector does not change. Finally, the images classified into different clusters are determined as images under different clustering themes. This yields event image frame sequences and non-event image frame sequences. Event image frame sequences can be further divided into steam leak event image sequences, electric leakage event image sequences, personnel intrusion into the sample reactor test scenario event image sequences, and high temperature event image sequences.

[0088] Furthermore, any sensor other than the reference sensor among the different sensors is taken as the sensor to be clustered. After clustering each frame of the reference sensor, it is also necessary to cluster each frame of the remaining sensors to be clustered. Specifically, the clustering category of each frame of the reference sensor can be used as the basis to cluster each frame of the other sensors to be clustered. For example, if the clustering of each frame of the reference sensor yields four categories, including steam leak event category, leakage event category, personnel intrusion into the sample reactor test scenario event category, and high temperature event category, then the frames of the sensors to be clustered are also clustered according to the above four categories to obtain images under the four categories. In another embodiment of the present invention, the image frame sequence number under different categories in the reference sensor can also be used as the basis to cluster other sensors to be clustered. Clustering is performed on the frames of images in the sensor. For example, if the clustering results of the frames of images in the reference sensor are: frames 1-3 in one category, frames 4-8 in another category, and frames 9-10 in another category, then the frames of images corresponding to other sensors to be clustered are also determined to be in one category: frames 1-3 in another category, frames 4-8 in another category, and frames 9-10 in another category. In this way, by using the clustering results of the frames of images in a certain sensor as a reference to cluster the images of other remaining sensors, it is possible to ensure that the images in the video frame sequence to be detected corresponding to different sensors are unified in spatiotemporal position. After unifying the spatiotemporal position, the detection results of different sensors can be compared and fused in the same framework, thereby eliminating errors caused by time difference or spatial misalignment and improving the accuracy of target event detection.

[0089] Furthermore, to improve the detection accuracy of target events, before performing target event detection on each frame of the event image frame sequence, it is necessary to perform denoising processing on each frame of the event image frame sequence. Based on this, the method includes: selecting a first image in the event image frame sequence as a reference image, and determining the other frames in the event image frame sequence excluding the first image as images to be registered; extracting target features from the reference image and the images to be registered, wherein the target features include at least one of feature points, feature lines, and feature regions; matching the target features in the images to be registered with the target features in the reference image to obtain feature pairs; determining a spatial transformation algorithm from the images to be registered to the reference image based on the feature pairs; and using the spatial transformation algorithm to transform the images to be registered to the same spatial position as the reference image to obtain the registered frames in each event image frame sequence. In each of the event image frame sequences, after registration, a target region with similar features to the registered frame image is determined within a preset spatial region corresponding to each registered frame image, to obtain a spatial similarity group corresponding to each registered frame image; the pixel values ​​of the images in the spatial similarity group are sorted to obtain sorted pixel values; a middle pixel value is determined from the sorted pixel values, and the middle pixel value is used to replace each pixel value in the corresponding region of each registered frame image to obtain filtered frame images in each of the event image frame sequences; based on the brightness information and pixel distribution information in each filtered frame image, color correction and contrast enhancement are performed on each filtered frame image to obtain denoised event image frame sequences; the step of using any frame image in each of the event image frame sequences as a target frame image includes: using any frame image in each of the denoised event image frame sequences as a target frame image.

[0090] Specifically, inter-frame temporal and spatial filtering is required on the event frame image sequence to avoid interference from random changes in image feature points. The first frame in the event frame image sequence is selected as the reference image, and feature points, lines, and regions are used to register the remaining images to be registered. Filtering is performed in the temporal and spatial domains of each event frame. By optimizing the smoothness energy function between input and output frames, short-term jitter and long-term variations in the event frame images are filtered out. The expression is as follows:

[0091]

[0092] Where E(J) represents the smoothness energy function of the image to be registered, I(x,y,t) is the reference image, J(x,y,t) is the output image (the image to be registered), I(x,y,t+1) represents the next frame image corresponding to the reference image, x represents the horizontal coordinate of a pixel in the image, y represents the vertical coordinate of a pixel in the image, t represents the acquisition time corresponding to the image, α represents the weight coefficient of smoothness, the first term ∑|J(x,y,t)-I(x,y,t)| represents the similarity between the input and output frames of the spatiotemporal window, and the second term α∑|J(x,y,t)-I(x,y,t)| characterizes the smoothness between output frames. Gradient optimization filtering is performed on a Markov random field. Using the brightness information and pixel distribution information in the filtered image, gamma color correction and contrast enhancement are performed on the filtered image in sequence to obtain the denoised image. Specifically, firstly, the first image in the event frame image sequence is selected as the reference image (also called the baseline image). This is because in many applications, such as video analysis and dynamic image processing, the first image is often considered as the starting point or standard point for comparison and registration of subsequent images. Image registration is the process of spatially transforming images acquired at different times, from different perspectives, or from different sensors so that corresponding points in these images achieve spatial consistency. For event frame image registration, the main steps are as follows: Feature extraction: Extracting target features such as feature points, lines, or regions from the baseline image and the image to be registered. These features are characteristics that are easily identifiable in the image and remain stable across different images. Feature matching: Matching the target features in the image to be registered with the target features in the baseline image, such as finding corresponding feature pairs based on the similarity of feature descriptors. Transformation model estimation: Determining the spatial transformation algorithm from the image to be registered to the baseline image based on the matched feature pairs, such as affine transformation, rigid body transformation, and non-rigid body transformation. Image transformation: Applying the spatial transformation model to transform the image to be registered to the same spatial position as the baseline image, thus achieving image registration. Similarity Group Establishment: Spatial Similarity Group: Within a preset spatial region around the reference image (the preset spatial region is set according to actual needs), image patches or target regions with similar features to the reference image are found to form a spatial similarity group. Temporal Similarity Group: In the time series, image frames that are close in time and have similar features to the reference image are selected to form a temporal similarity group. Filtering Process: The reference image is filtered using the image information in the similarity groups. The filtering method can be selected according to specific needs, such as mean filtering, median filtering, Gaussian filtering, etc., to remove noise, enhance edges, or smooth the image. If the purpose is to improve image quality or enhance specific features, more advanced filtering techniques can be considered, such as Wiener filtering, Kalman filtering, or deep learning-based filtering methods. Color Correction and Contrast Enhancement: Based on the brightness information and pixel distribution information of the image, color correction and contrast enhancement are performed on each frame of the filtered image to obtain a denoised sequence of event image frames.

[0093] 203. Obtain a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images.

[0094] In this embodiment of the invention, to improve the detection accuracy of the preset event detection model for target events, it is necessary to construct and train the preset event detection model in advance. Based on this, the method includes: constructing multiple preset initial event detection models; acquiring a sample dataset, wherein the sample dataset includes sample images of a sample reactor test scenario collected by multiple sensors, and actual event detection results corresponding to the sample images, wherein the actual event detection results include event location and event type, and the event type includes steam leak events, electrical leakage events, personnel intrusion into the sample reactor test scenario events, and high temperature events; and dividing the sample dataset into multiple training groups based on the number of models in the preset initial event detection models. The dataset and multiple test datasets are used to train corresponding preset initial event detection models, resulting in various trained preset initial event detection models. During the training process, sample training images from the reactor test scenario in the training dataset are used as input data, and the actual event detection results corresponding to the sample training images are used as output data. The trained preset initial event detection models are then tested using the test datasets to obtain the event detection accuracy of each model. Based on the event detection accuracy, a preset event detection model is determined from among the trained preset initial event detection models.

[0095] Specifically, firstly, multiple pre-defined initial event detection models are constructed. The structure of these initial event detection models includes a classification network, a residual backbone network, an aggregation network, and an event detection network. Secondly, a sample dataset is acquired, ensuring it contains all necessary files, including image files from multiple visual sensors (including blurred images, occluded images, and hot-crossed images) and corresponding annotation information. Finally, the models are trained and tested. Specifically, the sample dataset can be divided: based on the number of pre-defined initial event detection models, the dataset is divided into multiple training sets and multiple test sets using random or specific strategies (such as stratified sampling). Then, each training set is used to train the corresponding pre-defined initial event detection model, monitoring metrics such as loss values ​​during training to evaluate model performance. Learning rates, optimizers, and other model parameters are adjusted as needed to optimize training effectiveness. Finally, each test set is used to test the corresponding trained pre-defined initial event detection model, evaluating the performance of each trained model on unseen data. Calculate and record metrics such as accuracy on the test set, and finally select the model with the highest accuracy that meets the requirements as the preset event detection model. If the model performance does not meet the requirements, it can return to the training phase for more iterations or adjustments. By training and validating multiple initial models to select the event detection model, the event detection accuracy of the final obtained event detection model can be improved.

[0096] 204. Take any frame image from each event image frame sequence as the target frame image. Input the target frame images corresponding to the corresponding spatiotemporal locations of different sensors into the preset event detection model. Use a classification network to classify the target frame images corresponding to different sensors as defect images to obtain target frame images under different defect image categories. The different defect image categories include occlusion image category, blurred image category, and hot cross image category. Different multi-scale convolutional layers include occlusion image multi-scale convolutional layers corresponding to occlusion image category, blurred image multi-scale convolutional layers corresponding to blurred image category, and hot cross image multi-scale convolutional layers corresponding to hot cross image category. Each fusion layer includes occlusion image fusion layer corresponding to occlusion image multi-scale convolutional layers, blurred image fusion layer corresponding to blurred image multi-scale convolutional layers, and hot cross image fusion layer corresponding to hot cross image multi-scale convolutional layers.

[0097] 205. Input the target frame image under the occluded image category into the occluded image multi-scale convolutional layer to obtain multiple occluded image feature vectors at different scales. Input the target frame image under the blurred image category into the blurred image multi-scale convolutional layer to obtain multiple blurred image feature vectors at different scales. Input the target frame image under the hot cross image category into the hot cross image multi-scale convolutional layer to obtain multiple hot cross image feature vectors at different scales.

[0098] 206. Input multiple occluded image feature vectors of different scales output from the multi-scale convolutional layer of the occluded image to the occluded image fusion layer for feature fusion to obtain the occluded image fusion feature vector. Input multiple blurred image feature vectors of different scales output from the multi-scale convolutional layer of the blurred image to the blurred image fusion layer for feature fusion to obtain the blurred image fusion feature vector. Input multiple hot-cross image feature vectors of different scales output from the multi-scale convolutional layer of the hot-cross image to the hot-cross image fusion layer for feature fusion to obtain the hot-cross image fusion feature vector.

[0099] 207. Input the occluded image fusion feature vector, the blurred image fusion feature vector, and the hot cross-image fusion feature vector into the event detection network to determine the detected target event in the target frame image.

[0100] Specifically, target frame images corresponding to different spatiotemporal locations from different sensors are jointly input into the classification network of a pre-defined event detection model. This classification network categorizes the input images into defect categories, such as occlusion, blurred images, and thermal crossover images. Then, the target frame images in the occlusion, blurred, and thermal crossover categories are respectively input into the occlusion multi-scale convolutional layer, blurred image multi-scale convolutional layer, and thermal crossover image multi-scale convolutional layer of the residual backbone network. The occlusion multi-scale convolutional layer outputs occlusion image features of different scales corresponding to each image in the occlusion category; the blurred image multi-scale convolutional layer outputs blurred image features of different scales corresponding to each image in the blurred category; and the thermal crossover image multi-scale convolutional layer outputs features of different scales corresponding to each image in the thermal crossover category. The features of thermally crossed images at different scales are then input into the occlusion image fusion layer, the blurry image fusion layer, and the thermally crossed image fusion layer, respectively, to obtain the occlusion image fusion feature vector output by the occlusion image fusion layer, the blurry image fusion feature vector output by the blurry image fusion layer, and the thermally crossed image fusion feature vector output by the thermally crossed image fusion layer. Further, the occlusion image fusion feature vector, the blurry image fusion feature vector, and the thermally crossed image fusion feature vector are subjected to feature-level cross-processing to further obtain the latent features between the various fusion feature vectors. The cross-processed features are then input into the event detection network to determine the detected target event in the target frame image, including the event's location information and type information. The type information includes: steam leak event, electric leakage event, personnel intrusion event, high temperature event, etc. Therefore, by classifying defect images from images acquired by different sensors, and then performing targeted feature extraction and feature fusion on images under different defect image categories, the accuracy of feature extraction and feature fusion can be improved. Furthermore, multi-scale convolutional layers process images at different scales, enabling them to handle input images or image regions of varying sizes. These layers also possess different receptive fields, capturing features at different scales, including local details and broader contextual information, thus further improving the comprehensiveness and accuracy of feature extraction. Moreover, by fusing features at different scales, information from different levels can be comprehensively utilized, while implicit information can be extracted, leading to a more accurate understanding of image content and ultimately improving the accuracy of event detection in images.

[0101] According to another event detection method, apparatus, and storage medium based on multi-source sensor fusion provided by the present invention, compared with the current method of event detection based solely on images acquired by a single visible light sensor, the present invention acquires a sequence of video frames to be detected in a reactor test scenario acquired by each of multiple visual sensors, wherein the multiple visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors; and performs clustering processing on each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; then, a preset event detection model is obtained, wherein the preset event detection model includes a classification network for defect image classification of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The differential backbone network includes different multi-scale convolutional layers for extracting image features from different defect images. The aggregation network includes multiple fusion layers for fusing image features from different defect images. Finally, any frame image in each event image frame sequence is used as the target frame image. The target frame images corresponding to the corresponding spatiotemporal locations of different sensors are input into a preset event detection model. The classification network classifies the target frame images corresponding to different sensors to obtain target frame images under different defect image categories. The residual backbone network extracts image features from the target frame images under different defect image categories to obtain multiple image feature vectors at different scales output by each multi-scale convolutional layer under the corresponding defect image category. The fusion layers in the aggregation network fuse each image feature vector under the corresponding defect image category to obtain image fusion feature vectors output by each fusion layer. The image fusion feature vectors are input into the event detection network to determine the detected target event in the target frame image.Therefore, by pre-clustering the video frame sequence to be detected to determine the event frame sequence containing any target event, and then directly identifying the target event in the event frame sequence, the time wasted on event detection of non-event frame sequences that do not contain the target event can be avoided. Thus, this invention can improve the event detection efficiency. In addition, this invention adopts a specially designed model architecture, including a classification network suitable for defect image classification, a residual backbone network suitable for targeted feature extraction of different defect images, an aggregation network suitable for targeted fusion of image features of different defect images, and an event detection network. This enables targeted multi-scale feature extraction and fusion of different defect images collected by multiple sensors, avoiding interference from defects in the image. At the same time, multi-scale feature extraction enables the model to capture detailed and global information in the image, thereby improving the image feature extraction accuracy and feature fusion accuracy, and thus improving the target event detection accuracy. Meanwhile, this invention performs target event detection by comprehensively analyzing images collected by multiple visual sensors, avoiding the problem of limited image information or even inability to extract image features from images collected by a single sensor. Thus, this invention can further improve the target event detection accuracy.

[0102] Furthermore, if target event detection is performed on each event image in the video frame sequence to be detected, and the reactor experiment is still ongoing, target event tracking can be performed on the real-time images of the reactor experiment scenario acquired subsequently, based on the target event in the last frame image of the video frame sequence to be detected. Based on this, the method includes: using the last event image in each video frame sequence to be detected as a reference frame image, determining the event feature vector of the detected target event in each reference frame image; performing cross-processing on each event feature vector to obtain a cross-event feature vector; and after the video frame sequence to be detected is acquired, obtaining the real-time target frame image acquired by the reference sensor in each sensor, wherein the real-time target frame image... The image is the next frame image corresponding to the reference frame image. The reference sensor is a sensor capable of acquiring complete and clear real-time target frame images under a reactor test scenario. Based on the cross-event feature vector, target events are detected in the real-time target frame image, and the bounding rectangle of the target event is drawn in the real-time target frame image. Based on the coordinate information of the bounding rectangle, the adjustment parameters of the remaining adjustable sensors (excluding the reference sensor) in each sensor are determined, wherein the adjustment parameters include adjustment angle and adjustment direction. Based on the adjustment parameters, the remaining adjustable sensors are adjusted, and the target event is tracked using the reference sensor and the adjusted remaining adjustable sensors.

[0103] Specifically, the last frame of each video frame sequence to be detected is used as the reference image. The event feature vectors of the target event in each reference image are determined. Then, the event feature vectors in each reference image are cross-processed; for example, elements at the same position in each event feature vector are multiplied. A preset transformation coefficient (set according to actual needs) is determined, and finally, the multiplied vectors are transformed using the preset transformation coefficient to obtain the cross-event feature vector. Once the reference sensor, capable of clearly and accurately capturing the target event image, acquires the real-time target frame image during the reactor experiment, the event feature vectors of the target event in the reference image can be compared with the real-time target frame image. Target events are detected in real-time target frame images by similarity matching, and a bounding box is drawn for each target event. Based on the coordinates of the reference center point of the bounding box in the real-time target frame image, verification images acquired by other remaining sensors in the same spatiotemporal context are simultaneously obtained. The verification center point position information of the target event is identified in the verification images. The reference position information and the verification position information are compared to determine the adjustment parameters of the remaining sensors, including translation, rotation angle, and adjustment direction. The remaining sensors are then corrected according to the adjustment parameters. Finally, the corrected sensors and the reference sensor are used together to track the target event in the reactor test scenario. Therefore, by correcting sensors with unclear or incomplete image acquisition, accurate detection and precise tracking of target events can be achieved.

[0104] Furthermore, as Figure 1 In specific implementation, embodiments of the present invention provide an event detection device based on multi-source sensor fusion, such as... Figure 3 As shown, the device includes: a first acquisition unit 31, a clustering unit 32, a second acquisition unit 33, and an event detection unit 34.

[0105] The first acquisition unit 31 can be used to acquire the sequence of video frames to be detected in a reactor test scenario collected by each of the multiple visual sensors, wherein the multiple visual sensors include at least two of the following: visible light sensor, infrared sensor, thermal imaging sensor, ultrasonic sensor, and depth sensor.

[0106] The clustering unit 32 can be used to perform clustering processing on each frame image in each of the video frame sequences to be detected, so as to obtain the event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected.

[0107] The second acquisition unit 33 can be used to acquire a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images.

[0108] The event detection unit 34 can be used to take any frame image in each event image frame sequence as the target frame image, input the target frame images corresponding to the corresponding spatiotemporal positions of different sensors into the preset event detection model, classify the target frame images corresponding to different sensors into defect images through the classification network, obtain target frame images under different defect image categories, extract the image features of the target frame images under different defect image categories through the residual backbone network, obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers, fuse each image feature vector under the corresponding defect image category through each of the fusion layers in the aggregation network, obtain the image fusion feature vectors output by each of the fusion layers, and input each of the image fusion feature vectors into the event detection network to determine the detected target event in the target frame image.

[0109] In specific application scenarios, in order to perform clustering processing on each frame of images, such as Figure 4 As shown, the clustering unit 32 includes a feature extraction module 321, a feature splicing module 322, an initialization module 323, a partitioning module 324, a determination module 325, and a clustering module 326.

[0110] The feature extraction module 321 can be used to identify the sensor capable of acquiring images of key areas as the reference sensor among different sensors, extract the optical flow features of each frame image in the reference video frame sequence corresponding to the reference sensor using an optical flow feature extraction algorithm, and extract the local features of each frame image in the reference video frame sequence using a scale-invariant feature extraction transformation algorithm.

[0111] The feature stitching module 322 can be used to horizontally stitch the optical flow features and the local features to obtain the stitched feature vectors corresponding to each frame image in the reference video frame sequence.

[0112] The initialization module 323 can be used to initialize the centroid vectors corresponding to different clusters, wherein the different clusters include event clusters and non-event clusters.

[0113] The partitioning module 324 can be used to calculate the cosine similarity between the spliced ​​feature vector corresponding to each frame image in the reference video frame sequence and the centroid vector corresponding to the different clusters, and based on the cosine similarity corresponding to the different clusters, partition each frame image in the reference video frame sequence into the different clusters.

[0114] The determining module 325 can be used to obtain the updated centroid vector corresponding to the different clusters based on the spliced ​​feature vectors corresponding to each frame image in the different clusters.

[0115] The segmentation module 324 can be specifically used to re-segment each frame image in the reference video frame sequence into different clusters based on the updated centroid vector, until the updated centroid vector does not change, and finally determine the images finally segmented into different clusters as frame images under different cluster categories, wherein the different cluster categories include event categories and non-event categories, and the event categories include at least one of steam leak event category, electric leakage event category, personnel intrusion into sample reactor test scenario event category, and high temperature event category.

[0116] The determining module 325 can also be used to determine each frame image under the event category as an event image frame sequence, and to determine each frame image under the non-event category as a non-event image frame sequence.

[0117] The clustering module 326 can be used to take any sensor other than the reference sensor from different sensors as the sensor to be clustered, and use the clustering category of each frame image corresponding to the reference sensor as a reference to perform image clustering of the same clustering category on each frame image in the sensor to be clustered, so as to obtain each frame image under different clustering categories in the sensor to be clustered.

[0118] In specific application scenarios, in order to perform noise reduction processing on each frame of image, the device further includes a noise reduction unit 35.

[0119] The denoising unit 35 can be used to select a first image as a reference image in the event image frame sequence, and determine the other frame images in the event image frame sequence excluding the first image as images to be registered. It can extract target features from the reference image and the images to be registered, wherein the target features include at least one of feature points, feature lines, and feature regions. It can match the target features in the images to be registered with the target features in the reference image to obtain feature pairs. Based on the feature pairs, it can determine a spatial transformation algorithm from the image to be registered to the reference image. It can use the spatial transformation algorithm to transform the image to be registered to the same spatial position as the reference image to obtain the registered frame images in each event image frame sequence. It can then perform registration on each event image frame sequence. For each subsequent frame image, within a preset spatial region corresponding to each registered frame image, a target region with similar features to the registered frame images is determined to obtain a spatial similarity group corresponding to each registered frame image. The pixel values ​​of the images in the spatial similarity group are sorted to obtain sorted pixel values. A middle pixel value is determined from the sorted pixel values, and the middle pixel value is used to replace each pixel value in the corresponding region of each registered frame image to obtain filtered frame images in each event image frame sequence. Based on the brightness information and pixel distribution information in each filtered frame image, color correction and contrast enhancement are performed on each filtered frame image to obtain a denoised event image frame sequence. Any frame image in each denoised event image frame sequence is used as a target frame image.

[0120] In specific application scenarios, different defect image categories include occluded image categories, blurred image categories, and hot-crossing image categories; different multi-scale convolutional layers include occluded image multi-scale convolutional layers corresponding to the occluded image category, blurred image multi-scale convolutional layers corresponding to the blurred image category, and hot-crossing image multi-scale convolutional layers corresponding to the hot-crossing image category. Based on this, in order to extract image feature vectors of each frame image, the event detection unit 34 can specifically be used to input the target frame image under the occluded image category into the occluded image multi-scale convolutional layer to obtain multiple occluded image feature vectors of different scales, input the target frame image under the blurred image category into the blurred image multi-scale convolutional layer to obtain multiple blurred image feature vectors of different scales, and input the target frame image under the hot-crossing image category into the hot-crossing image multi-scale convolutional layer to obtain multiple hot-crossing image feature vectors of different scales.

[0121] In specific application scenarios, in order to fuse image feature vectors under different defect image categories, the event detection unit 34 can be specifically used to input multiple occluded image feature vectors of different scales output by the multi-scale convolutional layer of the occluded image to the occluded image fusion layer for feature fusion to obtain an occluded image fusion feature vector; input multiple blurred image feature vectors of different scales output by the multi-scale convolutional layer of the blurred image to the blurred image fusion layer for feature fusion to obtain a blurred image fusion feature vector; and input multiple hot-crossing image feature vectors of different scales output by the multi-scale convolutional layer of the hot-crossing image to the hot-crossing image fusion layer for feature fusion to obtain a hot-crossing image fusion feature vector.

[0122] In specific application scenarios, in order to construct a preset time detection model, the device further includes a construction unit 36.

[0123] The construction unit 36 ​​can be specifically used to construct multiple preset initial event detection models; acquire a sample dataset, wherein the sample dataset includes sample images of a sample reactor test scenario collected by multiple sensors, and actual event detection results corresponding to the sample images, wherein the actual event detection results include event location and event type, and the event type includes steam leak event, electric leakage event, personnel intrusion into the sample reactor test scenario event, and high temperature event; based on the number of models of the preset initial event detection models, divide the sample dataset into multiple training datasets and multiple test datasets, and use each set of training datasets to train the corresponding preset initial event detection models to obtain each trained preset initial event detection model, wherein, during the training process of the preset initial event detection models, the sample training images of the sample reactor test scenario in the training dataset are used as input data, and the actual event detection results corresponding to the sample training images are used as output data; use each set of test datasets to test the corresponding trained preset initial event detection models to obtain the event detection accuracy corresponding to each trained preset initial event detection model; based on the event detection accuracy, determine the preset event detection model among the various trained preset initial event detection models.

[0124] In specific application scenarios, in order to track target events in real time during reactor testing, the device also includes a target tracking unit 37.

[0125] The target tracking unit 37 can be used to take the last frame event image in each of the video frame sequences to be detected as a reference frame image, determine the event feature vector of the detected target event in each of the reference frame images; perform cross processing on each of the event feature vectors to obtain a cross event feature vector; after the video frame sequence to be detected is acquired, acquire the real-time target frame image acquired by the reference sensor in each sensor, wherein the real-time target frame image is the next frame image corresponding to the reference frame image, and the reference sensor is a sensor capable of acquiring a complete and clear real-time target frame image under the reactor test scenario; based on the cross event feature vector, perform target event detection on the real-time target frame image, and draw the outer rectangle of the target event in the real-time target frame image; based on the coordinate information of the outer rectangle, determine the adjustment parameters of the remaining adjustable sensors in each sensor excluding the reference sensor, wherein the adjustment parameters include adjustment angle and adjustment direction; based on the adjustment parameters, adjust the remaining adjustable sensors, and use the reference sensor and the adjusted remaining adjustable sensors to track the target event.

[0126] It should be noted that other corresponding descriptions of the functional modules involved in the event detection device based on multi-source sensor fusion provided in this embodiment of the invention can be found in the following references. Figure 1 The corresponding description of the method shown will not be repeated here.

[0127] Based on the above, Figure 1The method shown, correspondingly, also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, performs the following steps: acquiring a sequence of video frames to be detected in a reactor test scenario collected by each of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors; performing clustering processing on each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; acquiring a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events, wherein the residual backbone network ... the residual backbone network including a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, a residual backbone network for extracting features of each sensor image, and an event detection network for extracting features of each sensor image, the residual backbone network including a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, a residual backbone network for extracting defect images of each sensor image, and an event detection network for extracting features of each sensor image, the residual backbone network including a classification network for classifying defect images of each sensor image, a residual backbone network for extracting defect images of each sensor image, a residual backbone network for extracting defect images of each sensor image, and an event detection network for extracting features of each sensor image, the residual backbone network including a classification network for classifying defect images of each sensor image, a residual backbone network for extract The aggregation network includes multiple fusion layers that fuse image features from different defect images, using different multi-scale convolutional layers for image feature extraction from different defect images. Any frame image from each event image frame sequence is used as the target frame image. Target frame images from different sensors at corresponding spatiotemporal locations are input into a preset event detection model. The classification network classifies the target frame images from different sensors to obtain target frame images under different defect image categories. The residual backbone network extracts image features from the target frame images under different defect image categories, obtaining multiple image feature vectors at different scales output by each multi-scale convolutional layer for the corresponding defect image category. Each fusion layer in the aggregation network fuses these image feature vectors for the corresponding defect image category, obtaining image fusion feature vectors output by each fusion layer. These image fusion feature vectors are then input into the event detection network to determine the detected target event in the target frame image.

[0128] Based on the above, Figure 1 The method shown and as Figure 3 The embodiment of the device shown in the invention also provides a physical structure diagram of a computer device, such as... Figure 5As shown, the computer device includes: a processor 41, a memory 42, and a computer program stored in the memory 42 and executable on the processor. Both the memory 42 and the processor 41 are mounted on a bus 43. When the processor 41 executes the program, it performs the following steps: acquiring a sequence of video frames to be detected in a reactor test scenario collected by each of a plurality of visual sensors, wherein the plurality of visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors; performing clustering processing on each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; acquiring a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The event detection network includes a residual backbone network comprising multi-scale convolutional layers for extracting image features from different defect images, and an aggregation network comprising multiple fusion layers for fusing image features from different defect images. Any frame image from each event image frame sequence is used as a target frame image. Target frame images corresponding to different spatiotemporal locations from different sensors are input into a preset event detection model. The classification network classifies the target frame images corresponding to different sensors to obtain target frame images under different defect image categories. The residual backbone network extracts image features from the target frame images under different defect image categories to obtain multiple image feature vectors at different scales output by each multi-scale convolutional layer under the corresponding defect image category. Each fusion layer in the aggregation network fuses the image feature vectors under the corresponding defect image category to obtain image fusion feature vectors output by each fusion layer. These image fusion feature vectors are then input into the event detection network to determine the detected target event in the target frame image.

[0129] The present invention obtains a sequence of video frames to be detected in a reactor test scenario collected by each of multiple visual sensors, including at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors. Each frame in the video frame sequence is then clustered to obtain an event image frame sequence, wherein each frame in the event image frame sequence contains any target event to be detected. A preset event detection model is then obtained, comprising a classification network for classifying defect images from each sensor image, a residual backbone network for extracting features from each sensor image, an aggregation network for fusing features from each sensor image, and an event detection network for detecting target events. The residual backbone network includes different multi-scale convolutions for extracting image features from different defect images. The aggregation network includes multiple fusion layers that fuse image features of different defect images. Finally, any frame image from each event image frame sequence is used as the target frame image. Target frame images corresponding to different spatiotemporal locations of different sensors are input into a preset event detection model. The classification network classifies the target frame images corresponding to different sensors to obtain target frame images under different defect image categories. The residual backbone network extracts image features from the target frame images under different defect image categories, obtaining multiple image feature vectors at different scales output by each multi-scale convolutional layer under the corresponding defect image category. Each fusion layer in the aggregation network fuses the image feature vectors under the corresponding defect image category to obtain image fusion feature vectors output by each fusion layer. These image fusion feature vectors are then input into the event detection network to determine the detected target event in the target frame image.Therefore, by pre-clustering the video frame sequence to be detected to determine the event frame sequence containing any target event, and then directly identifying the target event in the event frame sequence, the time wasted on event detection of non-event frame sequences that do not contain the target event can be avoided. Thus, this invention can improve the event detection efficiency. In addition, this invention adopts a specially designed model architecture, including a classification network suitable for defect image classification, a residual backbone network suitable for targeted feature extraction of different defect images, an aggregation network suitable for targeted fusion of image features of different defect images, and an event detection network. This enables targeted multi-scale feature extraction and fusion of different defect images collected by multiple sensors, avoiding interference from defects in the image. At the same time, multi-scale feature extraction enables the model to capture detailed and global information in the image, thereby improving the image feature extraction accuracy and feature fusion accuracy, and thus improving the target event detection accuracy. Meanwhile, this invention performs target event detection by comprehensively analyzing images collected by multiple visual sensors, avoiding the problem of limited image information or even inability to extract image features from images collected by a single sensor. Thus, this invention can further improve the target event detection accuracy.

[0130] It is obvious to those skilled in the art that the modules or steps of the present invention described above can be implemented using general-purpose computing devices. They can be centralized on a single computing device or distributed across a network of multiple computing devices. Optionally, they can be implemented using computer-executable program code, thereby storing them in a storage device for execution by a computing device. In some cases, the steps shown or described can be performed in a different order than those presented herein, or they can be fabricated as separate integrated circuit modules, or multiple modules or steps can be fabricated as a single integrated circuit module. Thus, the present invention is not limited to any particular combination of hardware and software.

[0131] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. An event detection method based on multi-source sensor fusion, characterized in that, include: Acquire a sequence of video frames to be detected in a reactor test scenario collected by each of multiple visual sensors, wherein the multiple visual sensors include at least two of visible light sensors, infrared sensors, thermal imaging sensors, ultrasonic sensors, and depth sensors; Clustering processing is performed on each frame image in each of the video frame sequences to be detected to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected; A preset event detection model is obtained, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images. Each frame image in the event image frame sequence is used as a target frame image. Target frame images corresponding to different spatiotemporal locations of different sensors are input into a preset event detection model. The classification network classifies the target frame images corresponding to different sensors to obtain target frame images under different defect image categories. The residual backbone network extracts image features from the target frame images under different defect image categories to obtain multiple image feature vectors at different scales output by each multi-scale convolutional layer under the corresponding defect image category. Each fusion layer in the aggregation network fuses the image feature vectors under the corresponding defect image category to obtain image fusion feature vectors output by each fusion layer. The image fusion feature vectors are input into the event detection network to determine the detected target event in the target frame image.

2. The method according to claim 1, characterized in that, The step of clustering each frame image in each of the video frame sequences to be detected to obtain the event image frame sequence in each video frame sequence to be detected includes: Among different sensors, the sensor that can acquire images of key areas is determined as the benchmark sensor. The optical flow feature extraction algorithm is used to extract the optical flow features of each frame in the benchmark video frame sequence corresponding to the benchmark sensor, and the scale-invariant feature extraction transformation algorithm is used to extract the local features of each frame in the benchmark video frame sequence. The optical flow features and the local features are horizontally stitched together to obtain the stitched feature vectors corresponding to each frame in the reference video frame sequence. Initialize the centroid vectors corresponding to different clusters, wherein the different clusters include event clusters and non-event clusters; Calculate the cosine similarity between the stitched feature vector corresponding to each frame image in the reference video frame sequence and the centroid vector corresponding to the different clusters, and classify each frame image in the reference video frame sequence into the different clusters based on the cosine similarity corresponding to the different clusters; Based on the stitched feature vectors corresponding to each frame of the images in the different clusters, the updated centroid vectors corresponding to the different clusters are obtained; Based on the updated centroid vector, each frame image in the reference video frame sequence is reclassified into different clusters until the updated centroid vector does not change. The images finally classified into different clusters are determined as frame images under different cluster categories. The different cluster categories include event categories and non-event categories. Event categories include at least one of the following: steam leak event category, electric leakage event category, personnel intrusion into the sample reactor test scenario event category, and high temperature event category. Each frame image under the event category is determined as the event image frame sequence, and each frame image under the non-event category is determined as the non-event image frame sequence; Any sensor other than the reference sensor among different sensors is taken as the sensor to be clustered. Based on the clustering category of each frame image corresponding to the reference sensor, the frames of each frame image in the sensor to be clustered are clustered into the same clustering category to obtain the frames of each frame image under different clustering categories in the sensor to be clustered.

3. The method according to claim 1, characterized in that, After performing clustering processing on each frame image in each of the video frame sequences to be detected to obtain the event image frame sequences in each video frame sequence to be detected, the method further includes: The first image in the event image frame sequence is selected as the reference image, and the other frames in the event image frame sequence excluding the first image are determined as images to be registered. Target features are extracted from the reference image and the images to be registered, respectively. The target features include at least one of feature points, feature lines, and feature regions. The target features in the image to be registered are matched with the target features in the reference image to obtain feature pairs; Based on the feature pairs, a spatial transformation algorithm is determined from the image to be registered to the reference image; The spatial transformation algorithm is used to transform the image to be registered to the same spatial position as the reference image, so as to obtain the registered frame images in each event image frame sequence; For each registered frame image in each event image frame sequence, within a preset spatial region corresponding to each registered frame image, a target region with similar features to each registered frame image is determined to obtain a spatial similarity group corresponding to each registered frame image. The pixel values ​​of the images in the spatial similarity group are sorted to obtain the sorted pixel values; The middle pixel value is determined from the sorted pixel values, and the middle pixel value is used to replace each pixel value in the corresponding region of each registered frame image to obtain each filtered frame image in each event image frame sequence. Based on the brightness information and pixel distribution information in each of the filtered frames, color correction and contrast enhancement are performed on each of the filtered frames to obtain a denoised sequence of event image frames. The step of taking any frame image from each of the event image frame sequences as the target frame image includes: Each frame image in the denoised event image frame sequence is taken as the target frame image.

4. The method according to claim 1, characterized in that, Different defect image categories include occluded image category, blurred image category, and hot cross image category; different multi-scale convolutional layers include occluded image multi-scale convolutional layer corresponding to the occluded image category, blurred image multi-scale convolutional layer corresponding to the blurred image category, and hot cross image multi-scale convolutional layer corresponding to the hot cross image category; The step of extracting image features from target frame images under different defect image categories through the residual backbone network to obtain multiple image feature vectors at different scales corresponding to the defect image category output by each of the multi-scale convolutional layers includes: The target frame image under the occluded image category is input into the occluded image multi-scale convolutional layer to obtain multiple occluded image feature vectors at different scales. The target frame image under the blurred image category is input into the blurred image multi-scale convolutional layer to obtain multiple blurred image feature vectors at different scales. The target frame image under the hot cross image category is input into the hot cross image multi-scale convolutional layer to obtain multiple hot cross image feature vectors at different scales.

5. The method according to claim 4, characterized in that, Each of the fusion layers includes an occlusion image fusion layer corresponding to the occlusion image multi-scale convolutional layer, a blurred image fusion layer corresponding to the blurred image multi-scale convolutional layer, and a hot-cross image fusion layer corresponding to the hot-cross image multi-scale convolutional layer; The process involves fusing image feature vectors under the corresponding defect image category through each fusion layer in the aggregation network to obtain image fusion feature vectors output by each fusion layer, including: Multiple occluded image feature vectors of different scales output from the multi-scale convolutional layer of the occluded image are input into the occluded image fusion layer for feature fusion to obtain an occluded image fusion feature vector. Multiple blurred image feature vectors of different scales output from the multi-scale convolutional layer of the blurred image are input into the blurred image fusion layer for feature fusion to obtain a blurred image fusion feature vector. Multiple hot-crossing image feature vectors of different scales output from the multi-scale convolutional layer of the hot-crossing image are input into the hot-crossing image fusion layer for feature fusion to obtain a hot-crossing image fusion feature vector.

6. The method according to claim 1, characterized in that, Before obtaining the preset event detection model, the method further includes: Construct multiple pre-defined initial event detection models; Obtain a sample dataset, which includes sample images of a sample reactor test scenario collected by multiple sensors, and actual event detection results corresponding to the sample images. The actual event detection results include event location and event type, and the event type includes steam leak event, electric leakage event, personnel intrusion into the sample reactor test scenario event, and high temperature event. Based on the number of models in the preset initial event detection model, the sample dataset is divided into multiple training datasets and multiple test datasets. Each training dataset is used to train the corresponding preset initial event detection model to obtain each trained preset initial event detection model. In the training process of the preset initial event detection model, the sample training images of the sample reactor test scenario in the training dataset are used as input data, and the actual event detection results corresponding to the sample training images are used as output data. The corresponding pre-trained initial event detection models were tested using various test datasets to obtain the event detection accuracy of each pre-trained initial event detection model. Based on the event detection accuracy, a preset event detection model is determined from each trained preset initial event detection model.

7. The method according to claim 1, characterized in that, The method further includes: The last frame event image in each of the video frame sequences to be detected is used as a reference frame image to determine the event feature vector of the target event to be detected in each of the reference frame images; The event feature vectors are cross-processed to obtain cross-event feature vectors; After the video frame sequence to be detected is acquired, the real-time target frame image acquired by the reference sensor in each sensor is obtained, wherein the real-time target frame image is the next frame image corresponding to the reference frame image, and the reference sensor is a sensor that can acquire a complete and clear real-time target frame image under the reactor test scenario; Based on the cross-event feature vector, target event detection is performed on the real-time target frame image, and the bounding rectangle of the target event is drawn in the real-time target frame image; Based on the coordinate information of the circumscribed rectangle, the adjustment parameters of the remaining adjustable sensors in each sensor, excluding the reference sensor, are determined, wherein the adjustment parameters include the adjustment angle and the adjustment direction; Based on the adjustment parameters, the remaining adjustable sensors are adjusted, and the target event is tracked using the reference sensor and the adjusted remaining adjustable sensors.

8. An event detection device based on multi-source sensor fusion, characterized in that, include: The first acquisition unit is used to acquire the sequence of video frames to be detected in the reactor test scenario collected by each of the multiple visual sensors, wherein the multiple visual sensors include at least two of the following: visible light sensor, infrared sensor, thermal imaging sensor, ultrasonic sensor, and depth sensor. A clustering unit is used to perform clustering processing on each frame image in each of the video frame sequences to be detected, so as to obtain an event image frame sequence in each of the video frame sequences to be detected, wherein each frame image in the event image frame sequence contains any target event to be detected. The second acquisition unit is used to acquire a preset event detection model, wherein the preset event detection model includes a classification network for classifying defect images of each sensor image, a residual backbone network for extracting features of each sensor image, an aggregation network for fusing features of each sensor image, and an event detection network for detecting target events. The residual backbone network includes different multi-scale convolutional layers for extracting image features of different defect images, and the aggregation network includes multiple fusion layers for fusing image features of different defect images. The event detection unit is used to take any frame image in each event image frame sequence as the target frame image, input the target frame images corresponding to the corresponding spatiotemporal locations of different sensors into a preset event detection model, classify the target frame images corresponding to different sensors into defect images through the classification network, obtain target frame images under different defect image categories, extract the image features of the target frame images under different defect image categories through the residual backbone network, obtain multiple image feature vectors of different scales under the corresponding defect image categories output by each of the multi-scale convolutional layers, fuse each image feature vector under the corresponding defect image category through each of the fusion layers in the aggregation network, obtain the image fusion feature vector output by each of the fusion layers, and input each of the image fusion feature vectors into the event detection network to determine the detected target event in the target frame image.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images