High-speed weak small foreign matter detection method based on event and image fusion

Abstract

The application belongs to the technical field of weak and small target recognition of nuclear power plants, aims to solve the problem that the existing target detection method is difficult to effectively detect high-speed falling foreign matters, discloses a high-speed weak and small foreign matter detection method based on event and image fusion, carries out data acquisition, constructs a real data set, calibrates the real data, constructs a simulation data set for data enhancement, constructs a cross-modal feature fusion model by using a physical model of a motion image and an event stream generation mechanism, carries out adaptive space-time alignment on the event stream and the video frame image, carries out cross-modal data complementary enhancement fusion on the event stream and the video frame image after space-time alignment, carries out training by using a target detection network, and carries out feature extraction of weak and small targets by combining local and global attention. The application can effectively detect high-speed falling foreign matters by combining event and image information, combining the instant reactivity of event data and the detail retention of image data.

Description

Technical Field

[0001] This application belongs to the field of weak target recognition technology in nuclear power plants, and particularly relates to a high-speed method for detecting weak foreign objects based on event and image fusion. Background Technology

[0002] Currently, the detection of falling foreign objects in nuclear power plants is mainly based on video surveillance and conventional target detection methods. However, for foreign objects that are small in shape and volume and fall at high speed (such as screws and nuts), there are problems such as dynamic ambiguity caused by speed, signal energy dispersion within a limited observation window, and low signal-to-noise ratio caused by small size. Therefore, the detection effect is often unsatisfactory when using existing video surveillance systems and related target detection technologies.

[0003] With the development of vision and various hardware technologies, event cameras, as an emerging visual sensor technology, possess unique advantages in high-speed, high-dynamic-range scenes. Compared to traditional cameras, event cameras no longer rely on traditional frame-format imaging but focus on sensing minute changes in scene light intensity and asynchronously outputting this information in the form of event pulses. Event cameras also feature high temporal resolution and low latency, enabling them to more accurately capture the rapid movement of small targets. Therefore, event cameras have significant potential application value in high-speed foreign object drop detection. Summary of the Invention

[0004] The purpose of this application is to provide a high-speed, weak foreign object detection method based on event and image fusion, which solves the problem that existing target detection methods are difficult to effectively detect high-speed falling foreign objects. This application is used for the monitoring and identification of falling foreign objects in industrial sites, and realizes the identification of various high-speed, weak falling foreign objects in complex environments.

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] A high-speed method for detecting small foreign objects based on event and image fusion includes:

[0007] Step 1: Collect data and build a real dataset;

[0008] Step 2: Calibrate the real data;

[0009] Step 3: Construct a simulation dataset for data augmentation;

[0010] Step 4: Construct a cross-modal feature fusion model using the physical model of motion images and the event stream generation mechanism to adaptively align the event stream and video frame images in time and space;

[0011] Step 5: Perform cross-modal data complementarity enhancement fusion on the spatiotemporally aligned event stream and video frame images;

[0012] Step 6: Train an object detection network to extract features from weak targets by fusing local and global attention.

[0013] In some embodiments, in step 1, the dataset is acquired using an event camera and RGB data acquisition under different lighting and motion speed conditions.

[0014] In some embodiments, step 2, calibration includes the calibration of intrinsic and extrinsic parameters of the event camera and the RGB camera, and the creation of a real label for the target.

[0015] In some embodiments, step 3, the simulation data generation step includes:

[0016] Step 3.1: Use camera parameters to backproject the original image I into three-dimensional space, and at the initial moment, use the camera pose and camera parameters to reproject to obtain image I0;

[0017] Step 3.2: Calculate the next camera sampling time t based on the adaptive sampling strategy. k ;

[0018] Step 3.3: At t k New images are generated continuously by projecting the camera pose and camera parameters. k ;

[0019] Step 3.4: According to I k and I k-1 (The image generated at the previous sampling time) Calculates the image brightness change and generates all event points between the two frames;

[0020] Step 3.5: Repeat steps 3.2 to 3.4 until the end time T of the camera motion trajectory is reached.

[0021] In some embodiments, step 4 includes:

[0022] Step 4.1: Given the intrinsic and extrinsic parameters of the event camera and the ordinary camera, calculate the homography transformation matrix based on the camera intrinsic and extrinsic parameters and perform coarse spatial alignment;

[0023] Step 4.2: Perform coarse segmentation of the scene based on motion patterns and extract the common far-field region of image frames and event streams;

[0024] Step 4.3: Based on time alignment, a cross-modal feature adaptive selection and fusion module is introduced through multi-stage alignment, as follows:

[0025]

[0026] In the formula, These represent the disparity warp operators between the event stream and the traditional image frame, respectively; G l G e To generate an adaptive weighted mapping graph using subnetworks; This is the i-th adaptive weighted mapping generated using the subnetwork; The i-th input event frame and image frame; For the (i+1)th input event frame and image frame.

[0027] In some embodiments, in step 4.1,

[0028]

[0029] In the formula, K e and K b R and t are the intrinsic parameter matrices for the event camera and the normal camera, respectively. R and t are the rotation matrix and translation vector in the rigid body transformation of the camera coordinate system, respectively. n and d are the reference plane normal vector and the scene depth, respectively.

[0030] In some embodiments, in step 4.2, the common far-field regions are as follows:

[0031]

[0032] In the formula, L T E represents the image frame within the time window T; Te S is the set of all events under the corresponding event window T; S(·) is the coarse segmentation operator;

[0033] Fine-grained temporal correction function estimation for conventional image frames and events spatially aligned to a common far-field region:

[0034]

[0035] In the formula, τ is the fine time correction for aligning the two data streams; F(·) is the regression function used to estimate the time offset mentioned above.

[0036] In some embodiments, step 6, local self-attention extraction includes:

[0037] Transform the size of the input feature map to Then calculate self-attention locally;

[0038] Where P is the window size of the local self-attention mechanism, C is the number of channels in the output feature map, H is the image height, and W is the image width.

[0039] In some embodiments, step 6, global self-attention extraction includes:

[0040] Transform the size of the input feature map to Then calculate self-attention globally;

[0041] Where G is the window size of the global self-attention mechanism, C is the number of channels in the output feature map, H is the image height, and W is the image width;

[0042] The attention mechanism is calculated in the form of Q (query), K (key), and V (value), as follows:

[0043]

[0044] In the formula, d k Q is a constant; K is the information to be queried; V is the information that exists under this key; and V carries the actual information to be output.

[0045] In some embodiments, in step 6, weak target detection is performed based on a multi-level attention mechanism, and a channel attention map is generated using the inter-channel relationships of features, as follows:

[0046] By aggregating the spatial information of the feature maps using average pooling and max pooling operations, two different spatial context descriptors are generated: These descriptors represent the feature maps after average pooling and max pooling, respectively; they are then input into a shared network to generate channel attention maps. The shared network consists of a hidden layer MLP, and the channel attention is calculated as follows:

[0047]

[0048] In the formula, σ is the sigmoid activation function. r is the number of hidden layers in the MLP, and C is the number of channels.

[0049] In some embodiments, step 6, generating a spatial attention map using the spatial relationships of features, includes:

[0050] Apply average pooling and max pooling operations along the channel axis to generate two 2D feature maps. They are connected and convolved using standard convolutional layers to generate a two-dimensional spatial attention map. The calculation of spatial attention is as follows:

[0051]

[0052] In the formula, σ is the sigmoid activation function, and f 7×7 This indicates a convolution operation with a kernel size of 7×7.

[0053] In some embodiments, step 6, generating an instance attention map using the instance relationships of features, includes:

[0054] The spatial dimension is reduced from 2D to 1D by aggregating the spatial and channel information of the feature map using average pooling and max pooling operations, generating two different spatial context descriptors: These represent the feature maps after average pooling and max pooling, respectively.

[0055] Two descriptors are input into a shared network to generate an instance attention graph. The shared network consists of a multilayer perceptron (MLP) with a hidden layer, and channel attention is calculated as follows:

[0056]

[0057] Where σ is the sigmoid activation function, k is the number of hidden layers in the MLP, and N is the number of instances.

[0058] Compared with existing technologies, the high-speed weak foreign object detection method based on event and image fusion provided in this application has the following advantages:

[0059] This application combines event and image information, integrating the immediacy of event data with the detail retention of image data, to effectively detect high-speed falling foreign objects. Furthermore, it improves the accuracy of target detection and tracking performance, exhibits high robustness, and has a wide range of applications.

[0060] This application presents a high-speed, low-profile foreign object identification method based on the combination of event cameras and traditional frame cameras, which is applied to the detection of high-speed, low-profile foreign objects in nuclear power plants and has significant practical implications. Attached Figure Description

[0061] To more clearly illustrate the technical solution of this application, the accompanying drawings used in the technical description will be briefly introduced below.

[0062] Figure 1 A flowchart of the high-speed weak foreign object detection method based on event and image fusion provided in this application;

[0063] Figure 2 The overall testing flowchart provided for this application;

[0064] Figure 3 A schematic diagram of the feature extraction architecture provided for this application;

[0065] Figure 4 A schematic diagram of the attention mechanism provided in this application. Detailed Implementation

[0066] The following detailed description provides further details on specific implementation methods.

[0067] like Figures 1 to 4 As shown, this application provides a high-speed method for detecting small foreign objects based on event and image fusion, including:

[0068] Step 1: Construct a real dataset. The dataset is obtained by collecting data using a real event camera and RGB under different lighting conditions, motion speed, etc.

[0069] Step 2: Accurately calibrate the collected real data, including the calibration of the intrinsic and extrinsic parameters of the event camera and RGB camera, and the creation of real labels for the target;

[0070] Step 3: Construct a simulation dataset for data augmentation; the event generation model uses the ESIM model.

[0071] Step 4: Utilize the physical model of motion images and the event stream generation mechanism to construct a cross-modal feature fusion model for adaptive spatiotemporal alignment of event streams and video frame images;

[0072] Step 5: Perform cross-modal data complementarity enhancement fusion on the spatiotemporally aligned event stream and video frame images;

[0073] Step 6: Train the object detection network. During training, feature extraction of weak targets that integrates local and global attention is performed first.

[0074] In step 2, manual calibration is used, resulting in more accurate calibration results.

[0075] In step 3, the simulation dataset is generated by filtering small target images from the existing RGB target detection dataset and using an event simulator. The simulation data generation steps are as follows:

[0076] Step 3.1: Use camera parameters to backproject the original image I into three-dimensional space, and at the initial moment, use the camera pose and camera parameters to reproject to obtain image I0;

[0077] Step 3.2: Calculate the next camera sampling time t based on the existing adaptive sampling strategy. k ;

[0078] Step 3.3: At t k New images are generated continuously by projecting the camera pose and camera parameters. k ;

[0079] Step 3.4: According to I k and I k-1 Calculate image brightness changes and generate all event points between two frames;

[0080] Step 3.5: Repeat steps 3.2 to 3.4 until the end time T of the camera motion trajectory is reached.

[0081] In step 4, the event camera acquires event stream information, the RGB camera acquires RGB images, the event data is processed to obtain a 3D tensor compatible with 2D convolution, the RGB image is directly converted into a 3D tensor compatible with 2D convolution, and the two are adaptively spatiotemporally aligned.

[0082] Step 4 includes:

[0083] Step 4.1: Given the intrinsic and extrinsic parameters of the event camera and the ordinary camera, calculate the homography transformation matrix based on the camera's intrinsic and extrinsic parameters to complete the coarse spatial alignment:

[0084]

[0085] In the formula, K e and K b are the intrinsic parameter matrices for the event camera and the normal camera, respectively; r and t are the rotation matrix and translation vector (baseline) in the rigid body transformation of the camera coordinate system, respectively; n and d are the reference plane normal vector and scene depth, respectively; I is the third-order identity matrix; T is the camera translation.

[0086] Step 4.2: Perform coarse scene segmentation based on motion patterns, and extract the common far-field region of image frames and event streams, as follows:

[0087]

[0088] In the formula, For the common far-field region; L T E represents the image frame within the time window T; Te S is the set of all events under the corresponding event window T; S(·) is the coarse segmentation operator;

[0089] Fine-grained temporal correction function estimation for conventional image frames and events spatially aligned to a common far-field region:

[0090]

[0091] In the formula, τ is the fine time correction for aligning the two data streams; F(·) is the regression function used to estimate the time offset mentioned above.

[0092] Step 4.3: Building upon temporal alignment, a feature learning method that considers depth parallax is further utilized to address the cross-modal parallax problem caused by spatial mismatch. To effectively fuse event streams and local features of traditional image frames, a cross-modal feature adaptive selection and fusion module is introduced through multi-stage alignment:

[0093]

[0094]

[0095] In the formula, These represent the disparity warp operators between the event stream and the traditional image frame, respectively; G l G e To generate an adaptive weighted mapping graph using subnetworks; This is the i-th adaptive weighted mapping generated using the subnetwork; The i-th input event frame and image frame; For the (i+1)th input event frame and image frame.

[0096] In step 5, an algorithm that incorporates existing random masks is used to enhance the features of weak targets.

[0097] In step 6, the feature extraction architecture is as follows: Figure 3 As shown. Step 6 uses a weak target detection algorithm based on a multi-level attention mechanism (algorithm structure as shown). Figure 3 (As shown) Target detection is performed.

[0098] Local self-attention extraction first transforms the size of the input feature map to Where P is the window size of the local self-attention mechanism, C is the number of channels in the output feature map, H is the image height, and W is the image width; then self-attention is calculated locally.

[0099] Global self-attention extraction will transform the size of the input feature map to Where G is the window size of the global self-attention mechanism, C is the number of channels in the output feature map, and then global self-attention is calculated on it. The attention mechanism is calculated in the form of Q (query), K (key), and V (value). The specific calculation formula is as follows:

[0100]

[0101] In the formula, d k It is usually a constant.

[0102] Then, weak target detection is performed based on a multi-layered attention mechanism, such as... Figure 4 As shown, it consists of a cascaded channel attention module, a spatial attention module, and an instance attention module. First, a channel attention map is generated using the inter-channel relationships of features. Specifically, this involves aggregating the spatial information of the feature map using average pooling and max pooling operations to generate two different spatial context descriptors: These descriptors represent the feature maps after average pooling and max pooling, respectively; then, these two descriptors are input into a shared network to generate channel attention maps. The shared network consists of a multilayer perceptron (MLP) with a hidden layer, and the channel attention is calculated as follows:

[0103]

[0104] In the formula, σ is the sigmoid activation function. r is the number of hidden layers in the MLP, and C is the number of channels.

[0105] Then, spatial attention maps are generated using the spatial relationships of the features. The specific steps are as follows: average pooling and max pooling operations are applied along the channel axis to generate two 2D feature maps. Then, they are concatenated and convolved using standard convolutional layers to generate a two-dimensional spatial attention map. The calculation method for spatial attention is as follows:

[0106]

[0107] In the formula, σ is the sigmoid activation function, and f 7×7 This indicates a convolution operation with a kernel size of 7×7.

[0108] Finally, instance attention maps are generated using the instance relationships of the features. The specific steps are as follows: the spatial dimension is reduced from 2D to 1D, and then the spatial and channel information of the feature map is aggregated by using average pooling and max pooling operations to generate two different spatial context descriptors: These represent the feature maps after average pooling and max pooling, respectively. These two descriptors are then fed into a shared network to generate instance attention maps. The shared network consists of a multilayer perceptron (MLP) with a hidden layer, and channel attention is calculated as follows:

[0109]

[0110] Where σ is the sigmoid activation function, k is the number of hidden layers in the MLP, and N is the number of instances.

[0111] The above description is only a specific embodiment of this application, but the protection scope of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application.

Claims

1. A high-speed method for detecting small foreign objects based on event and image fusion, characterized in that, include: Step 1: Collect data and build a real dataset; Step 2: Calibrate the real data; Step 3: Construct a simulation dataset for data augmentation; Step 4: Construct a cross-modal feature fusion model using the physical model of motion images and the event stream generation mechanism to adaptively align the event stream and video frame images in time and space; Step 4 includes: Step 4.1: Given the intrinsic and extrinsic parameters of the event camera and the ordinary camera, calculate the homography transformation matrix based on the camera intrinsic and extrinsic parameters and perform coarse spatial alignment; In the formula, K e and K b Let R and t be the intrinsic parameter matrices for the event camera and the normal camera, respectively, and let R and t be the rotation matrix and translation vector in the rigid body transformation of the camera coordinate system, respectively. and These are the reference plane normal vector and the scene depth, respectively. Step 4.2: Perform coarse scene segmentation based on motion patterns, and extract the common far-field region between image frames and event streams; the common far-field region is as follows: In the formula, L T E represents the image frame within the time window T; Te S is the set of all events under the corresponding event window T; S(·) is the coarse segmentation operator; Fine-grained temporal correction function estimation for conventional image frames and events spatially aligned to a common far-field region: In the formula, τ is the fine time correction for aligning the two data streams; F(·) is the regression function used to estimate the time offset mentioned above. Step 4.3: Based on time alignment, a cross-modal feature adaptive selection and fusion module is introduced through multi-stage alignment, as follows: In the formula, , These represent the disparity warp operators between the event stream and the traditional image frame, respectively; , To generate an adaptive weighted mapping graph using subnetworks; , This is the i-th adaptive weighted mapping generated using the subnetwork; , The i-th input event frame and image frame; , For the (i+1)th input event frame and image frame; Step 5: Perform cross-modal data complementarity enhancement fusion on the spatiotemporally aligned event stream and video frame images; Step 6: Train an object detection network to extract features from weak targets by fusing local and global attention.

2. The high-speed weak foreign object detection method based on event and image fusion according to claim 1, characterized in that, Step 3, the simulation data generation steps include: Step 3.1: Use camera parameters to backproject the original image I into three-dimensional space, and at the initial moment, use the camera pose and camera parameters to reproject to obtain image I0; Step 3.2: Calculate the next camera sampling time t based on the adaptive sampling strategy. k ; Step 3.3: At t k New images are generated continuously by projecting the camera pose and camera parameters. k ; Step 3.4: According to I k and I k-1 Calculate image brightness changes and generate all event points between two frames; Step 3.5: Repeat steps 3.2 to 3.4 until the end time T of the camera motion trajectory is reached.

3. The high-speed weak foreign object detection method based on event and image fusion according to claim 1, characterized in that, In step 6, local self-attention extraction includes: Transform the size of the input feature map to Then, self-attention is calculated locally. Where P is the window size of the local self-attention mechanism, C is the number of channels in the output feature map, H is the image height, and W is the image width.

4. The high-speed weak foreign object detection method based on event and image fusion according to claim 3, characterized in that, In step 6, global self-attention extraction includes: Transform the size of the input feature map to Then, calculate self-attention globally. Where G is the window size of the global self-attention mechanism, and C is the number of channels in the output feature map; The attention mechanism is calculated in the form of Q, K, and V, as follows: In the formula, d k Q is a constant; K is the information to be queried; V is the information that exists under this key; and V carries the actual information to be output.

5. The high-speed weak foreign object detection method based on event and image fusion according to claim 1, characterized in that, In step 6, weak target detection is performed based on a multi-level attention mechanism. A channel attention map is generated using the inter-channel relationships of features, as follows: By aggregating the spatial information of the feature maps using average pooling and max pooling operations, two different spatial context descriptors are generated: , , representing the feature maps after average pooling and max pooling, respectively; These two descriptors are fed into a shared network to generate a channel attention map. The shared network consists of a hidden layer MLP, and the channel attention is calculated as follows: In the formula, σ is the sigmoid activation function. , r is the number of hidden layers in the MLP.

6. The high-speed weak foreign object detection method based on event and image fusion according to claim 1, characterized in that, In step 6, a spatial attention map is generated using the spatial relationships of features, including: Apply average pooling and max pooling operations along the channel axis to generate two 2D feature maps. , They are connected and convolved using standard convolutional layers to generate a two-dimensional spatial attention map. The calculation of spatial attention is as follows: In the formula, σ is the sigmoid activation function. This indicates a convolution operation with a kernel size of 7×7.

7. The high-speed weak foreign object detection method based on event and image fusion according to claim 1, characterized in that, In step 6, an instance attention map is generated using the instance relationships of the features, including: The spatial dimension is reduced from 2D to 1D by aggregating the spatial and channel information of the feature map using average pooling and max pooling operations, generating two different spatial context descriptors: , , representing the feature maps after average pooling and max pooling, respectively; Two descriptors are input into a shared network to generate an instance attention graph. The shared network consists of a multilayer perceptron with a hidden layer, and channel attention is calculated as follows: Where σ is the sigmoid activation function, , k is the number of hidden layers in the MLP, and N is the number of instances.

Images