Target detection method and device, electronic equipment, storage medium and product

Abstract

This invention discloses a target detection method, apparatus, electronic device, storage medium, and product. The method includes: determining a first feature map and a second feature map, which are feature maps of different modalities obtained by feature extraction from first image information and first event data, respectively; determining a third feature map based on the first and second feature maps, which is a feature map obtained by performing a bidirectional channel swap operation on the first and second feature maps; the bidirectional channel swap operation is used to replace or reallocate at least one channel feature in one modal feature map whose channel weight is less than the channel weight corresponding to the same channel index in the other modality using the feature of the corresponding channel in the other modality; and performing moving target detection based on the third feature map. This solution can improve the robustness and accuracy of moving target detection.

Description

Technical Field

[0001] This invention relates to the field of computer vision and intelligent sensing technology, and in particular to a target detection method, device, electronic device, storage medium, and product. Background Technology

[0002] Object detection, a crucial research area in computer vision, is widely applied in scenarios such as autonomous driving, intelligent transportation, video surveillance, and robotic environmental perception. Most existing object detection methods rely on image information acquired by frame-based RGB cameras, using convolutional neural networks to extract visual features and perform object localization and recognition. However, in complex scenes involving high-speed motion, low illumination, and strong light changes, RGB cameras are susceptible to motion blur, underexposure, and limited dynamic range, leading to degraded image quality and significantly reducing the accuracy and stability of object detection. Summary of the Invention

[0003] This invention provides a target detection method, apparatus, electronic device, storage medium, and product to enhance the collaborative perception capability of moving targets in complex environments and improve the robustness and accuracy of moving target detection.

[0004] According to one aspect of the present invention, a target detection method is provided, the method comprising: A first feature map and a second feature map are determined. The first feature map and the second feature map are feature maps of different modalities obtained by feature extraction of the first image information and the first event data, respectively. The first image information and the first event data are acquired by an RGB camera and an event camera respectively, facing the same acquisition area. A third feature map is determined based on the first feature map and the second feature map. The third feature map is obtained by performing a bidirectional channel exchange operation on the first feature map and the second feature map. The bidirectional channel exchange operation is used to replace or reallocate at least one channel feature in the feature map of one modality whose channel weight is less than the channel weight corresponding to the same channel index of the other modality, using the feature of the channel corresponding to the same channel index of the other modality. Moving target detection is performed based on the third feature map.

[0005] According to another aspect of the present invention, a target detection apparatus is provided, the apparatus comprising: The determination module is used to determine a first feature map and a second feature map. The first feature map and the second feature map are feature maps of different modalities obtained by feature extraction of the first image information and the first event data, respectively. The first image information and the first event data are acquired by an RGB camera and an event camera respectively, facing the same acquisition area. The determining module is further configured to determine a third feature map based on the first feature map and the second feature map. The third feature map is a feature map obtained by performing a bidirectional channel exchange operation on the first feature map and the second feature map. The bidirectional channel exchange operation is used to replace or reallocate at least one channel feature in the feature map of one modality whose channel weight is less than the channel weight corresponding to the same channel index of the other modality for two different modal feature maps. The detection module is used to detect moving targets based on the third feature map.

[0006] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, which enables the at least one processor to perform the target detection method according to any embodiment of the present invention.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the target detection method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the target detection method described in any embodiment of the present invention.

[0009] The technical solution of this invention allows the event camera to asynchronously perceive brightness change events in a scene, offering advantages such as high temporal resolution, low latency, and high dynamic range. By synchronously acquiring first image information and first event data from both the RGB camera and the event camera towards the same acquisition area, and by extracting feature maps from the first image information and first event data, static texture and color features of the RGB modality can be preserved, while simultaneously capturing the dynamic motion and edge change features of moving targets in the event modality, thus overcoming the limitation of dynamic blurring with a single RGB camera. For the first and second feature maps of two different modalities, the channel weight in one modality's feature map is less than that of the same channel in the other modality. At least one channel feature corresponding to the channel index of a channel weight is replaced or redistributed using the channel feature corresponding to the same channel index of another modality. The channel allocation relationship between the first feature map and the second feature map is dynamically adjusted according to the channel population of different modal feature maps. This allows for bidirectional interaction between the channel features in the first feature map and the channel features in the second feature map at the channel level, achieving adaptive complementary fusion of multimodal features. This enables the third feature map to simultaneously possess the advantages of RGB static detail and event dynamic response, thereby improving the detection stability in complex scenes. Furthermore, moving target detection based on the third feature map can significantly improve the accuracy, robustness, and anti-interference ability of moving target detection.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart of a target detection method provided in an embodiment of the present invention; Figure 2 This is a flowchart of another target detection method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a target detection device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device for implementing a target detection method according to an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] In one embodiment, Figure 1 This is a flowchart of a target detection method provided in an embodiment of the present invention. This embodiment is applicable to the detection of moving targets. The method can be executed by a target detection device, which can be implemented in hardware and / or software and can be configured in an electronic device.

[0016] like Figure 1 As shown, the target detection method in this embodiment may include the following process: S110. Determine the first feature map and the second feature map. The first feature map and the second feature map are feature maps of different modalities obtained by extracting features from the first image information and the first event data, respectively. The first image information and the first event data are acquired by using an RGB camera and an event camera to collect data from the same acquisition area.

[0017] The first image information can be image data acquired using an RGB camera. It can be a single frame or an image sequence, and includes at least one spatial detail information among static texture, color, and target contour. The format of the first image information can be RGB three-channel. The RGB camera is used to capture the light color and spatial details of the acquisition area, outputting RGB three-channel image data to record at least one spatial detail information among static texture, color, and target contour.

[0018] The first event data can be an asynchronous event stream collected by an event camera and an RGB camera targeting the same acquisition area. This event data can include pixel position, timestamp, and polarity. The event camera can capture events where pixel brightness changes abruptly within the acquisition area, outputting data in the form of discrete quadruples (x, y, t, p) to record the temporal information of the target's dynamic motion trajectory and pixel brightness changes. The overlap between the acquisition areas of the RGB camera and the event camera is greater than a preset overlap threshold. By synchronously acquiring the same acquisition area, the spatial and temporal alignment of the two types of data is ensured, guaranteeing that the RGB camera and the event camera acquire information about the same target and the same scene.

[0019] The first feature map can be an RGB modal feature map obtained by performing feature extraction on the first image information. The first feature map carries deep semantic features of the RGB modality, such as high-order features of target contours and textures. The second feature map can be an event modal feature map obtained by performing feature extraction on the first event data. The format of the second feature map is the same as that of the first feature map. Figure 1 The second feature map carries the deep semantic features of the event modality, such as the target motion trajectory and the high-order features of edge brightness changes, and together with the first feature map, they form a bimodal feature map.

[0020] In the above scheme, the RGB camera and the event camera face the same acquisition area to collect data, avoiding spatial misalignment problems, and synchronous acquisition ensures temporal matching. At the same time, the first feature map focuses on static spatial details, and the second feature map focuses on dynamic temporal changes. By using the first and second feature maps, the advantages of rich static details of the RGB modality are retained, while the advantages of fast dynamic response and no motion blur of the event modality are captured, making up for the shortcomings of single-modality detection.

[0021] S120. Determine a third feature map based on the first feature map and the second feature map. The third feature map is a feature map obtained by performing a bidirectional channel exchange operation on the first feature map and the second feature map. The bidirectional channel exchange operation is used to replace or reallocate at least one channel feature in the feature map of one modality whose channel weight is less than the channel weight corresponding to the same channel index of the other modality for two different modal feature maps.

[0022] The third feature map can be a bimodal fusion feature map generated by bidirectional channel swapping of the first and second feature maps. Each channel is the optimal channel feature of the corresponding channel in both modalities, combining the advantages of RGB static details and event dynamic response. The bidirectional channel swapping operation can perform cross-modal channel-level complementarity based on the comparison of the channel weights of the feature maps of the first and second modalities. At least one channel feature in any modal feature map whose channel weight is less than the channel weight of the same channel index in the other modality is replaced with the channel feature of the same channel index in the other modality. This integrates the advantages of RGB and event modalities to obtain channel interaction features, which not only retain the static texture and contour features of the RGB modality but also integrate the dynamic motion and edge change features of the event modality, thus achieving complementary fusion of bimodal information.

[0023] Channel weights in a feature map characterize the importance of each channel feature. These weights are generated based on the global statistics of each channel. Higher channel weights indicate a greater contribution of the corresponding channel feature to moving target detection; conversely, lower channel weights indicate a smaller contribution. Channel features can be all spatial features within a single channel of the feature map. Each channel corresponds to a semantic feature and is the smallest unit for bidirectional channel exchange operations to complement channel features. Channels with the same channel index can be channels with the same channel number in the feature maps of two modalities (e.g., the channel with channel index k in the first feature map corresponds to the channel with channel index k in the second feature map). This enables precise channel-by-channel matching and serves as the basis for replacing or redistributing channel features between the two modalities.

[0024] In an optional embodiment, determining the third feature map based on the first feature map and the second feature map may include: for the feature maps of the first and second modalities, the channel weights of each channel in the feature maps of the two modalities, comparing the channel weights of each channel one by one, selecting at least one channel in the feature map of any modality whose channel weight is less than the channel weight corresponding to the same channel index in the other modality, and replacing or redistributing the channel feature of the at least one channel selected in the feature map of any modality with the channel feature of the channel corresponding to the same channel index in the feature map of the other modality, thereby generating a third feature map that has the advantages of both modalities.

[0025] In one optional embodiment, replacing or reallocating features of the corresponding channel with the same channel index of another modality can include: directly using the channel features of the same channel index of another modality to cover at least one channel in any modality feature map whose channel weight is less than the channel weight corresponding to the same channel index of another modality, retaining advantageous features and eliminating weak features. This operation is simple, computationally inexpensive, and suitable for scenarios with low tolerance for redundant information. In one alternative embodiment, replacing or redistributing the features of the channel corresponding to the same channel index in another modality may include: not completely covering the channel features of at least one channel in the feature map of any modality whose channel weight is less than the channel weight corresponding to the same channel index in the feature map of another modality, but combining the channel features of the channel corresponding to the same channel index in the feature map of another modality, and weighting and fusing the channel features of the channel corresponding to the same channel index in the feature maps of the two modalities based on the weight ratio of the channel corresponding to the same channel index in the feature maps of the two modalities, so as to retain the advantageous features while taking into account a small amount of original channel information, thereby improving the smoothness and robustness of feature fusion.

[0026] The above scheme addresses the problem of insufficient representation of single-modal features by using channel-level bidirectional complementarity. By comparing the channel weights of the same channel index, each channel in the feature maps of two different modalities retains the optimal features of both modalities, integrating the advantages of dual modalities, strengthening the overall representation capability of features, providing high-quality feature maps for moving target detection, adapting to different dynamic scenarios such as high speed and low speed, as well as complex lighting scenarios such as strong light and weak light, taking into account both dynamic response and static recognition capabilities, and improving detection robustness.

[0027] S130. Detect moving targets based on the third feature map.

[0028] The third feature map integrates the dual advantages of RGB modal static texture and contour details as well as event modal dynamic motion and blur-free response. It can accurately adapt to different dynamic scenes such as high speed and low speed and complex lighting environments. It can effectively avoid the problems of missed detection and false detection in a single modality, enhance the recognition of target features, suppress background interference, accurately identify and locate moving targets, and take into account detection efficiency. It provides more accurate feature support for moving target detection tasks and adapts to the moving target detection needs of various scenarios.

[0029] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments to perform moving target detection based on the third feature map, which may include the following steps: Spatial attention weighted fusion is performed on the third feature map to generate the fourth feature map; after multi-scale feature enhancement of the fourth feature map, moving target detection is performed to output the target detection result.

[0030] Spatial attention-weighted fusion can be used to optimize the spatial dimension of feature maps. By generating a spatial attention weight map of the feature map, weights are applied to each spatial location of the feature map, amplifying the features of the target region and suppressing background interference, thus achieving feature focusing. The fourth feature map can be generated by spatial attention-weighted fusion of the third feature map. The fourth feature map focuses on the target region at the spatial level, significantly reducing background redundancy interference and resulting in higher target feature recognition.

[0031] Multi-scale feature enhancement can be used to optimize the features of moving targets of different sizes in the fourth feature map. By combining multi-scale convolution with feature fusion, the feature representation of moving targets of different sizes is strengthened, solving the problem of poor adaptability of single-scale features to targets of different sizes. Moving target detection can extract semantic information from the feature map to complete the identification and localization of moving targets, and output the target category and location coordinates. The target detection results can include the bounding box coordinates of the moving target, the target category label, and the confidence score of the detection result.

[0032] In one optional embodiment, performing spatial attention weighted fusion on the third feature map to generate a fourth feature map may include: performing channel average pooling and channel max pooling on the third feature map to obtain the pooling result of the third feature map; generating a spatial attention weight map of the third feature map based on the pooling result of the third feature map; and performing feature weighted fusion on the third feature map based on the spatial attention weight map of the third feature map to obtain the fourth feature map.

[0033] Channel average pooling is a pooling operation along the channel dimension of the feature map, averaging the values ​​of all pixels within each channel. Channel max pooling is a pooling operation along the channel dimension of the feature map, maximizing the value of all pixels within each channel. Spatial attention weight maps are used to quantify the importance of each spatial location in the feature map. Feature weighted fusion of the third feature map can be achieved by using spatial attention weight maps to perform pixel-by-pixel weighted operations on the third feature map, amplifying features in important regions, suppressing redundant interference, and completing feature optimization.

[0034] In an optional embodiment, channel-level average pooling and channel-level max pooling are performed on the third feature map to obtain the pooling result of the third feature map. This includes: performing two channel-level pooling operations, channel-level average pooling and channel-level max pooling, on the third feature map as input. Specifically, channel-level average pooling calculates the average value of all spatial pixels within each channel of the third feature map; similarly, channel-level max pooling extracts the maximum value of all spatial pixels within each channel of the third feature map. The two pooling results are combined to generate the pooling result of the third feature map, avoiding the limitations of a single pooling method.

[0035] In an optional embodiment, generating a spatial attention weight map of the third feature map based on the pooling result of the third feature map includes: fusing the pooling result of the third feature map, and then performing activation processing through an activation function (such as the Sigmoid function) to map the feature values ​​to the 0-1 interval, thereby obtaining the spatial attention weight map of the third feature map.

[0036] In an optional embodiment, a fourth feature map is obtained by weighted fusion of the third feature map based on the spatial attention weight map of the third feature map. This includes: performing dimensional adaptation and operation on the spatial attention weight map and the third feature map using a pixel-by-pixel weighted multiplication method, so that the weight of each pixel in the spatial attention weight map is applied to all channel pixels at the corresponding spatial location in the third feature map to obtain the fourth feature map. In the third feature map, the target region has a high weight, resulting in a magnified feature value and enhanced feature response; the background region has a low weight, resulting in a weakened feature value and suppressed redundant interference.

[0037] The above scheme combines multi-timescale event data aggregation with spatial attention weighted fusion mechanism to enhance robustness in complex environments such as high-speed motion, low illumination, and drastic changes in lighting. By accurately distinguishing the target from the background through spatial weights, it amplifies the target feature response, reduces background redundancy, solves the problem of background interference that may remain in the third feature map, retains the dual-modal complementary advantages of the third feature map, and focuses on the core target area at the spatial level to enhance the target area response and suppress background interference.

[0038] In an optional embodiment, performing multi-scale feature enhancement on the fourth feature map and then outputting the target detection result for moving target detection may include: constructing a feature pyramid structure to fuse features of different scales in the fourth feature map to obtain a fifth feature map, and then detecting the moving target based on the fifth feature map and outputting the target detection result.

[0039] The feature pyramid structure is a multi-scale feature processing architecture that constructs feature layers of different resolutions by performing multi-scale downsampling, upsampling, and feature fusion on the input feature map. It can simultaneously capture features of small, medium, and large-sized targets, adapting to the needs of moving target detection at various scales. The fifth feature map can be the final optimized feature map after multi-scale feature fusion processed by the feature pyramid structure. It integrates features from various scales in the fourth feature map, combining detailed accuracy with global semantic information, and is suitable for moving target detection at all scales.

[0040] In one optional embodiment, a fifth feature map is obtained by fusing features of different scales in the fourth feature map through the construction of a feature pyramid structure. This includes: constructing a multi-level feature pyramid structure; generating multiple low-resolution feature layers based on the fourth feature map through convolutional downsampling to focus on capturing the global semantic features of large-sized targets; aligning the lowest-level low-resolution features with the corresponding high-resolution features through upsampling; and fusing the aligned features at each level by adding or concatenating elements to integrate the detailed information of the high-resolution features with the global information of the low-resolution features, ultimately outputting a fifth feature map that integrates features across all scales. This solves the problem of missing small targets and inaccurate localization of large targets due to single-scale features.

[0041] In an optional embodiment, moving target detection based on the fifth feature map includes: inputting the enhanced fifth feature map into a detection network (such as YOLO or Faster R-CNN) to complete target recognition and localization, and finally outputting a detection result containing target coordinates, category, and confidence level.

[0042] The above scheme makes up for the shortcomings of single-scale features, while strengthening the feature representation of moving targets of different sizes such as large, medium and small, reducing the problems of missed detection of small targets and inaccurate positioning of large targets. At the same time, by combining the triple advantages of dual-modal complementarity, spatial target focusing and multi-scale enhancement, it can effectively adapt to high-speed, low-speed and complex lighting scenarios, reduce false detection and false detection rates, and improve the reliability of detection results.

[0043] The technical solution of this invention allows the event camera to asynchronously perceive brightness change events in a scene, offering advantages such as high temporal resolution, low latency, and high dynamic range. By synchronously acquiring first image information and first event data from both the RGB camera and the event camera towards the same acquisition area, and by extracting feature maps from the first image information and first event data, static texture and color features of the RGB modality can be preserved, while simultaneously capturing the dynamic motion and edge change features of moving targets in the event modality, thus overcoming the limitation of dynamic blurring with a single RGB camera. For the first and second feature maps of two different modalities, the channel weight in one modality's feature map is less than that of the same channel in the other modality. At least one channel feature corresponding to the channel index of a channel weight is replaced or redistributed using the channel feature corresponding to the same channel index of another modality. The channel allocation relationship between the first feature map and the second feature map is dynamically adjusted according to the channel population of different modal feature maps. This allows for bidirectional interaction between the channel features in the first feature map and the channel features in the second feature map at the channel level, achieving adaptive complementary fusion of multimodal features. This enables the third feature map to simultaneously possess the advantages of RGB static detail and event dynamic response, thereby improving the detection stability in complex scenes. Furthermore, moving target detection based on the third feature map can significantly improve the accuracy, robustness, and anti-interference ability of moving target detection.

[0044] In one embodiment, Figure 2 This is a flowchart of another target detection method provided by the embodiment of the present invention. The technical solution of this embodiment further optimizes the process of determining the third feature map based on the first feature map and the second feature map in the above embodiments. This embodiment can be combined with various optional solutions in one or more of the above embodiments.

[0045] like Figure 2 As shown, the target detection method in this embodiment may include the following process: S210. Determine the first feature map and the second feature map. The first feature map and the second feature map are feature maps of different modalities obtained by extracting features from the first image information and the first event data, respectively. The first image information and the first event data are obtained by using an RGB camera and an event camera to collect data from the same acquisition area.

[0046] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments to determine the first feature map and the second feature map, and may include the following steps 1a-3a: Step 1a: Extract features from the first image information to obtain the first feature map.

[0047] Step 2a: Determine multiple second event data associated with the first event data. The multiple second event data are event data within each event time window obtained by dividing the first event data into multiple event time windows according to multiple event time windows. The multiple event time windows are multiple event time windows of different durations constructed based on the exposure time or frame period of the first image information. The duration of each of the multiple event time windows is no greater than the duration corresponding to the exposure time or frame period of the first image information.

[0048] Step 3a: Extract features from multiple second event data and aggregate the features to obtain a second feature map.

[0049] The first image information and the first event data are image information and asynchronous event streams acquired by the RGB camera and the event camera respectively, facing the same acquisition area. The second event data can be a portion of the first event data within a single event time window after the first event data has been divided into multi-scale event time windows, with each second event data corresponding to an event time window of a certain length.

[0050] The event time window is a time interval used to divide event data to adapt to the asynchronous output characteristics of the event camera. It is usually constructed based on the exposure time or frame period of the RGB image, and multiple event time windows with different durations are set. The duration of each event time window does not exceed the exposure time or frame period of the RGB image, thus splitting the first event data into multiple second event data.

[0051] Exposure time is the duration of light exposure for a single frame of an image, and frame period is the acquisition interval between two adjacent frames. It serves as the basis for constructing the event time window, ensuring temporal matching between the first event data and the first image information. Feature aggregation can be the operation of integrating features extracted from multiple second event data into a single feature map through feature concatenation and weighted fusion. This eliminates the redundancy of multi-scale features, preserves the advantages of dynamic features at each scale, and outputs a feature carrier in a unified format. Second Feature Map In an optional embodiment, the process of extracting and aggregating features from multiple second event data to obtain a second feature map may include: projecting and mapping multiple second event data to obtain first event feature maps corresponding to different time scales; performing pooling and scale alignment processing on the first event feature maps corresponding to different time scales, and fusing the processed first event feature maps corresponding to different time scales to obtain a second event feature map; and extracting features from the second event feature map to obtain a second feature map.

[0052] Optionally, multiple second event data are projected and mapped to obtain first event feature maps corresponding to different time scales. This includes: performing projection mapping operations on each of the multiple second event data, mapping the spatial coordinates of the second event data to a fixed-size pixel grid, and combining the temporal information of the timestamps and the intensity information of the brightness change polarity in the second event data to transform the second event data into a two-dimensional structured feature map, thus obtaining the corresponding first event feature map. Second event data at different time scales correspond to first event feature maps at different time scales, ensuring that the dynamic features at each time scale can be structurally represented.

[0053] Optionally, the first event feature maps corresponding to different time scales are subjected to pooling and scale alignment processing, including: performing average pooling or max pooling operations on each first event feature map to compress spatial dimensions, retain key dynamic features, reduce redundant information and subsequent computation, and improve feature robustness; for the pooled multi-scale first event feature maps, upsampling or downsampling is used to ensure that the spatial resolution and number of channels of all first event feature maps are consistent, avoiding fusion failure caused by scale misalignment. Optionally, the second event feature maps are subjected to feature extraction to obtain a second feature map, including: using a convolutional neural network to perform multi-layer feature extraction on the second event feature map, and outputting a second feature map that matches the first feature map in terms of spatial resolution and number of channels.

[0054] S220. Perform global pooling operations on the first feature map and the second feature map respectively to obtain the channel statistics of the first feature map and the channel statistics of the second feature map.

[0055] S230. Based on the channel statistics of the first feature map and the channel statistics of the second feature map, generate first channel weight information and second channel weight information. The first channel weight information is used to quantify the importance of each channel feature in the first feature map when the RGB camera acquires the first image information. The second channel weight information is used to quantify the importance of each channel feature in the second feature map when the event camera acquires the first event data.

[0056] S240. Based on the weight information of the first channel and the weight information of the second channel, perform a bidirectional channel swap operation on the first feature map and the second feature map to obtain the third feature map.

[0057] The third feature map is obtained by performing a bidirectional channel exchange operation on the first and second feature maps. The bidirectional channel exchange operation is used to replace or reallocate at least one channel feature in the feature map of one modality whose channel weight is less than the channel weight of the same channel index in the other modality with the channel feature of the same channel index in the other modality.

[0058] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments. Based on the first channel weight information and the second channel weight information, a bidirectional channel swap operation is performed on the first feature map and the second feature map to obtain a third feature map. This may include the following steps 1b-3b: Step 1b: If, based on the first channel weight information and the second channel weight information, it is detected that the channel weight of the channel corresponding to the first channel index of the first feature map is greater than the channel weight of the channel corresponding to the second channel index of the second feature map, the channel feature of the channel corresponding to the first channel index of the first feature map is replaced or reallocated using the channel feature of the channel corresponding to the second channel index of the second feature map, and the channel feature of the channel corresponding to the second channel index of the second feature map is retained; the first channel index and the second channel index are the same channel index corresponding to the first feature map and the second feature map.

[0059] Step 2b: If, based on the first channel weight information and the second channel weight information, it is detected that the channel weight of the channel corresponding to the first channel index of the first feature map is less than the channel weight of the channel corresponding to the second channel index of the second feature map, then the channel feature of the channel corresponding to the second channel index of the second feature map is replaced or reallocated using the channel feature of the channel corresponding to the first channel index of the first feature map, and the channel feature of the channel corresponding to the first channel index of the first feature map is retained.

[0060] Step 3b: Perform feature fusion on the feature results after bidirectional channel swapping between the first feature map and the second feature map to obtain the third feature map.

[0061] S350. Detect moving targets based on the third feature map.

[0062] The technical solution of this invention allows the event camera to asynchronously perceive brightness change events in a scene, offering advantages such as high temporal resolution, low latency, and high dynamic range. By synchronously acquiring first image information and first event data from both the RGB camera and the event camera towards the same acquisition area, and by extracting feature maps from the first image information and first event data, static texture and color features of the RGB modality can be preserved, while simultaneously capturing the dynamic motion and edge change features of moving targets in the event modality, thus overcoming the limitation of dynamic blurring with a single RGB camera. For the first and second feature maps of two different modalities, the channel weight in one modality's feature map is less than that of the same channel in the other modality. At least one channel feature corresponding to the channel index of a channel weight is replaced or redistributed using the channel feature corresponding to the same channel index of another modality. The channel allocation relationship between the first feature map and the second feature map is dynamically adjusted according to the channel population of different modal feature maps. This allows for bidirectional interaction between the channel features in the first feature map and the channel features in the second feature map at the channel level, achieving adaptive complementary fusion of multimodal features. This enables the third feature map to simultaneously possess the advantages of RGB static detail and event dynamic response, thereby improving the detection stability in complex scenes. Furthermore, moving target detection based on the third feature map can significantly improve the accuracy, robustness, and anti-interference ability of moving target detection.

[0063] In one embodiment, Figure 3 This is a schematic diagram of the structure of a target detection device provided in an embodiment of the present invention. This embodiment is applicable to the detection of moving targets. The target detection device can be implemented in hardware and / or software and can be configured in an electronic device.

[0064] like Figure 3 As shown, the target detection device in this embodiment may include the following: The determining module 310 is used to determine the first feature map and the second feature map. The first feature map and the second feature map are feature maps of different modalities obtained by feature extraction of the first image information and the first event data, respectively. The first image information and the first event data are acquired by the RGB camera and the event camera respectively, facing the same acquisition area. The determining module 310 is further configured to determine a third feature map based on the first feature map and the second feature map. The third feature map is a feature map obtained by performing a bidirectional channel exchange operation on the first feature map and the second feature map. The bidirectional channel exchange operation is used to replace or reallocate at least one channel feature in the feature map of one modality whose channel weight is less than the channel weight corresponding to the same channel index of the other modality for two different modal feature maps. The detection module 320 is used to detect moving targets based on the third feature map.

[0065] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments to determine the first feature map and the second feature map, including: The first feature map is obtained by performing feature extraction on the first image information; A plurality of second event data associated with the first event data are determined. The plurality of second event data are event data within each event time window obtained by dividing the first event data into multiple event time windows according to multiple event time windows. The plurality of event time windows are multiple event time windows of different durations constructed based on the exposure time or frame period of the first image information. The duration of each of the plurality of event time windows is not greater than the duration corresponding to the exposure time or the duration corresponding to the frame period of the first image information. The second feature map is obtained by extracting features and aggregating features based on the multiple second event data.

[0066] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments to extract features and aggregate features based on the plurality of second event data to obtain the second feature map, including: The multiple second event data are projected and mapped to obtain first event feature maps corresponding to different time scales; The first event feature maps corresponding to different time scales are pooled and scale aligned, and the processed first event feature maps corresponding to different time scales are fused to obtain the second event feature map. The second feature map is obtained by performing feature extraction on the second event feature map.

[0067] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments to determine a third feature map based on the first feature map and the second feature map, including: Global pooling is performed on the first feature map and the second feature map respectively to obtain the channel statistics of the first feature map and the channel statistics of the second feature map; Based on the channel statistics of the first feature map and the channel statistics of the second feature map, first channel weight information and second channel weight information are generated. The first channel weight information is used to quantify the importance of each channel feature in the first feature map when the RGB camera acquires the first image information. The second channel weight information is used to quantify the importance of each channel feature in the second feature map when the event camera acquires the first event data. Based on the first channel weight information and the second channel weight information, a bidirectional channel swap operation is performed on the first feature map and the second feature map to obtain a third feature map.

[0068] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments. Based on the first channel weight information and the second channel weight information, a bidirectional channel swap operation is performed on the first feature map and the second feature map to obtain a third feature map, including: If, based on the first channel weight information and the second channel weight information, it is detected that the channel weight of the channel corresponding to the first channel index of the first feature map is greater than the channel weight of the channel corresponding to the second channel index of the second feature map, the channel feature of the channel corresponding to the first channel index of the first feature map is replaced or reallocated using the channel feature of the channel corresponding to the second channel index of the second feature map, and the channel feature of the channel corresponding to the second channel index of the second feature map is retained; the first channel index and the second channel index are the same channel index corresponding to the first feature map and the second feature map; If, based on the first channel weight information and the second channel weight information, it is detected that the channel weight of the channel corresponding to the first channel index of the first feature map is less than the channel weight of the channel corresponding to the second channel index of the second feature map, then the channel feature of the channel corresponding to the second channel index of the second feature map is replaced or reallocated using the channel feature of the channel corresponding to the first channel index of the first feature map, and the channel feature of the channel corresponding to the first channel index of the first feature map is retained. The third feature map is obtained by fusing the feature results of the bidirectional channel exchange operation between the first feature map and the second feature map.

[0069] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments to perform moving target detection based on the third feature map, including: Spatial attention weighted fusion is performed on the third feature map to generate a fourth feature map; After performing multi-scale feature enhancement on the fourth feature map, moving target detection is performed and the target detection result is output.

[0070] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments to perform spatial attention weighted fusion on the third feature map to generate a fourth feature map, including: Channel average pooling and channel max pooling are performed on the third feature map to obtain the pooling result of the third feature map; A spatial attention weight map of the third feature map is generated based on the pooling result of the third feature map; The fourth feature map is obtained by performing feature weighted fusion on the third feature map based on the spatial attention weight map of the third feature map.

[0071] In one embodiment, this embodiment can be combined with various optional schemes in one or more of the above embodiments to perform multi-scale feature enhancement on the fourth feature map and then perform moving target detection to output target detection results, including: By constructing a feature pyramid structure, features of different scales in the fourth feature map are fused to obtain a fifth feature map, and moving targets are detected based on the fifth feature map.

[0072] The technical solution of this invention allows the event camera to asynchronously perceive brightness change events in a scene, offering advantages such as high temporal resolution, low latency, and high dynamic range. By synchronously acquiring first image information and first event data from both the RGB camera and the event camera towards the same acquisition area, and by extracting feature maps from the first image information and first event data, static texture and color features of the RGB modality can be preserved, while simultaneously capturing the dynamic motion and edge change features of moving targets in the event modality, thus overcoming the limitation of dynamic blur in a single RGB camera. For the first and second feature maps of two different modalities, the channel weight in one modality's feature map is less than the same channel weight in the other modality. At least one channel feature corresponding to the channel weight of the index is replaced or redistributed using the same channel index feature of the other modality. The channel allocation relationship between the first feature map and the second feature map is dynamically adjusted according to the channel population of different modal feature maps. This allows the channel features in the first feature map and the channel features in the second feature map to interact bidirectionally at the channel level, achieving adaptive complementary fusion of multimodal features. This enables the third feature map to simultaneously possess the advantages of RGB static detail and event dynamic response to improve detection stability in complex scenes. Furthermore, moving target detection based on the third feature map can significantly improve the accuracy, robustness, and anti-interference ability of moving target detection.

[0073] The target detection device provided in the embodiments of the present invention can execute the target detection method provided in any of the embodiments of the present invention, and has the corresponding functions and beneficial effects of executing the target detection method. For details, please refer to the relevant operations of the target detection method in the foregoing embodiments.

[0074] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of the present invention.

[0075] In one embodiment, Figure 4 This is a structural block diagram of an electronic device provided in an embodiment of the present invention, such as... Figure 4 The diagram illustrates a schematic representation of an electronic device 10 that can be used to implement embodiments of the present invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0076] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., which is communicatively connected to the at least one processor 11. The memory stores a computer program that can be executed by the at least one processor 11, and the computer program is executed by the at least one processor 11 to enable the at least one processor 11 to execute the target detection method provided by the present invention.

[0077] The processor 11 can perform various appropriate actions and processes based on a computer program stored in the read-only memory (ROM) 12 or a computer program loaded from the storage unit 18 into the random access memory (RAM) 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0078] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0079] Processor 11 can be various general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, digital signal processors (DSPs), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as the target detection method provided by this invention.

[0080] In some embodiments, the target detection method provided herein may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the target detection method by any other suitable means (e.g., by means of firmware).

[0081] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard parts (ASSPs), systems-on-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0082] Computer programs for implementing the target detection method of the present invention can be written in any combination of one or more programming languages. These computer programs can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer programs can be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0083] In the context of this invention, a computer-readable storage medium stores computer instructions that are used to cause a processor to execute and implement the target detection method provided by this invention.

[0084] The present invention also provides a computer program product comprising a computer program that, when executed by a processor, implements the target detection method provided according to embodiments of the present invention. A computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, the computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0085] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0086] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0087] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0088] This invention also provides a computer program product, including a computer program that, when executed by a processor, can implement the target detection method provided in any embodiment of this application.

[0089] In the implementation of the computer program product, computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0090] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0091] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A target detection method, characterized in that, The method includes: A first feature map and a second feature map are determined. The first feature map and the second feature map are feature maps of different modalities obtained by feature extraction of the first image information and the first event data, respectively. The first image information and the first event data are acquired by an RGB camera and an event camera respectively, facing the same acquisition area. A third feature map is determined based on the first feature map and the second feature map. The third feature map is obtained by performing a bidirectional channel exchange operation on the first feature map and the second feature map. The bidirectional channel exchange operation is used to replace or reallocate at least one channel feature in the feature map of one modality whose channel weight is less than the channel weight corresponding to the same channel index of the other modality, using the feature of the channel corresponding to the same channel index of the other modality. Moving target detection is performed based on the third feature map.

2. The method according to claim 1, characterized in that, Determining the first feature map and the second feature map includes: The first feature map is obtained by performing feature extraction on the first image information; A plurality of second event data associated with the first event data are determined. The plurality of second event data are event data within each event time window obtained by dividing the first event data into multiple event time windows according to multiple event time windows. The plurality of event time windows are multiple event time windows of different durations constructed based on the exposure time or frame period of the first image information. The duration of each of the plurality of event time windows is not greater than the duration corresponding to the exposure time or the duration corresponding to the frame period of the first image information. The second feature map is obtained by extracting features and aggregating features based on the multiple second event data.

3. The method according to claim 2, characterized in that, The second feature map is obtained by extracting and aggregating features based on the multiple second event data, including: The multiple second event data are projected and mapped to obtain first event feature maps corresponding to different time scales; The first event feature maps corresponding to different time scales are pooled and scale aligned, and the processed first event feature maps corresponding to different time scales are fused to obtain the second event feature map. The second feature map is obtained by performing feature extraction on the second event feature map.

4. The method according to claim 1, characterized in that, Determining a third feature map based on the first and second feature maps includes: Global pooling is performed on the first feature map and the second feature map respectively to obtain the channel statistics of the first feature map and the channel statistics of the second feature map; Based on the channel statistics of the first feature map and the channel statistics of the second feature map, first channel weight information and second channel weight information are generated. The first channel weight information is used to quantify the importance of each channel feature in the first feature map when the RGB camera acquires the first image information. The second channel weight information is used to quantify the importance of each channel feature in the second feature map when the event camera acquires the first event data. Based on the first channel weight information and the second channel weight information, a bidirectional channel swap operation is performed on the first feature map and the second feature map to obtain a third feature map.

5. The method according to claim 4, characterized in that, Based on the first channel weight information and the second channel weight information, a bidirectional channel swap operation is performed on the first feature map and the second feature map to obtain a third feature map, including: If, based on the first channel weight information and the second channel weight information, it is detected that the channel weight of the channel corresponding to the first channel index of the first feature map is greater than the channel weight of the channel corresponding to the second channel index of the second feature map, the channel feature of the channel corresponding to the first channel index of the first feature map is replaced or reallocated using the channel feature of the channel corresponding to the second channel index of the second feature map, and the channel feature of the channel corresponding to the second channel index of the second feature map is retained; the first channel index and the second channel index are the same channel index corresponding to the first feature map and the second feature map; If, based on the first channel weight information and the second channel weight information, it is detected that the channel weight of the channel corresponding to the first channel index of the first feature map is less than the channel weight of the channel corresponding to the second channel index of the second feature map, then the channel feature of the channel corresponding to the second channel index of the second feature map is replaced or reallocated using the channel feature of the channel corresponding to the first channel index of the first feature map, and the channel feature of the channel corresponding to the first channel index of the first feature map is retained. The third feature map is obtained by fusing the feature results of the bidirectional channel exchange operation between the first feature map and the second feature map.

6. The method according to claim 1, characterized in that, Moving target detection based on the third feature map includes: Spatial attention weighted fusion is performed on the third feature map to generate a fourth feature map; After performing multi-scale feature enhancement on the fourth feature map, moving target detection is performed and the target detection result is output.

7. The method according to claim 6, characterized in that, Spatial attention-weighted fusion is performed on the third feature map to generate a fourth feature map, including: Channel average pooling and channel max pooling are performed on the third feature map to obtain the pooling result of the third feature map; A spatial attention weight map of the third feature map is generated based on the pooling result of the third feature map; The fourth feature map is obtained by performing feature weighted fusion on the third feature map based on the spatial attention weight map of the third feature map.

8. The method according to claim 6, characterized in that, After performing multi-scale feature enhancement on the fourth feature map, moving target detection is performed to output the target detection result, including: By constructing a feature pyramid structure, features of different scales in the fourth feature map are fused to obtain a fifth feature map, and moving targets are detected based on the fifth feature map.

9. A target detection device, characterized in that, The device includes: The determination module is used to determine a first feature map and a second feature map. The first feature map and the second feature map are feature maps of different modalities obtained by feature extraction of the first image information and the first event data, respectively. The first image information and the first event data are acquired by an RGB camera and an event camera respectively, facing the same acquisition area. The determining module is further configured to determine a third feature map based on the first feature map and the second feature map. The third feature map is a feature map obtained by performing a bidirectional channel exchange operation on the first feature map and the second feature map. The bidirectional channel exchange operation is used to replace or reallocate at least one channel feature in the feature map of one modality whose channel weight is less than the channel weight corresponding to the same channel index of the other modality for two different modal feature maps. The detection module is used to detect moving targets based on the third feature map.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the target detection method according to any one of claims 1-8.

11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the target detection method according to any one of claims 1-8.

12. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the target detection method according to any one of claims 1-8.

Images