Target detection model training method and electronic device

Abstract

The application provides a target detection model training method and an electronic device, wherein the method comprises: acquiring a training image, and performing event synthesis on the training image to generate a training data pair; inputting the training data pair into a feature extraction module to extract a training image feature and a training event feature; inputting the training image feature and the training event feature into a symmetric fusion module to obtain a fused feature, and inputting the fused feature into a detection head to obtain a class probability, a bounding box coordinate and a confidence of each to-be-detected object; and iteratively adjusting an initial target detection model according to the class probability, the bounding box coordinate and the confidence to obtain the target detection model. The application realizes efficient target detection based on mode importance balance under all-weather lighting conditions, so that the target detection model can maintain stable high-precision detection capability under various lighting conditions such as normal, underexposure and overexposure.

Description

Technical Field

[0001] This application relates to the field of object detection technology, and more specifically, to an object detection model training method and an electronic device. Background Technology

[0002] Object detection is one of the core tasks in the field of computer vision. It aims to locate and identify target objects of a specific category from digital images or videos, and has extremely important value in practical application scenarios such as autonomous driving, robot navigation, and intelligent monitoring.

[0003] Existing technologies can enhance detection robustness by fusing event camera data with red-green-blue (RGB) images in a multimodal manner. Specifically, based on the different fusion stages, it can be divided into two paradigms: intermediate fusion and post-fusion. Intermediate fusion refers to cross-modal feature interaction in the intermediate stage of feature extraction, while post-fusion refers to fusion at the result level after each modality has independently completed feature extraction.

[0004] However, since intermediate and post-fusion methods generally adopt asymmetric network structures, that is, they are dominated by RGB image features and the event camera data is only used as auxiliary or supplementary information, the improvement in detection performance is limited in normal scenes, while the detection performance drops significantly under extreme exposure conditions. Summary of the Invention

[0005] The purpose of this application is to address the shortcomings of the prior art by providing a target detection model training method and electronic device, so as to solve the problem that the existing technology has limited performance improvement in normal scenarios and significant performance degradation under extreme exposure conditions.

[0006] To achieve the above objectives, the technical solutions adopted in the embodiments of this application are as follows: In a first aspect, one embodiment of this application provides a method for training an object detection model, the method comprising: Acquire training images and perform event synthesis on the training images to generate training data pairs. Each training data pair includes an image data and event data corresponding to the image data. The training image includes at least one object to be detected. The training data is input into the feature extraction module of the initial target detection model to extract training image features and training event features at multiple scales. The training image features and training event features at multiple scales are input into the symmetric fusion module of the initial object detection model to obtain the fused features at multiple scales. The fused features at multiple scales are then input into the detection head of the initial object detection model to obtain the class probability, bounding box coordinates, and confidence of each object to be detected. Based on the category probability, bounding box coordinates, and confidence level of each object to be detected, the initial object detection model is iteratively adjusted to obtain the object detection model.

[0007] Secondly, another embodiment of this application provides a target detection device, the device comprising: The acquisition module is used to acquire training images and perform event synthesis on the training images to generate training data pairs. The training data pairs include an image data and event data corresponding to the image data. The training images include at least one object to be detected. The multi-scale feature extraction module is used to extract training image features and training event features at multiple scales from the training data input to the feature extraction module of the initial target detection model. The multi-scale fusion module is used to input the training image features and training event features at the multiple scales into the symmetric fusion module of the initial object detection model to obtain the fused features at the multiple scales, and input the fused features at the multiple scales into the detection head of the initial object detection model to obtain the class probability, bounding box coordinates and confidence of each object to be detected. The iterative adjustment module is used to iteratively adjust the initial target detection model based on the category probability, bounding box coordinates, and confidence level of each target object to obtain the target detection model.

[0008] Thirdly, another embodiment of this application provides an electronic device, including: a processor, a storage medium, and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform the steps of any of the methods described in the first aspect above.

[0009] Fourthly, another embodiment of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, performs the steps of any of the methods described in the first aspect above.

[0010] The beneficial effects of this application are as follows: By synthesizing events from training images to generate training data pairs, corresponding event frames can be directly generated from a single exposed image, solving the problem of scarce image-event pairing data. The training data pairs are then input into the feature extraction module of the initial object detection model to extract training image features and training event features at multiple scales. These features are then input into the symmetric fusion module of the initial object detection model to obtain fused features at multiple scales. This establishes an equitable feature interaction mechanism between image and event modalities, avoiding the modal imbalance problem of RGB-dominated event-assisted fusion architecture. Under extreme exposure conditions, the high dynamic range advantage of the event camera can be fully utilized to improve modal fusion balance. Furthermore, it achieves accurate cross-modal semantic alignment, effectively eliminating representational differences and interference noise between modalities.

[0011] Based on this, the fused features from multiple scales are input into the detection head of the initial target detection model to obtain the class probability, bounding box coordinates, and confidence score of each target object. Then, the initial target detection model is iteratively adjusted according to the class probability, bounding box coordinates, and confidence score of each target object to obtain the target detection model. This allows the target detection model to achieve balanced adaptive multimodal fusion without relying on a single modality, thereby achieving efficient target detection based on modal importance balance under all-weather lighting conditions. This enables the target detection model to maintain stable high-precision detection capability under various lighting conditions such as normal, underexposed, and overexposed conditions. Attached Figure Description

[0012] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 A schematic flowchart of a target detection model training method provided in an embodiment of this application; Figure 2 A schematic diagram of the structure of the initial target detection model provided in the embodiments of this application; Figure 3 This is a schematic flowchart illustrating the generation of training data pairs in the target detection model training method provided in this application embodiment. Figure 4 This is a flowchart illustrating the process of generating event data corresponding to each optional image in the target detection model training method provided in this application embodiment; Figure 5A schematic diagram of the feature extraction module in the initial target detection model provided in this application embodiment; Figure 6 This is a flowchart illustrating the process of generating event data corresponding to each optional image in the target detection model training method provided in this application embodiment; Figure 7 Another structural diagram of the feature extraction module in the initial target detection model provided in the embodiments of this application; Figure 8 This is a flowchart illustrating the process of generating event data corresponding to each optional image in the target detection model training method provided in this application embodiment; Figure 9 A schematic diagram of the event feature enhancement module in the initial target detection model provided in this application embodiment; Figure 10 This is a schematic diagram of the symmetric fusion module in the initial target detection model provided in the embodiments of this application; Figure 11 This is a flowchart illustrating the process of obtaining fused features at the current scale in the target detection model training method provided in this application embodiment. Figure 12 A schematic diagram of the spatial channel attention enhancement module in the initial target detection model provided in this application embodiment; Figure 13 This is a flowchart illustrating the process of obtaining attention-enhanced event features at the current scale in the target detection model training method provided in this application embodiment. Figure 14 This is a schematic diagram of the fusion processing module in the initial target detection model provided in the embodiments of this application; Figure 15 This is a schematic diagram of a process for obtaining spatially enhanced event features at the current scale in the target detection model training method provided in the embodiments of this application. Figure 16 This is a schematic diagram of the electronic device structure provided in an embodiment of this application. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. It should be understood that the accompanying drawings in this application are for illustrative and descriptive purposes only and are not intended to limit the scope of protection of this application. Furthermore, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate operations implemented according to some embodiments of this application. It should be understood that the operations in the flowcharts may not be implemented in sequence, and steps without logical contextual relationships may be reversed or implemented simultaneously. In addition, those skilled in the art, guided by the content of this application, may add one or more other operations to the flowcharts, or remove one or more operations from the flowcharts.

[0015] Furthermore, the described embodiments are merely some, not all, of the embodiments of this application. The components of the embodiments of this application described and illustrated herein can typically be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0016] It should be noted that the term "comprising" will be used in the embodiments of this application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0017] Existing technologies can enhance detection robustness by fusing event camera data with red-green-blue (RGB) images in a multimodal manner. Specifically, based on the different fusion stages, it can be divided into two paradigms: intermediate fusion and post-fusion. Intermediate fusion refers to cross-modal feature interaction in the intermediate stage of feature extraction, while post-fusion refers to fusion at the result level after each modality has independently completed feature extraction.

[0018] However, since intermediate and post-fusion methods generally adopt asymmetric network structures, that is, they are dominated by RGB image features and the event camera data is only used as auxiliary or supplementary information, the improvement in detection performance is limited in normal scenes, while the detection performance drops significantly under extreme exposure conditions.

[0019] Based on the aforementioned problems, this application proposes a target detection model training method. This method generates training data pairs by synthesizing events from training images. These training data pairs are then input into the feature extraction module of the initial target detection model to extract training image features and training event features at multiple scales. These features are then input into the symmetric fusion module of the initial target detection model to obtain fused features at multiple scales. Finally, these fused features are input into the detection head of the initial target detection model to obtain the class probability, bounding box coordinates, and confidence score of each target object. Based on these data, the initial target detection model is iteratively adjusted to obtain a target detection model. This model enables balanced adaptive multimodal fusion without relying on a single modality, achieving efficient target detection based on modal importance balance under all-weather lighting conditions. This ensures the target detection system maintains stable high-precision detection capabilities under various lighting conditions, including normal, underexposed, and overexposed conditions.

[0020] It is understood that the target detection model training method provided in this application embodiment can be applied to any electronic device with processing capabilities to achieve target detection.

[0021] In this application, the target refers to all objects in the RGB image that belong to the training category. In other words, in practical applications, the training category of each object in the RGB image can be configured in advance. For example, people, cars, bicycles, dogs and other objects appearing in the RGB image can all be configured to belong to the training category. Then, by executing the target detection model training method provided in the embodiment of this application, the detection of all people, cars, bicycles, dogs and other objects in the RGB image can be achieved.

[0022] The target detection model training method provided in this application will be described in detail below with reference to several embodiments.

[0023] Figure 1 This is a flowchart illustrating a target detection model training method provided in an embodiment of this application, with reference to... Figure 1 As shown, the subject executing this method can be any electronic device with processing capabilities, and the method includes: S101. Acquire training images and perform event synthesis on the training images to generate training data pairs.

[0024] It is understood that this application embodiment uses a single training image as an example. In practical applications, training images can be acquired in batches, and different batches of training data pairs can be generated.

[0025] Optionally, training images are acquired, wherein the training images include at least one object to be detected. The object to be detected may be, for example, a person, a car, a bicycle, a dog, etc.

[0026] Optionally, after obtaining the training image, event synthesis based on random optical flow is performed on the training image to obtain event data corresponding to the training image, and training data pairs are obtained based on the training image and the event data corresponding to the training image.

[0027] In one example, a velocity field can be randomly generated in the training image, and the gradient of the training image can be calculated. Based on the velocity field and gradient of the training image, event data corresponding to the training image can be generated. The training image is used as image data, and the event data corresponding to the training image is used as event data corresponding to the image data to obtain training data pairs.

[0028] The training data pair includes an image data set and the corresponding event data.

[0029] S102. Input the training data into the feature extraction module of the initial target detection model to extract training image features and training event features at multiple scales.

[0030] Optionally, Figure 2 This is a schematic diagram of the structure of the initial target detection model provided in the embodiments of this application, with reference to... Figure 2 As shown, the training data is input into the feature extraction module of the initial target detection model. The feature extraction module performs multi-scale feature extraction on the image data and event data in the training data pair, and extracts training image features and training event features at multiple scales.

[0031] For example, the feature extraction module may include an image feature extraction module and an event feature extraction module. The feature extraction module inputs the training data pairs into the initial target detection model. The image feature extraction module performs multi-scale feature extraction on the image data in the training data pairs to obtain training image features at multiple scales. The event feature extraction module performs multi-scale feature extraction on the time data in the training data pairs to obtain training event features at multiple scales.

[0032] Both the image feature extraction module and the event feature extraction module can be implemented based on CNN networks or Transformer structures.

[0033] S103. Input the training image features and training event features at multiple scales into the symmetric fusion module of the initial object detection model to obtain the fused features at multiple scales. Then, input the fused features at multiple scales into the detection head of the initial object detection model to obtain the class probability, bounding box coordinates and confidence of each object to be detected.

[0034] Optionally, training image features at multiple scales and training event features at multiple scales are input into the symmetric fusion module of the initial target detection model. The symmetric fusion module performs cross-modal symmetric fusion based on the training image features at multiple scales and the training event features at multiple scales to obtain fused features at multiple scales.

[0035] Cross-modal symmetric fusion refers to training image features by training event features during fusion, and training event features by training image features.

[0036] By adding two feature modulation paths with the same structure, opposite direction, and equal status during cross-modal fusion, the network can automatically determine which path dominates the fusion result based on the input quality. This ensures that the rich semantics of the training image features are not lost under normal lighting conditions, and the clear edges of the training event features are not lost under extreme lighting conditions, thus achieving truly all-weather robust detection.

[0037] Optionally, the fused features from multiple scales can be input into the detection head of the initial object detection model to obtain the class probability, bounding box coordinates, and confidence score of each object to be detected.

[0038] For example, the detection head can be a YOLOv3 detection head.

[0039] In one example, the fused features from multiple scales can be input into the cross-scale connection module of the initial object detection model. The cross-scale connection module performs cross-scale connections on the fused features from multiple scales to obtain enhanced fused features from multiple scales. The strong semantic information of the high-level model is passed to the low-level model through upsampling, and the detailed positional information of the low-level model is passed to the high-level model through downsampling. This makes the feature map of each scale have rich semantics and accurate position, thereby improving the detection accuracy.

[0040] Based on this, the enhanced and fused features at multiple scales are input into the detection head, which then predicts the enhanced and fused features at multiple scales to obtain the class probability, bounding box coordinates, and confidence of each object to be detected.

[0041] S104. Based on the category probability, bounding box coordinates, and confidence level of each object to be detected, the initial object detection model is iteratively adjusted to obtain the object detection model.

[0042] Optionally, based on the class probability, bounding box coordinates, and confidence score of each object to be detected, the classification loss, localization loss, and confidence score loss are calculated. The network parameters of the initial object detection model are then updated through backpropagation and iterated. After the iteration is completed, the object detection model is obtained.

[0043] It is understandable that after training, object detection can be performed based on the object detection model after the iteration. Specifically, the object detection model is deployed in an electronic device, which can be, for example, a control device inside or outside the robot.

[0044] For example, when performing target detection based on the target detection model after the iteration is completed, that is, when performing inference through the trained target detection model, RGB images can be acquired through an RGB camera and event streams can be acquired through an event camera. The event streams can be processed through an encoding and copying module to construct an event sequence. Target detection can then be performed based on the RGB images, the event sequence, the image feature extraction module, the event feature extraction module, the symmetric fusion module, and the detection head in the target detection model to obtain the target detection result.

[0045] In this embodiment, by performing event synthesis on training images to generate training data pairs, corresponding event frames can be directly generated from a single exposed image, solving the problem of scarce image-event pairing data. The training data pairs are then input into the feature extraction module of the initial object detection model to extract training image features and training event features at multiple scales. These features are then input into the symmetric fusion module of the initial object detection model to obtain fused features at multiple scales. This establishes an equitable feature interaction mechanism between the image and event modalities, avoiding the modal imbalance problem of RGB-dominated event-assisted fusion architecture. Under extreme exposure conditions, the high dynamic range advantage of the event camera can be fully utilized to improve the modal fusion balance. Furthermore, accurate cross-modal semantic alignment can be achieved, effectively eliminating representational differences and interference noise between modalities.

[0046] Based on this, the fused features from multiple scales are input into the detection head of the initial target detection model to obtain the class probability, bounding box coordinates, and confidence score of each target object. Then, the initial target detection model is iteratively adjusted according to the class probability, bounding box coordinates, and confidence score of each target object to obtain the target detection model. This allows the target detection model to achieve balanced adaptive multimodal fusion without relying on a single modality, thereby achieving efficient target detection based on modal importance balance under all-weather lighting conditions. This enables the target detection model to maintain stable high-precision detection capability under various lighting conditions such as normal, underexposed, and overexposed conditions.

[0047] In one possible implementation, Figure 3 This is a flowchart illustrating the generation of training data pairs in the target detection model training method provided in this application embodiment, with reference to... Figure 3 As shown, in S101 above, event synthesis is performed on the training images to generate training data pairs, including: S301. Based on the preset exposure factor group, perform brightness transformation on the training image to obtain selectable images under multiple exposure conditions.

[0048] Optionally, the training image is converted to the Hue Saturation Value (HSV) color space, and the converted training image is multiplied by each exposure factor in the preset exposure factor group to perform brightness transformation on the training image, thereby obtaining selectable images under multiple exposure conditions.

[0049] The exposure factor group includes exposure factors under multiple exposure conditions, such as exposure factors under underexposure, exposure factors under normal exposure, and exposure factors under overexposure. Specifically, the exposure factor under underexposure can be 0.2, the exposure factor under normal exposure can be 1.0, and the exposure factor under overexposure can be 5.0.

[0050] For example, multiplying the converted training image by the exposure factor under underexposure conditions yields an optional image under underexposure conditions. Multiplying the converted training image by the exposure factor under normal exposure conditions yields an optional image under normal exposure conditions. Multiplying the converted training image by the exposure factor under overexposure conditions yields an optional image under overexposure conditions.

[0051] S302. Based on the optional images under multiple exposure conditions, generate event data corresponding to each optional image, and treat each optional image and its corresponding event data as an optional training data pair.

[0052] Optionally, after obtaining multiple optional images under various exposure conditions, event synthesis based on random optical flow can be performed on each optional image under each exposure condition to generate event data corresponding to each optional image, and each optional image and the event data corresponding to each optional image can be used as an optional training data pair.

[0053] S303. Randomly select from each optional training data pair to obtain training data pairs.

[0054] Optionally, training data pairs can be obtained by randomly selecting from each of the optional training data pairs.

[0055] For example, training data pairs can be obtained by uniformly random sampling from each of the optional training data pairs.

[0056] By using a preset exposure factor group, the brightness of the training image is transformed to obtain multiple optional images under different exposure conditions, and each optional training data pair is obtained. Then, each optional training data pair is randomly selected to obtain training data pairs. This allows the target detection model to dynamically adjust its balance parameters based on the actual brightness of the input RGB image during training, thereby improving the accuracy and stability of the target detection model under all-weather lighting conditions.

[0057] In one possible implementation, Figure 4 This is a flowchart illustrating the process of generating event data corresponding to each selectable image in the target detection model training method provided in this application embodiment, with reference to... Figure 4 As shown, in step S302 above, event data corresponding to each optional image is generated based on multiple optional images under various exposure conditions, including: S401. Calculate the gradient matrix of the selectable image under the current exposure conditions according to the preset gradient operator.

[0058] Optionally, taking an optional image under a certain exposure condition as an example, the gradient matrix of the optional image under the current exposure condition is obtained by calculating the gradients in the horizontal and vertical directions of the optional image under the current exposure condition using a preset gradient operator.

[0059] The gradient matrix is ​​a vector field comprising a horizontal two-dimensional vector and a vertical two-dimensional vector for each pixel. The horizontal two-dimensional vector represents the rate of change of brightness of the pixel in the horizontal direction, and the vertical two-dimensional vector represents the rate of change of brightness of the pixel in the vertical direction.

[0060] For example, the preset gradient operator can be the Sobel operator, which can obtain the gradient matrix of the selectable image under the current exposure conditions by sliding convolution on two 3*3 convolution kernels.

[0061] S402. Based on each pixel of the selectable image under the current exposure conditions, generate a unit velocity vector corresponding to each pixel.

[0062] Optionally, for each pixel of the selectable image under the current exposure conditions, a motion direction angle is randomly generated to generate a unit velocity vector corresponding to each pixel.

[0063] For example, based on the assumption of brightness constancy, the brightness change over a short time interval can be approximated using Taylor expansion to satisfy the following equation:

[0064] Meanwhile, the brightness constancy condition can be obtained as follows:

[0065] Substituting the constant brightness condition into the approximate brightness change, we obtain the following formula:

[0066] in, =( / , / () represents the brightness gradient. ( ) is the location The optical flow velocity vector at that location.

[0067] Based on this, for each pixel position ( , Randomly generate a motion direction angle for each. The unit velocity vectors corresponding to each pixel are generated as follows:

[0068] Among them, the direction angle of motion Random sampling from a uniform distribution.

[0069] S403. Calculate the dot product of the gradient matrix and the unit velocity vector corresponding to each pixel to obtain the brightness change information of each pixel in the selectable image under the current exposure conditions.

[0070] Optionally, the gradient matrix and the unit velocity vector corresponding to each pixel are multiplied pixel by pixel to obtain the brightness change information of each pixel in the optional image under the current exposure conditions.

[0071] S404. Based on the brightness change information of each pixel and the preset brightness change threshold, generate corresponding events for each pixel to obtain the events corresponding to each pixel.

[0072] Optionally, each pixel is traversed. For the current pixel, the brightness change information of the current pixel is compared with a preset brightness change threshold. If the brightness change information of the current pixel is greater than the preset brightness change threshold, a brightness increase event is generated at the current pixel. If the brightness change information of the current pixel is less than or equal to the preset brightness change threshold, a brightness decrease event is generated at the current pixel. After the traversal is completed, the events corresponding to each pixel are obtained.

[0073] S405. Spatial accumulation is performed based on the events corresponding to each pixel to obtain the event data corresponding to the selectable image under the current exposure conditions.

[0074] Optionally, the events corresponding to all pixels are spatially accumulated and transformed into a two-dimensional event frame, which serves as the event data for the optional image under the current exposure conditions.

[0075] The gradient matrix is ​​calculated using a preset gradient operator. By generating unit velocity vectors corresponding to each pixel, the dot product of the gradient matrix and the unit velocity vectors corresponding to each pixel is calculated to obtain the brightness change information of each pixel. Based on the brightness change information of each pixel and a preset brightness change threshold, the event corresponding to each pixel is obtained. After spatial accumulation, the event data corresponding to the selectable image under the current exposure conditions is obtained. It can deduce a virtual event camera output from a static image through hypothetical random motion, realizing the direct generation of high-quality event data paired with the image from a single image. This solves the bottleneck problem of lack of training and evaluation data in all weather conditions, and enables the object detection model to maintain robustness when facing various motion directions.

[0076] In one possible implementation, Figure 5 This is a schematic diagram of a feature extraction module in the initial target detection model provided in an embodiment of this application. Figure 6 This is a flowchart illustrating the process of generating event data corresponding to each selectable image in the target detection model training method provided in this application embodiment, with reference to... Figure 5 as well as Figure 6 As shown, the feature extraction module includes: an image feature extraction module, an encoding and copying module, and an event feature extraction module. In S102 above, the feature extraction module, which inputs training data into the initial target detection model, extracts training image features at multiple scales and training event features at multiple scales, including: S601. Input the image data into the image feature extraction module to extract training image features at multiple scales.

[0077] Optionally, the image data is input into the image feature extraction module, which performs multi-scale feature extraction on the image data to obtain training image features at multiple scales.

[0078] The image feature extraction module can be implemented based on a CNN network, such as the CSPDarknet-Tiny network.

[0079] For example, training image features at three scales can be extracted. Specifically, training image features at multiple scales can be RGB feature maps at three different scales.

[0080] S602. Input the event data into the encoding and copying module to obtain the event sequence corresponding to the event data.

[0081] Optionally, the event data is input into the encoding and copying module, which copies the event data according to the time step length of the event feature extraction module to obtain the event sequence corresponding to the event data. Specifically, the copying can be implemented using constant encoding.

[0082] S603. Input the event sequence into the event feature extraction module to extract training event features at multiple scales.

[0083] Optionally, the event sequence is input into the event feature extraction module, which performs multi-scale feature extraction to obtain training event features at multiple scales.

[0084] The event feature extraction module can be a spiking ResNet-18 network. In the spiking ResNet-18 network, LIF neurons accumulate membrane potential based on input pulses, and fire a pulse (0 or 1) when a threshold is reached to reset the potential.

[0085] Among them, the training event features are pulse sequences that include a time dimension.

[0086] For example, training event features at three scales can be extracted. Specifically, the training event features at multiple scales can be feature maps in the form of pulses at three different scales.

[0087] By replicating the event data, an event sequence is obtained. The event feature extraction module then performs multi-scale feature extraction, extracting training event features at multiple scales. This allows the event feature extraction module to process temporal information, enabling matching of event feature extraction with image feature extraction and achieving accurate cross-modal semantic alignment. Furthermore, using a spiking neural network as the event feature extraction module achieves over 90% energy reduction compared to traditional artificial neural networks, while maintaining detection accuracy.

[0088] The above provides an exemplary description of obtaining training image features and training event features at multiple scales. The following provides an exemplary description of the process of inputting training image features and training event features at multiple scales into the symmetric fusion module of the initial object detection model to obtain fused features at multiple scales.

[0089] In one possible implementation, Figure 7 This is another schematic diagram of the feature extraction module in the initial target detection model provided in the embodiments of this application. Figure 8This is a flowchart illustrating the process of generating event data corresponding to each selectable image in the target detection model training method provided in this application embodiment, with reference to... Figure 7 as well as Figure 8 As shown, the initial object detection model also includes: an event feature enhancement module, which is connected to the feature extraction module and the symmetric fusion module. In S103 above, training image features at multiple scales and training event features at multiple scales are input into the symmetric fusion module of the initial object detection model to obtain fused features at multiple scales, including: S801. Input the training event features at the current scale into the event feature enhancement module to obtain the enhanced event features at the current scale.

[0090] Optionally, the training event features at the current scale are input into the event feature enhancement module, which then performs dediscretization processing on the training event features at the current scale and extracts key information in the time dimension to obtain the enhanced event features at the current scale.

[0091] Among them, the enhanced event features at the current scale are used to perform temporal and spatial enhancement on the training features at the current scale.

[0092] The event feature enhancement module processes the training event features at the current scale into enhanced event features at the current scale, which can soften discrete pulse signals into continuous and statistically significant intensity values.

[0093] S802. Input the enhanced event features, training event features, and training image features at the current scale into the symmetric fusion module. The symmetric fusion module determines the event weights of the training image features during fusion based on the enhanced event features, and determines the image weights of the training event features during fusion based on the training image features. Based on the training image features, event weights, training event features, and image weights, the fused features at the current scale are obtained.

[0094] Optionally, the enhanced event features at the current scale, the training event features at the current scale, and the training image features at the current scale are input into the symmetric fusion module. The symmetric fusion module determines the event weights of the training image features during fusion based on the enhanced event features, determines the image weights of the training event features during fusion based on the training image features, and obtains the fused features at the current scale based on the training image features, event weights, training event features, and image weights.

[0095] For example, the symmetric fusion module can calculate and determine the event weights of the training image features during fusion based on the enhanced event features, and calculate and determine the image weights of the training event features during fusion based on the training image features.

[0096] Building upon this foundation, event weights are applied to weight the training image features, enabling event features to assist in training the image features. If an event has a strong edge at a certain pixel, the image features are allowed to pass; if the event has no response at a certain pixel, the image features are suppressed. Image weights are also applied to weight the training event features, enabling image features to assist in training the event features. If the image recognizes this as part of the object to be identified, the event edge is preserved; if the image considers it as background noise, the event response is suppressed. The weighted results are then fused to obtain the fused features at the current scale, achieving adaptive and equitable modality fusion based on input quality.

[0097] Among them, the event weight is the confidence weight of the edge in the image, and the image weight is the confidence weight of the semantics in the image.

[0098] In one possible implementation, Figure 9 This is a schematic diagram of the event feature enhancement module in the initial target detection model provided in this application embodiment, with reference to... Figure 9 As shown, the event feature enhancement module includes: a max pooling layer, an average pooling layer, a concatenation layer, a first convolutional layer, a first batch of normalization layers, and a first activation layer. Step S801 above inputs the trained event features at the current scale into the event feature enhancement module to obtain enhanced event features at the current scale, including: The training event features at the current scale are input into a max pooling layer for max pooling along the time dimension to obtain max pooling features; the training event features at the current scale are input into an average pooling layer for average pooling along the time dimension to obtain average pooling features; the max pooling features and average pooling features are input into a concatenation layer for concatenation along the channel dimension to obtain concatenated features; based on the first convolutional layer, the first batch normalization layer, and the first activation layer, the concatenated features are sequentially processed by convolution, batch normalization, and activation to obtain enhanced event features at the current scale.

[0099] Optionally, refer to Figure 9 As shown, at the current scale For example, the current scale Training event features The input is max-pooled, and max-pooling is performed along the time dimension. The maximum value along the time dimension T is taken to obtain the max-pooled feature. and the current scale Training event features The input is averaged through a time-dimension average pooling layer, and the average value is taken along the time dimension T to obtain the average pooling feature. and max pooling features and average pooling characteristics The input concatenation layer performs channel-dimensional concatenation to obtain concatenated features, which are then processed by the first convolutional layer. The first batch of normalized layers and the first activation layer The concatenated features are then subjected to convolution, batch normalization, and activation processing in sequence to obtain the current scale. Enhanced event features Specifically, it can be implemented using the following formula:

[0100] in, . For the number of channels, For height, For width.

[0101] By introducing max pooling to capture the strongest edge signals and average pooling to capture the density of activity, the network can learn to distinguish which signals are important motion edges and which are background noise, thereby extracting cleaner event enhancement features.

[0102] In one possible implementation, Figure 10 This is a schematic diagram of a symmetric fusion module in the initial target detection model provided in an embodiment of this application. Figure 11 This is a flowchart illustrating the process of obtaining fused features at the current scale in the target detection model training method provided in this application embodiment, with reference to... Figure 10 as well as Figure 11 As shown, the symmetric fusion module includes: a spatial channel attention enhancement module and a fusion processing module; in S802 above, the enhanced event features at the current scale, the training event features at the current scale, and the training image features at the current scale are input into the symmetric fusion module. The symmetric fusion module determines the event weights of the training image features during fusion based on the enhanced event features, and determines the image weights of the training event features during fusion based on the training image features. Based on the training image features, event weights, training event features, and image weights, the fused features at the current scale are obtained, including: S1101. Input the enhanced event features at the current scale into the spatial channel attention enhancement module to obtain the attention enhancement event features at the current scale.

[0103] Optionally, the enhanced event features at the current scale can be input into the spatial channel attention enhancement module, which will then perform dual processing on the enhanced event features at the current scale in both spatial and channel dimensions to obtain attention-enhanced event features at the current scale, thereby making the edge information in the event more prominent.

[0104] S1102. Input the training image features at the current scale into the spatial channel attention enhancement module to obtain the attention-enhanced image features at the current scale.

[0105] Optionally, the training image features at the current scale are input into the spatial channel attention enhancement module, which performs dual processing on the training image features at the current scale in both spatial and channel dimensions to obtain attention-enhanced image features at the current scale, thereby making the semantic information in the image more prominent.

[0106] For example, the training image features at the current scale are input into the spatial attention enhancement module to obtain the spatially enhanced image features at the current scale, which enables the network to focus on the region with the richest color and texture and the most prominent objects in the image. The spatially enhanced image features at the current scale and the training image features at the current scale are input into the channel attention enhancement module to obtain the attention-enhanced image features at the current scale, thereby selecting the image feature channels with the most discriminative power for classification and localization.

[0107] It is understood that, in one example, the symmetric fusion module may include two identical spatial channel attention enhancement modules, which can simultaneously input training image features and training event features at the current scale into the spatial channel attention enhancement module. In another example, the symmetric fusion module may include one spatial channel attention module, which can separately input training image features and training event features at the current scale into the spatial channel attention enhancement module for processing.

[0108] S1103. Input the attention-enhanced event features and attention-enhanced image features at the current scale into the fusion processing module. The fusion processing module determines the event weights of the training image features during fusion based on the attention-enhanced event features, and determines the image weights of the training event features during fusion based on the attention-enhanced image features. Based on the training image features, event weights, training event features, and image weights, the fused features at the current scale are obtained.

[0109] For example, the fusion processing module can use attention-enhanced event features as event weights in the fusion of training image features, and use attention-enhanced image features as image weights in the fusion of training event features. The training image features are weighted by the event weights, and the training event features are weighted by the image weights, so as to realize the assistance of image features to training event features. The weighted results are summed to obtain the fused features at the current scale, thus realizing adaptive and equal modality fusion based on input quality.

[0110] After the spatial channel attention enhancement module performs dual spatial and channel enhancement on the training image features and enhanced event features at the current scale, the fusion processing module then fuses them to obtain the fused features at the current scale. This allows for the extraction of the most reliable information from both the event modality and the image modality, thereby achieving bidirectional and equivalent feature modulation. This results in more accurate and robust fused features.

[0111] The following example illustrates the processing of the spatial channel attention enhancement module by taking the acquisition of attention enhancement event features at the current scale as an example. The process of acquiring attention enhancement image features at the current scale can be implemented by referring to the process of acquiring attention enhancement event features at the current scale.

[0112] In one possible implementation, Figure 12 This is a schematic diagram of a spatial channel attention enhancement module in the initial target detection model provided in this application embodiment. Figure 13 This is a flowchart illustrating the process of obtaining attention-enhanced event features at the current scale in the object detection model training method provided in this application embodiment, with reference to... Figure 12 as well as Figure 13 As shown, the spatial channel attention enhancement module includes: a spatial attention enhancement module and a channel attention enhancement module; in S1101 above, the enhanced event features at the current scale are input into the spatial channel attention enhancement module to obtain the attention enhancement event features at the current scale, including: S1301. Input the enhanced event features at the current scale into the spatial attention enhancement module to obtain the spatial enhanced event features at the current scale.

[0113] Optionally, the enhanced event features at the current scale are input into the spatial attention enhancement module, which calculates the importance score of each spatial location in the enhanced event features and weights the importance scores of each spatial location into the corresponding features to obtain the spatial enhanced event features at the current scale. This allows the network to focus on the area that the event camera sees most clearly and suppresses flat areas without event signals.

[0114] S1302. Input the spatial augmentation event features and the augmentation event features at the current scale into the channel attention enhancement module to obtain the attention enhancement event features at the current scale.

[0115] Optionally, the spatial augmented event features and the augmented event features at the current scale are input into the channel attention enhancement module. The channel attention enhancement module performs global compression on the spatially augmented features and then multiplies them back into the augmented event features channel by channel to obtain the attention-enhanced event features at the current scale. This allows the most useful semantic channels for the current detection task to be selected and irrelevant channels to be suppressed.

[0116] In one possible implementation, continue to refer to Figure 12 As shown, the spatial attention enhancement module includes: a second convolutional layer; the enhanced event features at the current scale in S1301 are input into the spatial attention enhancement module to obtain the spatial enhanced event features at the current scale, including: The enhanced event features at the current scale are subjected to a first tensor transformation to obtain the features after the first tensor transformation; the enhanced event features at the current scale are input into a second convolutional layer for convolution processing to obtain the event features after convolution; the event features after convolution are activated according to a preset first activation function to obtain the features after activation; the features after activation are subjected to a second tensor transformation to obtain the features after the second tensor transformation; the matrix product of the features after the first tensor transformation and the features after the second tensor transformation is calculated to obtain the spatial enhanced event features at the current scale.

[0117] Optionally, continue to refer to Figure 12 As shown, at the current scale For example, the current scale Enhanced event features Perform the first tensor transformation to obtain the features after the first tensor transformation. and the current scale Enhanced event features Input to the second convolutional layer Perform convolution processing to obtain the event features after convolution. According to the preset first activation function The event features after convolution are activated to obtain the activated features. Perform a second tensor transformation on the activated features to obtain the features after the second tensor transformation. Calculate the matrix product of the features after the first tensor transformation and the features after the second tensor transformation to obtain the current scale. Spatial augmentation event characteristics For details, please refer to the following formula:

[0118] in, For the number of channels, For height, For width.

[0119] In one possible implementation, continue to refer to Figure 12 As shown, the channel attention enhancement module includes: a third convolutional layer, a fourth convolutional layer, and a second activation layer; in S1302 above, the spatial enhancement event features and the enhancement event features at the current scale are input into the channel attention enhancement module to obtain the attention enhancement event features at the current scale, including: The spatial augmentation event features at the current scale are subjected to a third tensor transformation to obtain the features after the third tensor transformation. The features after the third tensor transformation are then subjected to convolution and activation processing in sequence according to the third convolutional layer, the fourth convolutional layer, and the second activation layer to obtain the channel attention weights at the current scale. The dot product of the channel attention weights at the current scale and the augmentation event features at the current scale is calculated to obtain the attention-enhanced event features at the current scale.

[0120] Optionally, continue to refer to Figure 12 As shown, at the current scale For example, the current scale Spatial augmentation event characteristics Perform a third tensor transformation to obtain the features after the third tensor transformation. According to the third convolutional layer Fourth convolutional layer and the second activation layer The features after the third tensor transformation are sequentially processed by convolution and activation to obtain the channel attention weights at the current scale; the channel attention weights and the enhanced event features at the current scale are then calculated. The dot product yields the attention-enhanced event features at the current scale. Specifically, it can be implemented using the following formula:

[0121] in, For the number of channels, For height, For width.

[0122] In one possible implementation, Figure 14 This is a schematic diagram of a fusion processing module in the initial target detection model provided in an embodiment of this application. Figure 15 This is a flowchart illustrating the process of obtaining spatially augmented event features at the current scale in the target detection model training method provided in this application embodiment, with reference to... Figure 14 as well as Figure 15As shown, the fusion processing module includes: a third activation layer, a fourth activation layer, a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer. In step S1103, the attention-enhanced event features and attention-enhanced image features at the current scale are input into the fusion processing module. The fusion processing module determines the event weights of the training image features during fusion based on the attention-enhanced event features, and determines the image weights of the training event features during fusion based on the attention-enhanced image features. Based on the training image features, event weights, and image weights, the fused features at the current scale are obtained, including: S1501. Input the attention-enhanced event features at the current scale into the third activation layer for activation to obtain event weights. Calculate the product of the event weights and the training image features at the current scale to obtain the first product result. Calculate the sum of the first product result and the training image features at the current scale to obtain the event-enhanced image features. Input the event-enhanced image features into the fifth convolutional layer for convolution processing to obtain the first fusion feature.

[0123] Optionally, the training image features are weighted with attention-enhanced event features at the current scale. For example, the current scale Attention-enhancing event features Input third activation layer Activation is performed to obtain event weights, and the event weights are then compared with the current scale. Training image features The product of the first product and the current scale are calculated. Training image features The sum of these features yields the event-enhanced image features, which are then input into the fifth convolutional layer. Perform convolution processing to obtain the first fused feature. For details, please refer to the following formula:

[0124] S1502. Input the attention-enhanced image features at the current scale into the fourth activation layer for activation to obtain image weights. Calculate the product of the image weights and the training event features at the current scale to obtain the second product result. Calculate the sum of the second product result and the training event features at the current scale to obtain the image-enhanced event features. Input the image-enhanced event features into the sixth convolutional layer for convolution processing to obtain the second fusion feature.

[0125] Optionally, attention-enhanced image features are used to weight the training event features at the current scale. For example, the current scale Attention-enhanced image features Input the fourth activation layer Activation is performed to obtain image weights, and the image weights are then calculated relative to the current scale. Training event features The product of the first product and the second product is obtained, and the second product is then compared with the current scale. Training event features The sum of these features yields the event features after image enhancement, which are then input into the sixth convolutional layer. Perform convolution processing to obtain the second fused feature. For details, please refer to the following formula:

[0126] S1503. Perform channel splicing on the first fusion feature and the second fusion feature to obtain the spliced ​​fusion feature.

[0127] Optionally, for the first fusion feature and the second fusion feature Perform channel splicing to obtain the spliced ​​fused features. Specifically, please refer to the following formula:

[0128] S1504. Take the maximum value of the spliced ​​fused features in the channel dimension to obtain the maximum fused feature, and take the average value of the spliced ​​fused features in the channel dimension to obtain the average fused feature.

[0129] Optionally, the spliced ​​fusion features The maximum value is obtained by taking the maximum value in the channel dimension. The specific formula is as follows:

[0130] Optionally, the spliced ​​fusion features The maximum fusion feature is obtained by averaging the values ​​along the channel dimension. .

[0131] S1505. The maximum fusion feature and the average fusion feature are concatenated and then input into the seventh convolutional layer to obtain the fused features at the current scale.

[0132] Optionally, the maximum fusion feature and average fusion characteristics After splicing, input into the seventh convolutional layer To obtain the fused features at the current scale The specific formula is as follows:

[0133] By weighting training image features with attention-enhanced event features and vice versa, semantic information can be provided by image features and geometric information can be provided by event features. The sum of the two constitutes a complete perception, achieving true feature complementarity. This enables the network to adaptively adjust according to ambient lighting, thus achieving efficient target detection under various lighting conditions.

[0134] Based on the same inventive concept, this application also provides an object detection device corresponding to the object detection model training method. Since the principle of the device in this application is similar to the object detection model training method described above, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be described again.

[0135] The device includes: an acquisition module, a multi-scale feature extraction module, a multi-scale fusion module, and an iterative adjustment module; wherein... The acquisition module is used to acquire training images and perform event synthesis on the training images to generate training data pairs. Each training data pair includes an image data and event data corresponding to the image data. The training image includes at least one object to be detected. The multi-scale feature extraction module is used to extract training image features and training event features at multiple scales from the training data input to the feature extraction module of the initial target detection model. The multi-scale fusion module is used to input training image features and training event features at multiple scales into the symmetric fusion module of the initial object detection model to obtain fused features at multiple scales. The fused features at multiple scales are then input into the detection head of the initial object detection model to obtain the class probability, bounding box coordinates, and confidence of each object to be detected. The iterative adjustment module is used to iteratively adjust the initial target detection model based on the class probability, bounding box coordinates, and confidence score of each target object to obtain the target detection model.

[0136] Optionally, the acquisition module is specifically used for: Based on a preset exposure factor group, the brightness of the training image is transformed to obtain selectable images under multiple exposure conditions; Based on the available images under multiple exposure conditions, generate event data corresponding to each available image, and use each available image and its corresponding event data as a pair of available training data. Randomly select from each of the optional training data pairs to obtain training data pairs.

[0137] Optionally, the acquisition module is specifically used for: Based on the preset gradient operator, the gradient matrix of the selectable image under the current exposure conditions is calculated; Based on the pixels of the available images under the current exposure conditions, generate a unit velocity vector corresponding to each pixel; Calculate the dot product of the gradient matrix and the unit velocity vector corresponding to each pixel to obtain the brightness change information of each pixel in the selectable image under the current exposure conditions; Based on the brightness change information of each pixel and the preset brightness change threshold, a corresponding event is generated at each pixel to obtain the event corresponding to each pixel. By spatially accumulating the events corresponding to each pixel, the event data corresponding to the selectable image under the current exposure conditions is obtained.

[0138] Optionally, the feature extraction module includes: an image feature extraction module, an encoding / copying module, and an event feature extraction module; the multi-scale feature extraction module is specifically used for: Image data is input into the image feature extraction module to extract training image features at multiple scales; The event data is input into the encoding and copying module to obtain the event sequence corresponding to the event data; The event sequence is input into the event feature extraction module to extract training event features at multiple scales.

[0139] Optionally, the initial object detection model further includes: an event feature enhancement module, which is connected to the feature extraction module and the symmetric fusion module; and a multi-scale fusion module, specifically used for: Input the training event features at the current scale into the event feature enhancement module to obtain the enhanced event features at the current scale; The augmented event features, training event features, and training image features at the current scale are input into the symmetric fusion module. The symmetric fusion module determines the event weights of the training image features during fusion based on the augmented event features, and determines the image weights of the training event features during fusion based on the training image features. Finally, based on the training image features, event weights, training event features, and image weights, the fused features at the current scale are obtained.

[0140] Optionally, the event feature enhancement module includes: a max pooling layer, an average pooling layer, a concatenation layer, a first convolutional layer, a first batch of normalization layers, and a first activation layer; the multi-scale fusion module is specifically used for: The training event features at the current scale are input into the max pooling layer and max pooled along the time dimension to obtain max pooled features. The training event features at the current scale are input into the average pooling layer and average pooling is performed along the time dimension to obtain average pooling features. The max pooling features and average pooling features are input into the concatenation layer and concatenated along the channel dimension to obtain the concatenated features. Based on the first convolutional layer, the first batch of normalization layers, and the first activation layer, the concatenated features are sequentially processed by convolution, batch normalization, and activation to obtain the enhanced event features at the current scale.

[0141] Optionally, the symmetric fusion module includes: a spatial channel attention enhancement module and a fusion processing module; a multi-scale fusion module, specifically used for: The enhanced event features at the current scale are input into the spatial channel attention enhancement module to obtain the attention enhancement event features at the current scale. Input the training image features at the current scale into the spatial channel attention enhancement module to obtain the attention-enhanced image features at the current scale. The attention-enhanced event features and attention-enhanced image features at the current scale are input into the fusion processing module. The fusion processing module determines the event weights of the training image features during fusion based on the attention-enhanced event features, and determines the image weights of the training event features during fusion based on the attention-enhanced image features. Based on the training image features, event weights, training event features, and image weights, the fused features at the current scale are obtained.

[0142] Optionally, the spatial channel attention enhancement module includes: a spatial attention enhancement module and a channel attention enhancement module; a multi-scale fusion module, specifically used for: The enhanced event features at the current scale are input into the spatial attention enhancement module to obtain the spatial enhanced event features at the current scale. Input the spatial augmentation event features and the augmentation event features at the current scale into the channel attention augmentation module to obtain the attention augmentation event features at the current scale.

[0143] Optionally, the spatial attention enhancement module includes: a second convolutional layer; and a multi-scale fusion module, specifically used for: Perform the first tensor transformation on the enhanced event features at the current scale to obtain the features after the first tensor transformation; The enhanced event features at the current scale are input into the second convolutional layer for convolution processing to obtain the convolutional event features. Based on the preset first activation function, the convolutional event features are activated to obtain the activated features. The activated features are subjected to a second tensor transformation to obtain the features after the second tensor transformation. Calculate the matrix product of the features after the first tensor transformation and the features after the second tensor transformation to obtain the spatial augmentation event features at the current scale.

[0144] Optionally, the channel attention enhancement module includes: a third convolutional layer, a fourth convolutional layer, and a second activation layer; the multi-scale fusion module is specifically used for: Perform a third tensor transformation on the spatial augmentation event features at the current scale to obtain the features after the third tensor transformation; The features after the third tensor transformation are sequentially convolved and activated by the third convolutional layer, the fourth convolutional layer, and the second activation layer to obtain the channel attention weights at the current scale. Calculate the dot product of the channel attention weights at the current scale and the enhanced event features at the current scale to obtain the attention-enhanced event features at the current scale.

[0145] Optionally, the fusion processing module includes: a third activation layer, a fourth activation layer, a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer; the multi-scale fusion module is specifically used for: The attention-enhanced event features at the current scale are input into the third activation layer for activation to obtain event weights. The product of the event weights and the training image features at the current scale is calculated to obtain the first product result. The sum of the first product result and the training image features at the current scale is calculated to obtain the event-enhanced image features. The event-enhanced image features are then input into the fifth convolutional layer for convolution processing to obtain the first fusion feature. The attention-enhanced image features at the current scale are input into the fourth activation layer for activation to obtain image weights. The product of the image weights and the training event features at the current scale is calculated to obtain the second product result. The sum of the second product result and the training event features at the current scale is calculated to obtain the image-enhanced event features. The image-enhanced event features are then input into the sixth convolutional layer for convolution processing to obtain the second fusion feature. Channel splicing is performed on the first fusion feature and the second fusion feature to obtain the spliced ​​fusion feature; The maximum value of the spliced ​​fused features in the channel dimension is obtained to get the maximum fused feature, and the average value of the spliced ​​fused features in the channel dimension is obtained to get the average fused feature. The maximum fusion feature and the average fusion feature are concatenated and then input into the seventh convolutional layer to obtain the fused features at the current scale.

[0146] The processing flow of each module in the device and the interaction flow between each module can be referred to the relevant descriptions in the above method embodiments, and will not be detailed here.

[0147] This application also provides an electronic device, such as... Figure 16 As shown, Figure 16The schematic diagram of the electronic device structure provided in this application embodiment includes: a processor 1601 and a memory 1602, and optionally, a bus 1603. The memory 1602 stores machine-readable instructions executable by the processor 1601. When the electronic device is running, the processor 1601 and the memory 1602 communicate via the bus 1603, and the processor 1601 executes the machine-readable instructions to perform the steps of the above-described target detection model training method.

[0148] This application also provides a computer-readable storage medium storing a computer program, which, when run by a processor, executes the steps of the above-described object detection model training method.

[0149] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems and devices described above can be referred to the corresponding processes in the method embodiments, and will not be repeated here. In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods can be implemented in other ways. The device embodiments described above are merely illustrative. For example, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the displayed or discussed mutual coupling or direct coupling or communication connection can be through some communication interfaces; the indirect coupling or communication connection of devices or modules can be electrical, mechanical, or other forms.

[0150] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0151] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for training an object detection model, characterized in that, include: Acquire training images and perform event synthesis on the training images to generate training data pairs. Each training data pair includes an image data and event data corresponding to the image data. The training image includes at least one object to be detected. The training data is input into the feature extraction module of the initial target detection model to extract training image features and training event features at multiple scales. The training image features and training event features at multiple scales are input into the symmetric fusion module of the initial object detection model to obtain the fused features at multiple scales. The fused features at multiple scales are then input into the detection head of the initial object detection model to obtain the class probability, bounding box coordinates, and confidence of each object to be detected. Based on the category probability, bounding box coordinates, and confidence level of each object to be detected, the initial object detection model is iteratively adjusted to obtain the object detection model.

2. The target detection model training method according to claim 1, characterized in that, The step of synthesizing events from training images to generate training data pairs includes: Based on a preset exposure factor group, the brightness of the training image is transformed to obtain selectable images under multiple exposure conditions; Based on the optional images under multiple exposure conditions, event data corresponding to each optional image is generated, and each optional image and the event data corresponding to each optional image are respectively used as an optional training data pair; Randomly select from each of the optional training data pairs to obtain training data pairs.

3. The target detection model training method according to claim 2, characterized in that, The step of generating event data corresponding to each selectable image based on multiple exposure conditions includes: Based on the preset gradient operator, the gradient matrix of the selectable image under the current exposure conditions is calculated; Based on the pixels of the available images under the current exposure conditions, generate a unit velocity vector corresponding to each pixel; Calculate the dot product of the gradient matrix and the unit velocity vector corresponding to each pixel to obtain the brightness change information of each pixel in the selectable image under the current exposure conditions; Based on the brightness change information of each pixel and the preset brightness change threshold, a corresponding event is generated at each pixel to obtain the event corresponding to each pixel. By spatially accumulating the events corresponding to each pixel, the event data corresponding to the selectable image under the current exposure conditions is obtained.

4. The target detection model training method according to claim 1, characterized in that, The feature extraction module includes: an image feature extraction module, an encoding copying module, and an event feature extraction module; The feature extraction module that inputs the training data into the initial target detection model extracts training image features at multiple scales and training event features at multiple scales, including: The image data is input into the image feature extraction module to extract training image features at multiple scales; The event data is input into the encoding and copying module to obtain the event sequence corresponding to the event data; The event sequence is input into the event feature extraction module to extract training event features at multiple scales.

5. The target detection model training method according to claim 1, characterized in that, The target detection model further includes an event feature enhancement module, which is connected to the feature extraction module and the symmetric fusion module. The step of inputting the training image features and training event features at multiple scales into the symmetric fusion module of the initial object detection model to obtain the fused features at multiple scales includes: The training event features at the current scale are input into the event feature enhancement module to obtain the enhanced event features at the current scale; The enhanced event features, training event features, and training image features at the current scale are input into the symmetric fusion module. The symmetric fusion module determines the event weights of the training image features during fusion based on the enhanced event features, and determines the image weights of the training event features during fusion based on the training image features. Finally, based on the training image features, the event weights, the training event features, and the image weights, the fused features at the current scale are obtained.

6. The target detection model training method according to claim 5, characterized in that, The event feature enhancement module includes: a max pooling layer, an average pooling layer, a splicing layer, a first convolutional layer, a first batch of normalization layers, and a first activation layer; The step of inputting the training event features at the current scale into the event feature enhancement module to obtain enhanced event features at the current scale includes: The training event features at the current scale are input into the max pooling layer for max pooling along the time dimension to obtain max pooling features. The training event features at the current scale are input into the average pooling layer for average pooling along the time dimension to obtain average pooling features. The max pooling feature and the average pooling feature are input into the stitching layer and stitched together along the channel dimension to obtain the stitched feature. Based on the first convolutional layer, the first batch normalization layer, and the first activation layer, the concatenated features are sequentially subjected to convolution processing, batch normalization processing, and activation processing to obtain the enhanced event features at the current scale.

7. The target detection model training method according to claim 5, characterized in that, The symmetric fusion module includes: a spatial channel attention enhancement module and a fusion processing module; The process involves inputting the enhanced event features, training event features, and training image features at the current scale into the symmetric fusion module. The symmetric fusion module then determines the event weights of the training image features during fusion based on the enhanced event features, and determines the image weights of the training event features during fusion based on the training image features. Finally, based on the training image features, the event weights, the training event features, and the image weights, the fused features at the current scale are obtained, including: The enhanced event features at the current scale are input into the spatial channel attention enhancement module to obtain the attention enhancement event features at the current scale. The training image features at the current scale are input into the spatial channel attention enhancement module to obtain the attention-enhanced image features at the current scale. The attention-enhanced event features and attention-enhanced image features at the current scale are input into the fusion processing module. The fusion processing module determines the event weights of the training image features during fusion based on the attention-enhanced event features, and determines the image weights of the training event features during fusion based on the attention-enhanced image features. Finally, based on the training image features, the event weights, the training event features, and the image weights, the fused features at the current scale are obtained.

8. The target detection model training method according to claim 7, characterized in that, The spatial channel attention enhancement module includes: a spatial attention enhancement module and a channel attention enhancement module; The step of inputting the enhanced event features at the current scale into the spatial channel attention enhancement module to obtain the attention enhancement event features at the current scale includes: The enhanced event features at the current scale are input into the spatial attention enhancement module to obtain the spatial enhanced event features at the current scale. The spatial enhancement event features and the enhanced event features at the current scale are input into the channel attention enhancement module to obtain the attention enhancement event features at the current scale.

9. The target detection model training method according to claim 8, characterized in that, The spatial attention enhancement module includes: a second convolutional layer; The step of inputting the enhanced event features at the current scale into the spatial attention enhancement module to obtain the spatial enhanced event features at the current scale includes: Perform a first tensor transformation on the enhanced event features at the current scale to obtain the features after the first tensor transformation; The enhanced event features at the current scale are input into the second convolutional layer for convolution processing to obtain the convolutional event features. The convolutional event features are activated according to a preset first activation function to obtain activated features. The activated features are subjected to a second tensor transformation to obtain the features after the second tensor transformation. Calculate the matrix product of the features after the first tensor transformation and the features after the second tensor transformation to obtain the spatial augmentation event features at the current scale.

10. The target detection model training method according to claim 8, characterized in that, The channel attention enhancement module includes: a third convolutional layer, a fourth convolutional layer, and a second activation layer; The step of inputting the spatial augmentation event features and the augmentation event features at the current scale into the channel attention enhancement module to obtain the attention enhancement event features at the current scale includes: The spatial augmentation event features at the current scale are subjected to a third tensor transformation to obtain the features after the third tensor transformation. The features after the third tensor transformation are sequentially convolved and activated by the third convolutional layer, the fourth convolutional layer, and the second activation layer to obtain the channel attention weights at the current scale. Calculate the dot product of the channel attention weights at the current scale and the enhanced event features at the current scale to obtain the attention-enhanced event features at the current scale.

11. The target detection model training method according to claim 7, characterized in that, The fusion processing module includes: a third activation layer, a fourth activation layer, a fifth convolutional layer, a sixth convolutional layer, and a seventh convolutional layer; The process involves inputting attention-enhanced event features and attention-enhanced image features at the current scale into the fusion processing module. The fusion processing module determines the event weights of the training image features during fusion based on the attention-enhanced event features, and determines the image weights of the training event features during fusion based on the attention-enhanced image features. Finally, based on the training image features, the event weights, the training event features, and the image weights, the fused features at the current scale are obtained, including: The attention-enhanced event features at the current scale are input into the third activation layer for activation to obtain the event weights. The product of the event weights and the training image features at the current scale is calculated to obtain the first product result. The sum of the first product result and the training image features at the current scale is calculated to obtain the event-enhanced image features. The event-enhanced image features are then input into the fifth convolutional layer for convolution processing to obtain the first fusion feature. The attention-enhanced image features at the current scale are input into the fourth activation layer for activation to obtain the image weights. The product of the image weights and the training event features at the current scale is calculated to obtain the second product result. The sum of the second product result and the training event features at the current scale is calculated to obtain the image-enhanced event features. The image-enhanced event features are then input into the sixth convolutional layer for convolution processing to obtain the second fusion feature. The first fusion feature and the second fusion feature are spliced ​​together to obtain the spliced ​​fusion feature; The maximum value of the spliced ​​fusion feature in the channel dimension is obtained to obtain the maximum fusion feature, and the average value of the spliced ​​fusion feature in the channel dimension is obtained to obtain the average fusion feature. The maximum fusion feature and the average fusion feature are concatenated and then input into the seventh convolutional layer to obtain the fused features at the current scale.

12. An electronic device, characterized in that, include: The processor and memory, the memory storing machine-readable instructions executable by the processor, wherein when the electronic device is running, the processor executes the machine-readable instructions to perform the steps of the object detection model training method as described in any one of claims 1 to 11.

Images