RAW image robust target detection method based on environmental perception and mixed feature modulation
By using a lightweight neural image signal processing and hybrid feature modulation module, the problem of insufficient robustness of sRGB detection in complex environments is solved, achieving efficient and interpretable target detection in RAW images, and improving detection accuracy and robustness.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-01-04
- Publication Date
- 2026-03-27
AI Technical Summary
Existing sRGB-based detection methods lack robustness in complex environments, making it difficult to adapt to conditions such as rain, snow, fog, haze, backlight, and low illumination, resulting in false detections, missed detections, and insufficient generalization. Furthermore, existing RAW methods have complex architectures and limited interpretability, making them difficult to use in conjunction with mainstream sRGB pre-trained detectors.
A lightweight neural image signal processor (MonoISP) is used for pixel-level conversion from RAW to sRGB, and a scene-aware hybrid feature modulation module (MoM) is designed to dynamically adjust the internal features of the detection model through environmental perception to achieve adaptive perception.
It significantly improves the robustness of target detection in robot systems under varying degradation conditions, reduces computational overhead, maintains compatibility with existing sRGB pre-trained detectors, improves detection accuracy and robustness, and reduces false negative and false positive rates.
Smart Images
Figure CN121746818A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of image processing and understanding, and relates to a robust target detection method for RAW images based on environmental perception and hybrid feature modulation. Background Technology
[0002] In real-world scenarios, robot vision systems often operate under complex imaging conditions such as rain, snow, fog, haze, backlighting, and low illumination. This increases image noise and blurs details, making it difficult for detection models to function stably. Existing methods generally rely on sRGB images generated through camera image signal processing for training and inference. However, during denoising, color correction, and tone mapping, many visual cues relevant to the recognition task are weakened, making it difficult to represent true optical characteristics. With the increasing demand for long-term reliable perception in applications such as autonomous driving, drone inspection, and service robots, this sRGB detection paradigm based on ideal imaging conditions exhibits significant false positives, false negatives, and insufficient generalization in harsh environments, becoming a core bottleneck limiting system robustness. Therefore, breaking through existing sRGB perception models and constructing a detection system more adaptable to complex environments has become a crucial problem urgently needing to be solved in this field.
[0003] Therefore, to address the insufficient robustness of sRGB-based detection in complex environments, existing research mainly follows two paths: one approach (such as Zhong-Yu Li's "Towards RAW Object Detection in Diverse Conditions") attempts to directly train object detection models on RAW data to fully utilize the original imaging information; the other approach (such as Ziteng Cui's "RAW-Adapter: Adapting Pre-trained Visual Model to Camera RAW Images and A Benchmark") constructs a learnable ISP to convert RAW into a more task-appropriate RGB representation, thereby reusing existing sRGB pre-trained detectors. However, while the former can acquire richer underlying signals, it typically relies on large-scale synthetic RAW data for pre-training, resulting in high computational costs and susceptibility to domain bias, making it difficult to generalize in real-world scenarios. The latter approach improves model compatibility and stability through task-driven image reconstruction, but generally suffers from structural complexity, limited interpretability, and insufficient adaptability to environmental changes. Summary of the Invention
[0004] While existing RAW-based object detection methods have explored various aspects such as model structure design, learnable ISP design, and environmental adaptation, they still generally suffer from problems such as complex architecture, insufficient interpretability, lack of a unified adaptation mechanism for various harsh conditions, and difficulty in cooperating with mainstream sRGB pre-trained detectors, limiting their application efficiency and generalization ability in real-world scenarios. Therefore, this invention proposes a robust object detection method for RAW images based on environmental perception and mixed feature modulation. This method adopts a lightweight and interpretable two-layer adaptive structure: First, a lightweight neural image signal processor (MonoISP) is introduced, simulating only core transformations such as white balance, color correction, and tone mapping to achieve pixel-level conversion from RAW to sRGB, reducing computational overhead and preserving key visual information; Second, a scene-aware mixed feature modulation module (MoM) is designed to dynamically adjust the internal features of the detection model based on the environmental estimation results, achieving adaptive perception of complex conditions such as rain, fog, and low light. The method of this invention is lightweight, efficient and generalizable. It can achieve end-to-end training while maintaining full compatibility with existing sRGB pre-trained backbones, and significantly improve the robustness of robot system to target detection under variable degradation conditions.
[0005] The technical solution of this invention:
[0006] A robust target detection method for RAW images based on environment perception and hybrid feature modulation, comprising the following steps:
[0007] Step 1: RAW image processing;
[0008] Step 1.1: Extract features from the input RAW image using a RAW feature extractor to obtain a RAW domain feature representation;
[0009] The RAW feature extractor consists of multiple cascaded lightweight visual Transformer blocks, each of which contains a multi-head self-attention mechanism and a feedforward neural network.
[0010] Step 1.2: Based on RAW domain feature representation, predict the processing parameters of the lightweight neural image signal processor, and predict the weight coefficients corresponding to one or more environmental conditions;
[0011] The lightweight neural image signal processor includes a white balance module, a color correction module, and a tone mapping module. Its processing parameters are predicted by different linear prediction layers based on RAW domain feature representations.
[0012] Step 1.2.1: Input the RAW domain feature representation into three independent linear prediction layers to obtain the three-channel gain vectors used for white balance operation. 3×3 color correction matrix for color correction and the lookup table residual vector used for tone mapping These three parameters are processing parameters for a lightweight neural image signal processor; among them, Let L be the set of real numbers, and L be the length of the lookup table;
[0013] Step 1.2.2: Input the RAW domain feature representation into a conditional classifier to predict the probability that the current input RAW image belongs to one or more preset environmental conditions. ;in, An index for environmental conditions;
[0014] Steps 1, 2, and 3: Probability Normalization is performed to obtain the weighting coefficients corresponding to various environmental conditions. .
[0015] Step 1.3: Using the processing parameters of the lightweight neural image signal processor predicted in Step 1.2, the input RAW image is converted into an sRGB domain image by the lightweight neural image signal processor; the lightweight neural image signal processor performs the following operations in sequence:
[0016] Step 1.3.1, White Balance Correction: The white balance module uses the three-channel gain vector predicted in Step 1.2.1. For RAW images that have undergone de-mosaicing and black level correction Perform channel-level multiplication to obtain the white balance image. ;in, This indicates multiplication by channel;
[0017] Step 1.3.2, Color Correction: The color correction module uses the 3×3 color correction matrix predicted in Step 1.2.1. For the white balance image obtained in step 1.3.1 Perform color space transformation to obtain the corrected image. ;in, Represents matrix multiplication;
[0018] Step 1.3.3, Tone Mapping: The tone mapping module utilizes a fixed gamma curve. Compared with the lookup table residual vector predicted in step 1.2.1 Commonly defined 1D lookup table For the image corrected in step 1.3.2 Perform tone mapping and output an sRGB domain image. ;in, Represents a 1D lookup table Mapping operations, This represents the input to a 1D lookup table, i.e., an image. The normalized intensity value of the middle pixel.
[0019] Step 2: Target detection in environmental perception;
[0020] Step 2.1: Convert the sRGB domain image Input object detection network;
[0021] Step 2.2: After the backbone network of the target detection network completes the first downsampling stage, extract its output intermediate feature map. ; Use the weighting coefficients of various environmental conditions obtained in step 1.2.3 The intermediate feature map is processed by the hybrid feature modulation module. Adaptive modulation is performed, and the modulated feature map is obtained. The data is then fed into subsequent layers of the backbone network to complete target localization and classification, and output the detection results.
[0022] Step 2.2.1: The hybrid feature modulation module includes multiple condition-specific feature modulators, each corresponding to a preset environmental condition; the k-th feature modulator is parameterized as a channel-level scaling factor. With bias term intermediate feature map The output is obtained after passing through the kth feature modulator. Its formula is ;
[0023] Step 2.2.2: Utilize the weighting coefficients of various environmental conditions obtained in Step 1.2.3. The output of each characteristic modulator Weighted fusion is performed to obtain the modulated feature map. Its formula is: Where N is the number of preset environmental conditions;
[0024] Step 2.2.3: Modulate the feature map obtained in step 2.2.2. The data is then fed into subsequent layers of the backbone network to complete target localization and classification, and output the detection results.
[0025] The aforementioned RAW feature extractor, lightweight neural image signal processor, hybrid feature modulation module, and target detection network are jointly trained end-to-end.
[0026] Total loss function during training Loss due to target detection Image reconstruction loss Losses classified according to environmental conditions The weighted sum is obtained as follows:
[0027]
[0028] in, and This is the balance coefficient;
[0029] The target detection loss includes classification loss and bounding box regression loss, and its loss function definition is consistent with Cascade R-CNN;
[0030] For image reconstruction loss, the L1 loss function is used to supervise sRGB domain images. Compared to real RGB images Consistency between them;
[0031] For the environmental condition classification loss, a binary cross-entropy loss function is used, defined as:
[0032]
[0033] in, This is the true label for the k-th environmental condition. The number of preset environmental conditions.
[0034] The target detection network is the Cascade R-CNN detection framework, whose backbone network is ResNet, SwinTransformer, or ConvNeXt.
[0035] The beneficial effects of this invention are:
[0036] 1. The robust target detection method for RAW images based on environment perception and hybrid feature modulation provided by this invention overcomes the technical problems of insufficient robustness of existing sRGB detection in harsh scenes such as rain, fog and low light, as well as the complex structure of RAW methods and the difficulty in uniformly adapting to various degradation conditions, and significantly improves the robust detection capability of the model in real environment.
[0037] 2. This invention designs a simple and efficient neural ISP that enables rapid conversion of RAW images to task-optimized sRGB representations. Through end-to-end joint training with subsequent detectors, the ISP can adaptively retain detailed visual information that is crucial for target recognition, thereby significantly improving overall detection performance.
[0038] 3. This invention innovatively proposes a hybrid feature modulation module, which enables the model to perceive adverse imaging conditions such as rain, fog, and low light in the scene, and adaptively modulate the visual features accordingly. This effectively enhances the detector's representation and discrimination capabilities in complex environments, and improves the model's uniform adaptability and environmental robustness to various degradation scenarios.
[0039] 4. Compared with existing RAW target detection technology, the method of this invention achieves better detection accuracy under various mainstream pre-trained backbone networks, effectively reduces the false negative rate and false positive rate, and has more stable and reliable overall performance, with significant engineering application value.
[0040] 0015.5. While introducing key modules such as feature modulation and neural ISP, this invention can still maintain a baseline speed of about 96% in the inference stage, which is significantly better than the real-time performance degradation problem that commonly occurs in existing methods after increasing model complexity, and achieves a good balance between performance improvement and efficient inference. Attached Figure Description
[0041] Figure 1 This is a schematic diagram of the overall framework of the present invention.
[0042] Figure 2 This is a schematic diagram of the feature modulation module in this invention. Detailed Implementation
[0043] The specific embodiments of the present invention will be further described below with reference to the accompanying drawings and technical solutions.
[0044] Example
[0045] In this embodiment, the overall framework of the present invention is built based on the Cascade R-CNN detection framework. The backbone networks of the object detection network are ResNet, Swin Transformer, and ConvNeXt models pre-trained on the ImageNet dataset, respectively, to verify the adaptability and effectiveness of the method under different network structures. The experimental data uses the AODRaw dataset and is divided into training and test sets according to conventional settings. During the training phase, data augmentation strategies such as random scaling and horizontal flipping are used, and a total of 12 training rounds are performed.
[0046] The AdamW optimizer was used during model training, with a weight decay factor of 0.05, a learning rate of 0.0002, and a batch size of 16. Both training and testing were performed on a server platform equipped with four NVIDIA A800 Tensor Core GPUs.
[0047] like Figure 1As shown, a robust target detection method for RAW images based on environmental perception and hybrid feature modulation is described, with the following steps:
[0048] Step 1: RAW image processing;
[0049] Step 1.1: Extract features from the input RAW image using a RAW feature extractor to obtain a RAW domain feature representation;
[0050] The RAW feature extractor consists of multiple cascaded lightweight visual Transformer blocks, each of which contains a multi-head self-attention mechanism and a feedforward neural network.
[0051] Step 1.2: Based on RAW domain feature representation, predict the processing parameters of the lightweight neural image signal processor and predict the weight coefficients corresponding to one or more environmental conditions; the lightweight neural image signal processor includes a white balance module, a color correction module, and a tone mapping module, and its processing parameters are predicted by different linear prediction layers based on RAW domain feature representation;
[0052] Step 1.2.1: Input the RAW domain feature representation into three independent linear prediction layers to obtain the three-channel gain vectors used for white balance operation. 3×3 color correction matrix for color correction and the lookup table residual vector used for tone mapping These three parameters are processing parameters for a lightweight neural image signal processor; among them, Let L be the set of real numbers, and L be the length of the lookup table;
[0053] Step 1.2.2: Input the RAW domain feature representation into a conditional classifier to predict the probability that the current input RAW image belongs to one or more preset environmental conditions. ;in, An index for environmental conditions;
[0054] Steps 1, 2, and 3: Probability Normalization is performed to obtain the weighting coefficients corresponding to various environmental conditions. .
[0055] Step 1.3: Using the processing parameters of the lightweight neural image signal processor predicted in Step 1.2, the input RAW image is converted into an sRGB domain image by the lightweight neural image signal processor.
[0056] The lightweight neural image signal processor performs the following operations in sequence:
[0057] Step 1.3.1, White Balance Correction: The white balance module uses the three-channel gain vector predicted in Step 1.2.1. For RAW images that have undergone de-mosaicing and black level correction Perform channel-level multiplication to obtain the white balance image. ;in, This indicates multiplication by channel;
[0058] Step 1.3.2, Color Correction: The color correction module uses the 3×3 color correction matrix predicted in Step 1.2.1. For the white balance image obtained in step 1.3.1 Perform color space transformation to obtain the corrected image. ;in, Represents matrix multiplication;
[0059] Step 1.3.3, Tone Mapping: The tone mapping module utilizes a fixed gamma curve. Compared with the lookup table residual vector predicted in step 1.2.1 Commonly defined 1D lookup table For the image corrected in step 1.3.2 Perform tone mapping and output an sRGB domain image. ;in, Represents a 1D lookup table Mapping operations, This represents the input to a 1D lookup table, i.e., an image. The normalized intensity value of the middle pixel.
[0060] Step 2: Target detection in environmental perception;
[0061] Step 2.1: Convert the sRGB domain image Input the object detection network for subsequent steps; the object detection network is the Cascade R-CNN detection framework, whose backbone network is ResNet, Swin Transformer or ConvNeXt;
[0062] Step 2.2: After the backbone network of the target detection network completes the first downsampling stage, extract its output intermediate feature map. ; Use the weighting coefficients of various environmental conditions obtained in step 1.2.3 The intermediate feature map is processed by the hybrid feature modulation module. Adaptive modulation is performed, and the modulated feature map is obtained. The data is then fed into subsequent layers of the backbone network to complete target localization and classification, and output the detection results.
[0063] Step 2.2.1: The hybrid feature modulation module includes multiple condition-specific feature modulators, each corresponding to a preset environmental condition; the k-th feature modulator is parameterized as a channel-level scaling factor. With bias term intermediate feature map The output is obtained after passing through the kth feature modulator. Its formula is ;
[0064] Step 2.2.2: Utilize the weighting coefficients of various environmental conditions obtained in Step 1.2.3. The output of each characteristic modulator Weighted fusion is performed to obtain the modulated feature map. Its formula is: Where N is the number of preset environmental conditions;
[0065] Step 2.2.3: Modulate the feature map obtained in step 2.2.2. The data is then fed into subsequent layers of the backbone network to complete target localization and classification, and output the detection results.
[0066] This invention validates its technical advantages on the AODRaw object detection dataset: 1) In terms of detection capability, this invention achieves stable improvements under various pre-trained backbone networks such as ResNet-18, Swin-T, and ConvNeXt-T, with mAP improvements of 0.6%, 1.0%, and 0.6% respectively compared to the baseline model. Furthermore, it achieves gains of 0.5%, 0.6%, and 1.1% under adverse conditions such as low light, rain, and fog, effectively demonstrating that the proposed MonoISP and MoM modules possess significant feature representation enhancement capabilities in complex and degraded environments. In addition, on the LOD dataset, this invention achieves improvements of 2.8% and 0.8% respectively compared to existing methods RAOD and AODRaw, further demonstrating the universality and robustness of this invention. 2) In terms of computational efficiency, this invention introduces only minimal training and inference overhead, reducing inference speed by only about 4%, far superior to the additional 62.8% training time and 36.3% inference time of the RAOD method, and the significant efficiency decrease caused by the lengthy pre-training and additional CPU processing required by the AODRaw method.
[0067] Experimental results show that the present invention significantly enhances the model's adaptability to harsh environments while maintaining efficient inference, providing an efficient, reliable and widely applicable solution for robust object detection under RAW image conditions.
[0068] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.
Claims
1. A robust target detection method for RAW images based on environmental perception and hybrid feature modulation, characterized in that, The steps are as follows: Step 1: RAW image processing; Step 1.1: Extract features from the input RAW image using a RAW feature extractor to obtain a RAW domain feature representation; The RAW feature extractor consists of multiple cascaded lightweight visual Transformer blocks, each of which contains a multi-head self-attention mechanism and a feedforward neural network. Step 1.2: Based on RAW domain feature representation, predict the processing parameters of the lightweight neural image signal processor, and predict the weight coefficients corresponding to one or more environmental conditions; The lightweight neural image signal processor includes a white balance module, a color correction module, and a tone mapping module. Its processing parameters are predicted by different linear prediction layers based on RAW domain feature representations. Step 1.3: Using the processing parameters of the lightweight neural image signal processor predicted in Step 1.2, the input RAW image is converted into an sRGB domain image by the lightweight neural image signal processor. Step 2: Target detection in environmental perception; Step 2.1: Convert the sRGB domain image Input object detection network; Step 2.2: After the backbone network of the target detection network completes the first downsampling stage, extract its output intermediate feature map. ; Use the weighting coefficients of various environmental conditions obtained in step 1.2.3 The intermediate feature map is processed by the hybrid feature modulation module. Adaptive modulation is performed, and the modulated feature map is obtained. The data is then fed into subsequent layers of the backbone network to complete target localization and classification, and output the detection results.
2. The robust target detection method for RAW images based on environmental perception and hybrid feature modulation according to claim 1, characterized in that, The RAW feature extractor, lightweight neural image signal processor, hybrid feature modulation module and object detection network are jointly trained end-to-end; Total loss function during training Loss due to target detection Image reconstruction loss Losses classified according to environmental conditions The weighted sum is obtained as follows: ； in, and This is the balance coefficient; The target detection loss includes classification loss and bounding box regression loss, and its loss function definition is consistent with CascadeR-CNN; For image reconstruction loss, the L1 loss function is used to supervise sRGB domain images. Compared to real RGB images Consistency between them; For the environmental condition classification loss, a binary cross-entropy loss function is used, defined as: L cond =- 1 N ∑ k=1 N [ y k log p k +(1- y k ) log (1- p k )] ； in, This is the true label for the k-th environmental condition. The number of preset environmental conditions.
3. The robust target detection method for RAW images based on environmental perception and hybrid feature modulation according to claim 2, characterized in that, The object detection network is the Cascade R-CNN detection framework, whose backbone network is ResNet, SwinTransformer, or ConvNeXt.
4. The robust target detection method for RAW images based on environmental perception and hybrid feature modulation according to claim 3, characterized in that, The specific implementation process of step 1.2 is as follows: Step 1.2.1: Input the RAW domain feature representation into three independent linear prediction layers to obtain the three-channel gain vectors used for white balance operation. 3×3 color correction matrix for color correction and the lookup table residual vector used for tone mapping These three parameters are processing parameters for a lightweight neural image signal processor; among them, Let L be the set of real numbers, and L be the length of the lookup table; Step 1.2.2: Input the RAW domain feature representation into a conditional classifier to predict the probability that the current input RAW image belongs to one or more preset environmental conditions. ;in, An index for environmental conditions; Steps 1, 2, and 3: Probability Normalization is performed to obtain the weighting coefficients corresponding to various environmental conditions. .
5. The robust target detection method for RAW images based on environmental perception and hybrid feature modulation according to claim 4, characterized in that, The lightweight neural image signal processor performs the following operations in sequence: Step 1.3.1, White Balance Correction: The white balance module uses the three-channel gain vector predicted in Step 1.2.
1. For RAW images that have undergone de-mosaicing and black level correction Perform channel-level multiplication to obtain the white balance image. ;in, This indicates multiplication by channel; Step 1.3.2, Color Correction: The color correction module uses the 3×3 color correction matrix predicted in Step 1.2.
1. For the white balance image obtained in step 1.3.1 Perform color space transformation to obtain the corrected image. ;in, Represents matrix multiplication; Step 1.3.3, Tone Mapping: The tone mapping module utilizes a fixed gamma curve. Compared with the lookup table residual vector predicted in step 1.2.1 Commonly defined 1D lookup table For the image corrected in step 1.3.2 Perform tone mapping and output an sRGB domain image. ;in, Represents a 1D lookup table Mapping operations, This represents the input to a 1D lookup table, i.e., an image. The normalized intensity value of the middle pixel.
6. The robust target detection method for RAW images based on environmental perception and hybrid feature modulation according to claim 5, characterized in that, The specific implementation process of step 2.2 is as follows: Step 2.2.1: The hybrid feature modulation module includes multiple condition-specific feature modulators, each corresponding to a preset environmental condition; the k-th feature modulator is parameterized as a channel-level scaling factor. With bias term intermediate feature map The output is obtained after passing through the kth feature modulator. Its formula is ; Step 2.2.2: Utilize the weighting coefficients of various environmental conditions obtained in Step 1.2.
3. The output of each characteristic modulator Weighted fusion is performed to obtain the modulated feature map. Its formula is: Where N is the number of preset environmental conditions; Step 2.2.3: Modulate the feature map obtained in step 2.2.
2. The data is then fed into subsequent layers of the backbone network to complete target localization and classification, and output the detection results.