A raw image processing method for target detection

Abstract

The present application belongs to the field of image processing and target detection, and specifically provides a RAW image processing method for target detection, which is used to improve the compression efficiency of the image, and can significantly improve the recognition accuracy when the processed image is applied to the target detection task. The present application constructs a semantic fusion network to directly generate an image from the RAW image, which is more suitable for detection and occupies less storage. In the present application, the RAW image retains a large amount of original information, and the semantic information is obtained through the JPEG image and the label provided thereby, which is integrated into the RAW image processed by the ISP. While taking into account the detection accuracy, the required number of bits for image transmission is effectively reduced. At the same time, an efficient and close-to-training-target loss function is used to constrain the entire training process, which can significantly reduce the space occupied by the image and improve the detection rate. In summary, after introducing the RAW image processing method for target detection, the present application can improve the recognition accuracy and reduce the demand for storage space.

Description

Technical Field

[0001] This invention belongs to the field of image processing and object detection, and specifically provides a RAW image processing method for object detection. Background Technology

[0002] With the continuous advancement of social automation, the application of image information is no longer limited to human vision. Machine-to-machine image communication is becoming increasingly frequent and its importance has significantly increased. Object detection, as one of the core tasks in computer vision, aims to accurately identify and locate specific target objects from images or videos. Currently, most image object detection tasks utilize JPEG images. JPEG images are typically generated from raw RAW images through a series of processes by an Image Signal Processor (ISP). RAW images are digital images directly generated by the image sensor of a digital camera, containing a wealth of detailed information. Furthermore, to provide a better viewing experience for the human eye, major camera manufacturers optimize the parameter values ​​of each step of the Image Signal Processor based on empirical values. However, this optimization approach, geared towards human visual perception, conflicts with the requirements of object detection tasks. In object detection tasks, images are divided into foreground and background. The foreground is the region of interest (ROI) where the target is located, while the background is background information irrelevant to the task. To improve detection accuracy and efficiency, the features of the foreground target need to be enhanced, while background information should be suppressed. However, optimization methods based on human visual perception perform indiscriminate enhancement of both foreground and background, overemphasizing visual effects and details, resulting in insufficient distinction between foreground and background. When the optimized JPEG image is applied to object detection, it greatly limits the detection accuracy. Therefore, this invention provides a RAW image processing method for object detection that preserves as many key features of the RAW image as possible and focuses on enhancing them, so that the processed image is still suitable for object detection tasks. Summary of the Invention

[0003] The purpose of this invention is to provide a RAW image processing method for object detection, which improves the compression efficiency of images and significantly improves the recognition accuracy when the processed images are applied to object detection tasks.

[0004] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0005] A RAW image processing method for object detection, characterized by comprising the following steps:

[0006] Step 1. Acquire the RAW raw image generated by the image sensor of the digital camera, and generate a JPEG input image through the image signal processor (ISP); at the same time, convert the RAW raw image into a format according to single-channel RGGB segmentation to form a 4-channel RAW input image;

[0007] Step 2. Use a feature extraction network to extract features from the JPEG input image to obtain multi-scale features;

[0008] Step 3. Construct and train an ISP network with semantic fusion. Input the RAW input image and multi-scale features into the ISP network, and the ISP network outputs a 3-channel RGB image.

[0009] Furthermore, in step 2, the feature extraction network uses a pre-trained RESNET101 network.

[0010] Furthermore, in step 3, the input to the ISP network with semantic fusion is a RAW input image and multi-scale features (Features 1 to 3), and the output is a 3-channel RGB image. The ISP network with semantic fusion includes: network units U1 to U6, a downsampling module, an upsampling module, an SFT module, a residual network, and an output layer, wherein:

[0011] The structure of network unit U1 is: CONV 3×3×3×32+LeakyReLU+CONV 3×3×32×32+LeakyReLU, its input U1in is the RAW input image, and its output is the feature U1out;

[0012] After being downsampled by the downsampling module, U1out is input into the SFT module, where it is fused with feature1. The SFT module then outputs featureU1sft.

[0013] The structure of network unit U2 is: CONV 3×3×32×64+LeakyReLU+CONV 3×3×64×64+LeakyReLU, with its input U2in being feature U1sft and its output being feature U2out;

[0014] After being downsampled by the downsampling module, the feature U2out is input into the SFT module, where it is fused with the feature Feature2. The SFT module then outputs the feature U2sft.

[0015] The structure of network unit U3 is: CONV3×3×64×128+LeakyReLU+CONV 3×3×128×128+LeakyReLU, with its input U3in being feature U2sft and its output being feature U3out;

[0016] After being downsampled by the downsampling module, feature U3out is input into the SFT module, where it is fused with feature Feature3. The SFT module then outputs feature U3sft.

[0017] After passing through the residual network, feature U3sft is input into the SFT module, where it is fused with feature Feature3, and the SFT module outputs feature U4sft.

[0018] Feature U4sft is upsampled by the upsampling module and then concatenated with feature U3out. The concatenated feature is then input into network unit U4.

[0019] The structure of network unit U4 is: CONV3×3×(256+128)×256+LeakyReLU+CONV 3×3×256×128+LeakyReLU, and its output is feature U4out;

[0020] Feature U4out is input into the SFT module, where it is fused with feature Feature2, and the SFT module outputs feature U5sft.

[0021] Feature U5sft is upsampled by the upsampling module and then concatenated with feature U2out. The concatenated feature is then input into network unit U5.

[0022] The structure of network unit U5 is: CONV 3×3×(128+64)×128+LeakyReLU+CONV 3×3×128×64+LeakyReLU, and its output is feature U5out;

[0023] Feature U5out is input into the SFT module, where it is fused with feature Feature1, and the SFT module outputs feature U6sft.

[0024] Feature U6sft is upsampled by the upsampling module and then concatenated with feature U1out. The concatenated feature is then input into network unit U6.

[0025] The structure of network unit U6 is: CONV 3×3×(64+32)×64+LeakyReLU+CONV 3×3×64×32+Leaky ReLU, and its output is feature U6out;

[0026] The structure of the output layer is: CONV 1×1×32×3+Sigmoid, with U6out as the input and Out as the output, which is the 3-channel RGB image output by the feature fusion network.

[0027] Furthermore, in the ISP network, the downsampling module consists of a dilated convolutional layer with the structure: Dil-CONV 3×3×(in_channel)×(in_channel), with a stride of 2, padding of 2, and dilation of 2.

[0028] Furthermore, in the ISP network, the structure of the upsampling module is: CONV 3×3×(in_channel)×(in_channel×4)+PixelShuffle(2); a convolution is used to make the number of output channels four times the number of input channels, and after the convolution, PixelShuffle(2) is used to complete the upsampling.

[0029] Furthermore, in step 3, the training process of the ISP network is as follows: A loss function is set, and end-to-end offline training is performed on the ISP network; the loss function is specifically:

[0030] L = Perceptual Loss +Task Loss +Pixel Loss +Bpp

[0031] Where L is the total loss function, Perceptual_Loss is the perceptual loss, and Task is the perceptual loss function. Loss Pixel_Loss represents the task loss, Bpp represents the pixel loss, and Pixel_Loss represents the compression level.

[0032] Based on the above technical solution, the beneficial effects of the present invention are as follows:

[0033] This invention proposes a RAW image processing method for object detection. By constructing an ISP network with semantic fusion, it directly generates images more suitable for detection and with lower storage requirements from RAW images. In this invention, the RAW image retains a large amount of original information, and semantic information is obtained from the JPEG image and its provided tags. This information is then integrated into the RAW image processed by the ISP, effectively reducing the number of bits required for image transmission while maintaining detection accuracy. Furthermore, an efficient loss function closely aligned with the training objective is used to constrain the entire training process, significantly reducing image space requirements and improving the detection rate. In summary, this invention, by introducing a RAW image processing method for object detection, can improve recognition accuracy and reduce storage space requirements. Attached Figure Description

[0034] Figure 1 This is a flowchart illustrating the RAW image processing method for target detection in an embodiment of the present invention.

[0035] Figure 2This is a schematic diagram of the structure of an ISP network with semantic fusion in an embodiment of the present invention.

[0036] Figure 3 This is a comparison diagram of bpp-mAP between the embodiments and comparative examples of the present invention. Detailed Implementation

[0037] To make the objectives, technical solutions, and beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments.

[0038] This embodiment provides a RAW image processing method for object detection, the process of which is as follows: Figure 1 As shown, the specific steps include:

[0039] Step 1. Read the RAW image and complete the data preprocessing;

[0040] The system acquires RAW raw images generated by the image sensor of a digital camera, which are then processed by an image signal processor (ISP) to generate JPEG input images. Simultaneously, the RAW raw images are divided into single-channel RGGB segments and converted to form 4-channel RAW input images.

[0041] Step 2. Use a feature extraction network to extract features from the JPEG input image to obtain multi-scale features Feature1 to Feature3. The feature extraction network is a pre-trained RESNET101 network.

[0042] Step 3. Construct and train an ISP network with semantic fusion. Input the RAW input image and multi-scale features into the ISP network, and the ISP network outputs a 3-channel RGB image.

[0043] The ISP network, such as Figure 2 As shown, its input consists of a RAW input image and multi-scale features (Features 1 to 3), and the output is a 3-channel RGB image. Specifically, the ISP network with semantic fusion adopts the Unet framework, including: network units U1 to U6, a downsampling module, an upsampling module, an SFT module, a residual network, and an output layer, wherein:

[0044] The structure of network unit U1 is: CONV 3×3×3×32+LeakyReLU+CONV 3×3×32×32+LeakyReLU, its input U1in is the RAW input image, and its output is the feature U1out;

[0045] After being downsampled by the downsampling module, U1out is input into the SFT module, where it is fused with feature1. The SFT module then outputs featureU1sft.

[0046] The downsampling module consists of a dilated convolutional layer with the structure: Dil-CONV 3×3×(in_channel)×(in_channel), with a stride of 2, padding of 2, and dilation of 2.

[0047] The structure of network unit U2 is: CONV 3×3×32×64+LeakyReLU+CONV 3×3×64×64+LeakyReLU, with its input U2in being feature U1sft and its output being feature U2out;

[0048] After being downsampled by the downsampling module, the feature U2out is input into the SFT module, where it is fused with the feature Feature2. The SFT module then outputs the feature U2sft.

[0049] The structure of network unit U3 is: CONV3×3×64×128+LeakyReLU+CONV 3×3×128×128+LeakyReLU, with its input U3in being feature U2sft and its output being feature U3out;

[0050] After being downsampled by the downsampling module, feature U3out is input into the SFT module, where it is fused with feature Feature3. The SFT module then outputs feature U3sft.

[0051] After passing through the residual network, feature U3sft is input into the SFT module, where it is fused with feature Feature3, and the SFT module outputs feature U4sft.

[0052] Feature U4sft is upsampled by the upsampling module and then concatenated with feature U3out. The concatenated feature is then input into network unit U4.

[0053] The structure of the upsampling module is: CONV 3×3×(in_channel)×(in_channel×4)+PixelShuffle(2); a convolutional layer is used to make the number of output channels four times the number of input channels, and then PixelShuffle(2) is used to complete the upsampling after convolution;

[0054] The structure of network unit U4 is: CONV3×3×(256+128)×256+LeakyReLU+CONV 3×3×256×128+LeakyReLU, and its output is feature U4out;

[0055] Feature U4out is input into the SFT module, where it is fused with feature Feature2, and the SFT module outputs feature U5sft.

[0056] Feature U5sft is upsampled by the upsampling module and then concatenated with feature U2out. The concatenated feature is then input into network unit U5.

[0057] The structure of network unit U5 is: CONV 3×3×(128+64)×128+LeakyReLU+CONV 3×3×128×64+LeakyReLU, and its output is feature U5out;

[0058] Feature U5out is input into the SFT module, where it is fused with feature Feature1, and the SFT module outputs feature U6sft.

[0059] Feature U6sft is upsampled by the upsampling module and then concatenated with feature U1out. The concatenated feature is then input into network unit U6.

[0060] The structure of network unit U6 is: CONV 3×3×(64+32)×64+LeakyReLU+CONV 3×3×64×32+Leaky ReLU, and its output is feature U6out;

[0061] The structure of the output layer is: CONV 1×1×32×3+Sigmoid, with U6out as the input and Out as the output, which is the 3-channel RGB image output by the feature fusion network;

[0062] It should be noted that in this invention, "CONV A×B×C×D" represents a convolutional layer with a kernel size of A×B, a number of input channels of C, and a number of output channels of D; "Dil-CONV A×B×C×D" represents a dilated convolutional layer with a kernel size of A×B, a number of input channels of C, and a number of output channels of D; LeakyReLU represents the LeakyReLU (Leaky Rectified LinearUnit) activation function; Sigmoid represents the Sigmoid activation function; and PixelShuffle(2) represents 2x upsampling.

[0063] The training process for the ISP network is as follows: set the loss function, and perform end-to-end offline training on the ISP network.

[0064] In this embodiment, the PASCALRAW dataset is used, providing 600×400 JPEG images and 6034×4012 RAW images. The 6034×4012 RAW images are cropped to remove redundant data on the right and bottom sides, and then downsampled to 600×400 and stored in .npy format. The training dataset is formed by using multi-scale features extracted from JPEG images and RAW images as input, and using the official COCO annotations as labels.

[0065] The loss function is specifically as follows:

[0066] L = Perceptual Loss +Task Loss +Pixel Loss +Bpp

[0067] Where L is the total loss function, Perceptual_Loss is the perceptual loss, and Task is the perceptual loss function. Loss For task loss, Pixel_Loss is the pixel loss (L2 Loss), and Bpp is the compression degree characterization value;

[0068]

[0069] Where, x n and y n Let x be the value of the input image x and the target image y at pixel n, where N is the number of pixels;

[0070]

[0071] Where, φ i (x) and φ i (y) represents the feature representation of the input image x and the target image y at the i-th layer, λ i Let ||·||2 be the weight of each feature layer, and ||·||2 be the L2 norm.

[0072] Task Loss =L cls +L reg +L preg +L obj

[0073] Among them, L cls The classification loss is used to evaluate the model's performance in classification tasks. It is obtained by calculating the difference between the model's predicted class probabilities and the true labels, using the cross-entropy loss function.

[0074] L regThe regression loss is used to evaluate the difference between the continuous values ​​predicted by the model (such as the coordinates of the bounding box) and the true values. The regression loss function used in this embodiment is the mean squared error (MSE).

[0075] L preg The pre-regression loss is used to adjust intermediate features and prevent overfitting. The L2 norm is used to regularize the features, and its expression is:

[0076]

[0077] Where ||·||2 is the L2 norm, λ is the regularization coefficient, and f is the feature extracted by the network;

[0078] L obj For object detection tasks, the object loss is used to evaluate whether the model's predicted region contains an object. It is used to calculate the probability that the model predicts an object in each candidate region and uses the binary cross-entropy loss function.

[0079] It should also be noted that this invention provides a RAW image processing method for object detection. When the image processed by this invention is applied to different detection tasks, the task loss can be adaptively modified, which can further improve the detection rate.

[0080] Based on the above technical solution, this embodiment is tested on the PASCALRAW dataset, which contains 4259 images (each image contains its corresponding .NEF and .JPG files, with a resolution of 6016×4000). The dataset has three categories, and 300 images from each category are randomly selected, for a total of 900 images as the test set. In this embodiment, the images are downsampled to 600×400, and four quality levels (quality = 2, 3, 4, 5) are selected for comparison. During the test, the RGB images output from the ISP network are sequentially passed through a compression network and an object detection network to complete the object detection task. To illustrate the beneficial effects of this invention, Comparative Example 1 is set up: ISP processing is performed using Nikon Workshop software provided by Nikon, and end-to-end compression is performed using Cheng-2020; Comparative Example 2: Processing is performed using a de-mosaic network, and compression is performed using Cheng-2020. The test results of this embodiment and Comparative Examples 1 and 2 are as follows: Figure 3 As shown, the horizontal axis represents bpp (bits per pixel), indicating the number of bits required to transmit each pixel in the image, while the vertical axis represents mAP (mean Average Precision), used to measure the overall accuracy of the object detection task. The four points for each method represent four different image qualities, with the quality increasing from left to right. Figure 3 It can be seen that the method proposed in this invention has a significantly higher detection accuracy than the comparative method at the same or similar bit rate; especially when the image quality is high, this invention can still obtain higher mAP at a lower bit rate, which proves that the proposed method has better overall performance and practical value in the application of image compression and target detection.

[0081] In summary, the RAW image processing method for object detection proposed in this invention has excellent performance. In the PASCALRAW dataset, compared with Nikon Workshop, it can improve mAP by more than 10% at the same image quality.

[0082] The above description is merely a specific embodiment of the present invention. Any feature disclosed in this specification may be replaced by other equivalent or similar features unless otherwise specified. All disclosed features, or steps in all methods or processes, may be combined in any way except for mutually exclusive features and / or steps.

Claims

1. A RAW image processing method for object detection, characterized in that, Includes the following steps: Step 1. Acquire the RAW raw image generated by the image sensor of the digital camera, and generate a JPEG input image through the image signal processor; at the same time, convert the RAW raw image into a format according to single-channel RGGB segmentation to form a 4-channel RAW input image; Step 2. Use a feature extraction network to extract features from the JPEG input image to obtain multi-scale features; Step 3. Construct and train an ISP network with semantic fusion. Input the RAW input image and multi-scale features into the ISP network, and the ISP network outputs a 3-channel RGB image. The input to the ISP network with semantic fusion is a RAW input image and multi-scale features (Feature 1 to Feature 3), which are feature maps that incorporate semantics. An ISP network with semantic fusion includes: network units U1 to U6, a downsampling module, an upsampling module, an SFT module, a residual network, and an output layer, wherein: The structure of network unit U1 is: CONV 3×3×3×32+LeakyReLU+CONV 3×3×32×32+LeakyReLU, its input U1in is the RAW input image, and its output is the feature U1out; After being downsampled by the downsampling module, U1out is input into the SFT module, where it is fused with feature1. The SFT module then outputs featureU1sft. The structure of network unit U2 is: CONV 3×3×32×64+LeakyReLU+CONV 3×3×64×64+LeakyReLU, with its input U2in being feature U1sft and its output being feature U2out; After being downsampled by the downsampling module, the feature U2out is input into the SFT module, where it is fused with the feature Feature2. The SFT module then outputs the feature U2sft. The structure of network unit U3 is: CONV3×3×64×128+LeakyReLU+CONV 3×3×128×128+LeakyReLU, with its input U3in being feature U2sft and its output being feature U3out; After being downsampled by the downsampling module, feature U3out is input into the SFT module, where it is fused with feature Feature3. The SFT module then outputs feature U3sft. After passing through the residual network, feature U3sft is input into the SFT module, where it is fused with feature Feature3, and the SFT module outputs feature U4sft. Feature U4sft is upsampled by the upsampling module and then concatenated with feature U3out. The concatenated feature is then input into network unit U4. The structure of network unit U4 is: CONV3×3×(256+128)×256+LeakyReLU+CONV 3×3×256×128+ LeakyReLU, and its output is feature U4out; Feature U4out is input into the SFT module, where it is fused with feature Feature2, and the SFT module outputs feature U5sft. Feature U5sft is upsampled by the upsampling module and then concatenated with feature U2out. The concatenated feature is then input into network unit U5. The structure of network unit U5 is: CONV 3×3×(128+64)×128+LeakyReLU+CONV 3×3×128×64+LeakyReLU, and its output is feature U5out; Feature U5out is input into the SFT module, where it is fused with feature Feature1, and the SFT module outputs feature U6sft. Feature U6sft is upsampled by the upsampling module and then concatenated with feature U1out. The concatenated feature is then input into network unit U6. The structure of network unit U6 is: CONV 3×3×(64+32)×64+LeakyReLU+CONV 3×3×64×32+LeakyReLU, and its output is feature U6out; The structure of the output layer is: CONV 1×1×32×3+Sigmoid, with U6out as the input and Out as the output, which is the 3-channel RGB image output by the feature fusion network.

2. The RAW image processing method for target detection according to claim 1, characterized in that, In step 2, the feature extraction network uses a pre-trained RESNET101 network.

3. The RAW image processing method for target detection according to claim 1, characterized in that, In ISP networks, the downsampling module consists of dilated convolutional layers with the structure: Dil-CONV 3×3×(in_channel)×(in_channel), with a stride of 2, padding of 2, and dilation of 2.

4. The RAW image processing method for target detection according to claim 1, characterized in that, In an ISP network, the structure of the upsampling module is: CONV 3×3×(in_channel)×(in_channel×4)+PixelShuffle(2), where in_channel represents the number of input channels.

5. The RAW image processing method for target detection according to claim 1, characterized in that, In step 3, the training process of the ISP network is as follows: A loss function is set, and end-to-end offline training is performed on the ISP network; the specific loss function is: in, For the total loss function, In order to perceive loss, For mission losses, denoted by Pixel loss, and Bpp represents the degree of compression.

Images