RAW image multi-task processing network obtaining method and device

Abstract

The invention relates to a RAW image multi-task processing network obtaining method and device. The method comprises the following steps: inputting an RGB image into a mirror image backbone network to obtain an RGB feature map; an RAW image obtained through RGB image degradation is input into a backbone network of an image multi-task processing network to be trained to obtain an RAW feature map, and the backbone network and a mirror image backbone network form a twin network; splicing the RGB feature map and the RAW feature map to obtain a spliced feature map; inputting the spliced feature map into an automatic coding module to obtain a degradation prediction parameter; constructing a loss function including loss between the degradation prediction parameter and the degradation parameter; and training the image multi-task processing network to obtain an RAW image multi-task processing network. The RAW image multi-task processing network obtained by the invention can be directly applied to the RAW image obtained by the image acquisition equipment, and the image multi-task processing efficiency is improved.

Description

Technical Field

[0001] The present disclosure relates to the technical field of image processing, and in particular to a method and device for obtaining a RAW image multi-task processing network. Background Art

[0002] At present, all multi-task processing solutions for images start from RGB images. For example, an image multi-task processing network is used to extract, analyze, and infer features of RGB images to obtain results of multiple tasks, including scene classification, target recognition, key point detection, image segmentation, etc.

[0003] For image acquisition devices, in order to obtain RGB images, it is necessary to obtain RAW images from image sensors and then process the RAW images through ISP (Image Signal Processing) to obtain RGB images. It can be seen that in order to obtain multi-task processing results from images acquired by image acquisition devices in real time, in addition to the work of the image multi-task processing network, the ISP processing process is also required. Because both the image multi-task processing network and the ISP processing require corresponding processing time, there is a bottleneck in further reducing the multi-task processing delay in real-time monitoring.

[0004] Based on this, if the ISP processing flow is abandoned and real-time end-to-end multi-tasking processing is performed directly from the RAW image, the efficiency can be improved, the latency can be reduced, and the latency bottleneck can be broken through from the perspective of the task processing flow.

[0005] However, there is currently a lack of a multi-tasking network for RAW images, so the efficiency of image multi-tasking cannot be further improved. Summary of the invention

[0006] In view of this, the present disclosure provides a method and device for obtaining a RAW image multi-task processing network, so as to obtain a RAW image multi-task processing network for RAW images by using an image training set based on RGB images, so that the RAW image multi-task processing network can be directly applied to the RAW images obtained by the image acquisition device, thereby bypassing the ISP processing flow in the image acquisition device, directly obtaining the image multi-task processing results, and improving the image multi-task processing efficiency.

[0007] The technical solution of the present disclosure is achieved as follows:

[0008] A RAW image multi-task processing network acquisition method, comprising:

[0009] Acquire an image multi-task processing network and a mirror image backbone network to be trained, wherein the image multi-task processing network includes a backbone network, and a twin network is formed by the backbone network and the mirror image backbone network;

[0010] Inputting an RGB image into the mirror backbone network, and obtaining an RGB feature map associated with the RGB image from the mirror backbone network;

[0011] Inputting a RAW image obtained by degrading the RGB image into the backbone network, and obtaining a RAW feature map associated with the RAW image from the backbone network;

[0012] Splicing the RGB feature map and the RAW feature map to obtain a spliced ​​feature map;

[0013] Inputting the splicing feature map into an automatic encoding module, and obtaining degradation prediction parameters through the automatic encoding module, wherein the degradation prediction parameters are used to degenerate the RGB image into a RAW image;

[0014] Constructing a loss function, wherein the loss function includes a loss between the degradation prediction parameter and a degradation parameter, wherein the degradation parameter is a parameter used by the RGB image to degenerate to the RAW image;

[0015] The image multi-task processing network to be trained is trained according to the loss function to obtain the RAW image multi-task processing network.

[0016] Furthermore, the image multi-task processing network also includes a neck and a detection head, wherein the neck is connected to the backbone network, and the detection head is connected to the neck;

[0017] The RAW image multi-task processing network acquisition method further comprises:

[0018] Obtaining object detection prediction data associated with the RAW image from the detection head;

[0019] Inputting the RAW feature map into a classification module, and obtaining classification prediction data through the classification module;

[0020] Obtaining an object detection label value and a classification label value associated with the RGB image;

[0021] The loss function further includes the loss between the classification prediction data and the classification label value, and the loss between the object detection prediction data and the object detection label value.

[0022] Further, the degradation parameters include: brightness adjustment parameters, red gain values, blue gain values, color correction inverse transformation parameters, and gamma transformation parameters;

[0023] The degradation prediction parameters include: brightness adjustment prediction parameters, red gain prediction values, blue gain prediction values, and gamma transformation prediction parameters;

[0024] The loss between the degradation prediction parameter and the degradation parameter includes: the loss between the brightness adjustment prediction parameter and the brightness adjustment parameter, the loss between the red gain prediction value and the red gain value, the loss between the blue gain prediction value and the blue gain value, and the loss between the gamma transformation prediction parameter and the gamma transformation parameter.

[0025] Further, the training of the image multi-task processing network to be trained according to the loss function to obtain the RAW image multi-task processing network includes:

[0026] Using a back propagation algorithm, the parameters of the image multi-task processing network are updated according to the gradient of the difference loss until the difference loss is lower than a preset loss threshold or reaches a set number of iterations, thereby obtaining the RAW image multi-task processing network.

[0027] Furthermore, the auto-encoding module includes a global average pooling layer and a fully connected layer.

[0028] Further, the object detection prediction data includes at least one of target detection prediction data and key point detection prediction data, and the object detection label value includes at least one of target detection label value and key point detection label value, wherein the target detection prediction data corresponds to the target detection label value, and the key point detection prediction data corresponds to the key point detection label value;

[0029] The classification prediction data is scene classification prediction data, and the classification label value is a scene classification label value;

[0030] The loss between the object detection prediction data and the object detection label value includes:

[0031] At least one of the loss between the target detection prediction data and the target detection label value, and the loss between the key point detection prediction data and the key point detection label value.

[0032] Furthermore, the classification module includes a global average pooling layer, a convolutional layer and a fully connected layer.

[0033] Furthermore, after obtaining the RAW image multi-task processing network, the method further comprises:

[0034] Acquire a RAW original image from a camera, wherein the RAW original image is a single-channel image arranged in a Bayer format;

[0035] The RAW original image is converted into an image in a red, green and blue three-channel matrix arrangement format by using a space-to-depth transformation plus interpolation method;

[0036] Normalizing the image in the matrix arrangement format of red, green and blue color channels to obtain a normalized image;

[0037] The normalized image is input into the RAW image multi-task processing network to obtain a multi-task processing result.

[0038] A RAW image multi-task processing network acquisition device, comprising:

[0039] A network acquisition module is configured to execute acquisition of an image multi-task processing network and a mirror backbone network to be trained, wherein the image multi-task processing network includes a backbone network, and a twin network is formed by the backbone network and the mirror backbone network;

[0040] An RGB feature map acquisition module is configured to input an RGB image into the mirror backbone network and obtain an RGB feature map associated with the RGB image from the mirror backbone network;

[0041] A RAW feature map acquisition module is configured to input a RAW image obtained by degrading the RGB image into the backbone network, and obtain a RAW feature map associated with the RAW image from the backbone network;

[0042] A stitching module is configured to stitch the RGB feature map and the RAW feature map to obtain a stitching feature map;

[0043] A prediction parameter acquisition module is configured to input the splicing feature map into an automatic encoding module, and obtain a degradation prediction parameter through the automatic encoding module, wherein the degradation prediction parameter is used to degenerate the RGB image to a RAW image;

[0044] A loss construction module is configured to execute construction of a loss function, wherein the loss function includes a loss between the degradation prediction parameter and the degradation parameter, wherein the degradation parameter is a parameter used by the RGB image to degenerate to the RAW image;

[0045] The training module is configured to train the image multi-task processing network to be trained according to the loss function to obtain the RAW image multi-task processing network.

[0046] An electronic device, comprising:

[0047] processor;

[0048] a memory for storing executable instructions for the processor;

[0049] Wherein, the processor is configured to execute the executable instructions to implement the RAW image multi-tasking processing network acquisition method as described in any one of the above items.

[0050] It can be seen from the above scheme that the RAW image multi-task processing network acquisition method and device disclosed in the present invention utilize the backbone network part in the image multi-task processing network to form a twin network, and use a self-supervised learning method to train the backbone network for the degradation parameter part of the RGB image degenerated to the RAW image in the public image training set, thereby realizing the backbone network's feature extraction of the RAW image, so that the RAW image multi-task processing network obtained after training can directly extract features for the RAW image, thereby solving the problem that the existing image multi-task processing network needs to use the RGB image after ISP processing as input, and advances the input data acquisition of the image multi-task processing network to the RAW image stage, saving the complex ISP parameter debugging process and numerous ISP processing modules, improving the processing speed of the system, and better meeting the requirements of the real-time processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of a RAW image multi-task processing network acquisition method according to an exemplary embodiment;

[0052] Figure 2 is a flow chart showing degrading an RGB image to a RAW image according to an exemplary embodiment;

[0053] Figure 3 is a flow chart showing acquisition of part data of multi-tasking processing according to an exemplary embodiment;

[0054] Figure 4 is a flowchart showing a method of performing multi-task processing on a RAW image according to an exemplary embodiment;

[0055] Figure 5 is a flow chart of an application scenario of a RAW image multi-task processing network acquisition method according to an exemplary embodiment;

[0056] Figure 6 is a schematic diagram of data flow in an image multi-task processing network according to an exemplary embodiment;

[0057] Figure 7 is a logical structure diagram of a RAW image multi-task processing network acquisition device according to an exemplary embodiment;

[0058] Figure 8 It is a structural schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0059] In order to make the objectives, technical solutions and advantages of the present disclosure more clearly understood, the present disclosure is further described in detail below with reference to the accompanying drawings and examples.

[0060] Multi-task learning refers to learning and processing multiple tasks simultaneously in a single model. These tasks can be related, such as speech recognition and language translation, or unrelated, such as image classification and text generation. By learning multiple tasks in the same model, the efficiency and generalization ability of the model can be improved. Specifically, by sharing the underlying model structure and parameters, different tasks can influence and promote each other, thereby improving the performance of the entire model.

[0061] Multi-task learning aims to improve learning efficiency and prediction accuracy by learning multiple goals from shared representations. It is a common task in computer vision and is widely used in fields such as intelligent security, autonomous driving, and speech recognition. Multi-tasks in computer vision usually refer to high-level semantic tasks that extract common feature information: such as scene classification, object recognition, key point detection, image segmentation, etc.

[0062] In the current industrial multi-task learning network, existing methods usually use RGB images processed by ISP as input. Since the ISP processing process usually contains a lot of modules, the overall computational overhead is relatively large, and the processing results are closely related to the image sensor. If the image sensor changes, it will lead to more external resource consumption. If the image data can be obtained earlier for corresponding processing, the processing links can be reduced, thereby improving the processing speed and real-time performance of the entire system application.

[0063] In response to the above problems, the present disclosure provides a solution for end-to-end multi-task learning for RAW images, including scene classification tasks, target detection and key point detection tasks. In the present disclosure, the Siamese Network scheme is used to train the image multi-task processing network in a self-supervised learning manner. During the training process, the RGB images in the image training set and the RAW images obtained by the inverse process of the RGB images through the camera imaging are respectively used as the input of the twin network, and added to the twin network for self-supervised learning to adjust the image multi-task processing network for the RAW image feature extraction capability. Among them, for RAW images, the present disclosure uses a reverse mosaic operation to convert the Bayer arrangement RAW image into a red, green and blue three-channel image, wherein the two green channels in the Bayer arrangement are averaged to obtain a green channel value, and the red channel and blue channel values ​​remain unchanged. Among them, the image training set comes from a public data set, for example, it can be a COCO data set with pseudo labels annotated by a large model, or a data set containing multi-task labels after the Widerface data set is manually revised.

[0064] On the other hand, the backbone network (BackBone) in the image multi-task processing network outputs feature maps from the RAW image through the neck in the image multi-task processing network and the detection heads of each task to complete tasks such as scene classification, object detection and key point detection.

[0065] Figure 1 is a flow chart of a RAW image multi-task processing network acquisition method according to an exemplary embodiment. Figure 1 As shown, the RAW image multi-task processing network acquisition method mainly includes the following steps 101 to 107.

[0066] Step 101: obtain an image multi-task processing network and a mirror backbone network to be trained, wherein the image multi-task processing network includes a backbone network, and a twin network is composed of the backbone network and the mirror backbone network.

[0067] In an exemplary embodiment, the image multi-task processing network is, for example, a MobileNETV2 network, and the backbone network directly adopts the backbone network in the MobileNETV2 network.

[0068] The twin network is a deep learning model for comparing and matching the similarities or differences between two input instances (such as the RGB image and the RAW image in the disclosed embodiment). The twin network uses a symmetrical structure to share weights between two independent sub-networks (i.e., two backbone networks with the same structure, such as the backbone network and the mirror backbone network in the disclosed embodiment), so that the two inputs (such as the RGB image and the RAW image in the disclosed embodiment) go through the same feature extraction process. These features are used to calculate the similarity measure between the input instances.

[0069] Step 102: Input the RGB image into the mirror backbone network, and obtain an RGB feature map associated with the RGB image from the mirror backbone network.

[0070] Among them, the RGB image is an RGB image obtained from the image training set. In the exemplary embodiment, the image training set is a public data set based on RGB format images, such as the COCO data set with pseudo labels annotated by a large model, or the Widerface data set containing multi-task labels after manual revision. Because what the disclosed embodiment wants to obtain is a RAW image multi-task processing network, but there are very few RAW image training sets, and there is no existing public RAW image training set. In addition, the difficulty and workload of independently establishing a complete set of RAW image training sets with sufficient data volume are huge, and it is impossible to meet the needs of training a multi-task processing network.

[0071] Step 103: Input the RAW image obtained by degrading the RGB image into the backbone network, and obtain a RAW feature map associated with the RAW image from the backbone network.

[0072] Among them, the RAW image input to the backbone network in step 103 is the RAW image obtained by pre-degrading the RGB image input to the mirror backbone network in step 102. In the embodiment of the present disclosure, starting from the public image training set, the RGB image is degraded to the RAW image, thereby realizing the training of the image multi-task processing network.

[0073] In an exemplary embodiment, step 102 , a degradation parameter is obtained, and the RGB image is degraded to a RAW image using the degradation parameter.

[0074] In the embodiment of the present disclosure, the process of converting an RGB image to a RAW image is called degradation, that is, converting an RGB image to a RAW image is called degrading an RGB image to a RAW image. Because the image processing process of the camera is to obtain an RGB image from a RAW image through an ISP processing module, which is a forward image processing process, therefore, in the embodiment of the present disclosure, the inverse process of this forward image processing process is called degradation.

[0075] Figure 2 is a flow chart showing the degeneration of an RGB image to a RAW image according to an exemplary embodiment. Figure 2 As shown, degenerating an RGB image to a RAW image mainly includes the following steps 201 to 209.

[0076] Step 201: Perform global tone mapping inverse transformation on the RGB image.

[0077] The inverse transformation of global tone mapping is to map the luminous intensity of the display device to the signal intensity output by the image sensor of the camera. In an exemplary embodiment, the following characteristic curve formula is used to perform the inverse transformation of global tone mapping:

[0078]

[0079] Wherein, y is the image pixel information after the inverse transformation of the global tone mapping, and x is the image pixel information before the inverse transformation of the global tone mapping.

[0080] Step 202: Perform gamma correction inverse transformation on the image obtained by the global tone mapping inverse transformation, wherein the gamma correction inverse transformation adopts gamma transformation parameters.

[0081] Among them, the inverse gamma correction transform is a nonlinear transformation of the analog camera front end to convert the optical signal into an electrical signal. This nonlinear transformation can be approximated as a power function:

[0082] Y=Xγ

[0083] Among them, X is the input image, that is, the image obtained by inverse transformation of global tone mapping, Y is the image after inverse transformation of gamma correction, and γ is the gamma transformation parameter.

[0084] Here, γ may be any value randomly selected within a preset gamma transform parameter value range, which can increase the diversity of the RAW image obtained after conversion.

[0085] Step 203: Perform color correction inverse transformation on the image obtained by the gamma correction inverse transformation.

[0086] Since there is a difference between the spectrum responsivity of visible light of the human eye and the responsivity of the image sensor of the camera, the obtained RGB color value will have deviations. Color correction is usually performed through a 3×3 color transformation matrix. In the exemplary embodiment, step 203 is to multiply the image obtained by the gamma correction inverse transformation by the inverse matrix of the color transformation matrix for color correction inverse transformation, that is:

[0087]

[0088] Among them, R' is the red value obtained after the color correction inverse transformation, G' is the green value obtained after the color correction inverse transformation, B' is the blue value obtained after the color correction inverse transformation, R is the red value before the color correction inverse transformation, G is the green value before the color correction inverse transformation, B is the blue value before the color correction inverse transformation, a1 to c3 are parameter values ​​in the inverse matrix of the color transformation matrix, collectively referred to as color correction inverse transformation parameters. In the exemplary embodiment, in step 203, the values ​​of a1 to c3 are randomly selected within the range of ±20% of the fixed color correction transformation parameters in the color transformation matrix to increase the diversity of the RAW image obtained after the conversion.

[0089] Step 204: perform white balance inverse transformation on the image obtained by the color correction inverse transformation, wherein the white balance inverse transformation uses a red gain value and a blue gain value.

[0090] The white balance is to restore white objects in different ambient color temperatures to true white. In the exemplary embodiment, the image obtained after the inverse transformation of the white balance is the image information obtained by simulating a real image sensor.

[0091] In an exemplary embodiment, the following formula is used to perform the inverse white balance transformation:

[0092]

[0093] Among them, x r 、x g 、x bare the red value, green value, and blue value of any pixel in the image before the inverse white balance transformation, respectively. r ,y g ,y b are the red value, green value, and blue value of any pixel in the image after inverse white balance transformation, respectively. r The red gain value used in the inverse white balance transform process, g b The blue gain value used in the inverse white balance transform process.

[0094] Among them, g r and g b It can be any value randomly selected within the preset red gain value range and blue gain value range, which can increase the diversity of the RAW image obtained after conversion.

[0095] Step 205: Convert the image obtained after the inverse white balance transformation into a Bayer arrangement RAW domain image.

[0096] In the exemplary embodiment, when capturing an image, the light passes through a Bayer filter before the image sensor to obtain a RAW domain image, and then the Bayer arrangement image is converted into an RGB image through an interpolation algorithm. The inverse process is to convert the RGB image into a RAW domain image of the Bayer arrangement.

[0097] In an exemplary embodiment, the RAW domain image of the Bayer arrangement includes one red channel pixel value, one blue channel pixel value, and two green channel pixel values.

[0098] Step 206: extract the red channel pixel value, the blue channel pixel value and two green channel pixel values ​​from the Bayer arrangement RAW domain image.

[0099] In the exemplary embodiment, in order to ensure the consistency of the image pixel values ​​before and after degradation, it is also necessary to divide the width and height of the Bayer arrangement RAW domain image by 2 respectively.

[0100] Step 207: Averaging the two green channel pixel values ​​to obtain a green channel average pixel value.

[0101] Step 208: splice the red channel pixel value, the blue channel pixel value and the green channel average pixel value according to the RGB channel format to obtain a three-channel RAW domain image.

[0102] The operation process from step 206 to step 208 may be referred to as a demosaic operation.

[0103] Step 209: performing noise addition processing on the three-channel RAW domain image to obtain a degraded RAW image, wherein the noise addition processing uses a brightness adjustment parameter.

[0104] In the exemplary embodiment, the noise addition process mainly includes the addition of two types of noise: read noise and shot noise.

[0105] Read noise refers to the inherent electronic noise of a digital camera or image sensor, which is caused by the constant fluctuation of the amount of electrons in the pixel. Read noise is a Gaussian white noise and is usually considered an unavoidable noise source in digital image processing.

[0106] Shot noise is also called granular noise or noise. Shot noise is a random, particle-like image noise that usually appears in some high-sensitivity environments (such as low light). The reason for the generation of shot noise is the statistical uncertainty introduced when the image sensor is exposed to a weak signal or grayscale area. The image characteristics of shot noise are manifested as many tiny, bright or dim granular points in the grayscale value, which makes the image appear distorted and snowy.

[0107] In an exemplary embodiment, in step 209, the three-channel RAW domain image is subjected to noise addition processing, and the following noise generation formula may be used:

[0108]

[0109] y noise =kx+x noise

[0110] Among them, in x noise In the formula, N is the modeling of Gaussian distribution, μ is the mean, μ is set to kx, kx is the mean in the Gaussian model, kx is a distribution, noise will be sampled from the Gaussian distribution of kx to obtain specific samples, x is the pixel value of the input image before noise addition, k is the brightness adjustment parameter, σ 2 is the variance, where σ 2 Set to δ r is the read noise, δ s is the shot noise, x noise The noise information is sampled from a Gaussian distribution.

[0111] In y noise In the formula, kx represents a process of increasing brightness, y noise It is the real image pixel information after brightness adjustment and the image pixel information containing noise after the noise information is superimposed.

[0112] The entire degradation process from step 201 to step 209 is to convert the RGB image into a RAW image directly output by the image sensor. The degradation process includes multiple transformation steps, each of which has a separate control parameter. The entire degradation process is modeled in a certain simplified manner and controllable random parameters (such as randomly selected gamma transformation parameters, randomly selected color correction inverse transformation parameters, randomly selected red gain values, and randomly selected blue gain values) are introduced to improve the diversity of generated data and the robustness of the network model.

[0113] Step 104: splice the RGB feature map and the RAW feature map to obtain a spliced ​​feature map.

[0114] In an exemplary embodiment, after global average pooling is performed on the RAW feature map and the RGB feature map, they are spliced ​​according to the number of channels to obtain a spliced ​​feature map.

[0115] Step 105: input the spliced ​​feature map into an automatic encoding module, and obtain degradation prediction parameters through the automatic encoding module, wherein the degradation prediction parameters are used to degenerate the RGB image into a RAW image.

[0116] In an exemplary embodiment, the automatic encoding module includes a global average pooling layer and a fully connected layer, for example, the automatic encoding module may be composed of 1 global average pooling layer and 2 fully connected layers. In an exemplary embodiment, the automatic encoding module may further include a convolutional layer, for example, the automatic encoding module may be composed of 2 3×3 convolutional layers, 1 global average pooling layer and 2 fully connected layers.

[0117] Step 106: construct a loss function, where the loss function includes a loss between a degradation prediction parameter and a degradation parameter, wherein the degradation parameter is a parameter used to degrade an RGB image to a RAW image.

[0118] In an exemplary embodiment, the degradation prediction parameter corresponds to the degradation parameter. In an exemplary embodiment, the degradation parameter includes a brightness adjustment parameter, a red gain value, a blue gain value, and a gamma transformation parameter; and the degradation prediction parameter corresponding to the degradation parameter includes a brightness adjustment prediction parameter, a red gain prediction value, a blue gain prediction value, and a gamma transformation prediction parameter.

[0119] In an exemplary embodiment, the loss between the degradation prediction parameter and the degradation parameter includes: the loss between the brightness adjustment prediction parameter and the brightness adjustment parameter, the loss between the red gain prediction value and the red gain value, the loss between the blue gain prediction value and the blue gain value, and the loss between the gamma transformation prediction parameter and the gamma transformation parameter.

[0120] Step 107: Train the image multi-task processing network to be trained according to the loss function to obtain the RAW image multi-task processing network.

[0121] In an illustrative embodiment, step 107 may specifically include: using a back propagation algorithm to update the parameters of the image multi-task processing network according to the gradient of the difference loss until the difference loss is lower than a preset loss threshold or reaches a set number of iterations, thereby obtaining a RAW image multi-task processing network.

[0122] In the disclosed embodiment, the degradation parameter belongs to unlabeled data and has no clear label. Therefore, the learning process about degradation in step 107 in the disclosed embodiment is self-supervised learning. Self-supervised learning uses the structure of the data itself or generates tasks to create virtual labels (such as degradation parameters) and enables the model to establish meaningful feature representations without the need for real labels.

[0123] In an exemplary embodiment, the image multi-task processing network further includes a neck and a detection head, wherein the neck is connected to the backbone network and the detection head is connected to the neck. Since the disclosed embodiment is a method for obtaining a multi-task processing network for RAW images, it further includes a process for the multi-task processing part. Figure 3 is a flow chart showing how to obtain part of multi-task processing data according to an exemplary embodiment. Figure 3 As shown, in the exemplary embodiment, the RAW image multi-task processing network acquisition method of the embodiment of the present disclosure further includes the following steps 301 to 303.

[0124] Step 301: Obtain object detection prediction data associated with the RAW image from the detection head.

[0125] In an illustrative embodiment, the object detection prediction data includes at least one of target detection prediction data and key point detection prediction data.

[0126] Step 302: input the RAW feature map into the classification module, and obtain classification prediction data through the classification module.

[0127] In an exemplary embodiment, the classification prediction data is scene classification prediction data.

[0128] Step 303: Obtain object detection label values ​​and classification label values ​​associated with the RGB image.

[0129] In an exemplary embodiment, the object detection label value includes at least one of a target detection label value and a key point detection label value, wherein the target detection prediction data corresponds to the target detection label value, and the key point detection prediction data corresponds to the key point detection label value. In an exemplary embodiment, the classification label value is a scene classification label value.

[0130] Based on this, the loss function further includes the loss between the classification prediction data and the classification label value, and the loss between the object detection prediction data and the object detection label value.

[0131] In an exemplary embodiment, the loss between the object detection prediction data and the object detection label value includes at least one of the loss between the target detection prediction data and the target detection label value and the loss between the key point detection prediction data and the key point detection label value. Based on this, the learning about classification and object detection is supervised learning.

[0132] In an illustrative embodiment, the classification module includes a global average pooling layer, a convolutional layer and a fully connected layer. For example, the classification module may be composed of 1 global average pooling layer, 1 1×1 convolutional layer and 1 fully connected layer.

[0133] In an exemplary embodiment, after the RAW image multi-task processing network is obtained, the RAW image multi-task processing network may be used to perform multi-task processing on the RAW image. Figure 4 FIG. 1 is a flowchart showing a method of performing multi-task processing on a RAW image according to an exemplary embodiment. Figure 4 As shown, in the exemplary embodiment, the embodiment of the present disclosure further includes the following steps 401 to 404.

[0134] Step 401: Obtain a RAW original image from a camera, where the RAW original image is a single-channel image arranged in a Bayer format;

[0135] Step 402: Convert the RAW original image into an image in a matrix arrangement format of red, green and blue color channels by using a space-to-depth transformation plus interpolation method;

[0136] Step 403, normalizing the image in the matrix arrangement format of the red, green and blue color channels to obtain a normalized image;

[0137] Step 404: input the normalized image into the RAW image multi-task processing network to obtain a multi-task processing result.

[0138] In an illustrative embodiment, the classification task further includes: obtaining the probability of each category through the Softmax function, and finally finding the maximum value and its index in the channel direction to obtain the classification result and confidence; the target detection task further includes: obtaining the center point position and offset of each object in the feature map to obtain the real center point position, and obtaining the real target frame information together with the width and height of the rectangular frame; the key point detection task further includes: the center point position of the object obtained by the target detection task, and the offset of the predicted pixel coordinate values ​​of different key points, and the position of each key point is obtained by calculation.

[0139] The method for obtaining a RAW image multi-task processing network in the disclosed embodiment utilizes the backbone network part in the image multi-task processing network to form a twin network, and uses a self-supervised learning method to train the backbone network for the degradation parameter part of the RGB image in the public image training set degenerated to the RAW image, thereby realizing the feature extraction of the RAW image by the backbone network, so that the RAW image multi-task processing network obtained after training can directly extract features for the RAW image, thereby solving the problem that the existing image multi-task processing network needs to use the RGB image after ISP processing as input, and advances the input data acquisition of the image multi-task processing network to the RAW image stage, saving the complicated ISP parameter debugging process and numerous ISP processing modules, improving the processing speed of the system, and better meeting the requirements of the real-time processing system.

[0140] The following is a further description of the RAW image multi-task processing network acquisition method of the embodiment of the present disclosure in conjunction with a specific process.

[0141] Figure 5 FIG. 1 is a flow chart of an application scenario of a RAW image multi-task processing network acquisition method according to an exemplary embodiment. Figure 5 , including the following steps 501 to 513.

[0142] Step 501: Obtain an RGB image from a public image training set, and then execute step 502.

[0143] Step 502 : Obtain degradation parameters, use the degradation parameters to degrade the RGB image to a RAW image, and then execute step 503 .

[0144] In step 502, it mainly includes:

[0145] Perform global tone mapping inverse transform on RGB images;

[0146] Performing an inverse gamma correction transformation on the image obtained by the inverse transformation of the global tone mapping, wherein the inverse gamma correction transformation uses a gamma transformation parameter, and the gamma transformation parameter is an arbitrary value randomly selected within the range of the gamma transformation parameter value;

[0147] Performing color correction inverse transformation on the image obtained by the gamma correction inverse transformation, wherein the color correction inverse transformation parameters used are randomly selected within the range of ±20% of the fixed color correction transformation parameters in the color transformation matrix;

[0148] Performing an inverse white balance transformation on the image obtained by the color correction inverse transformation, wherein the inverse white balance transformation uses a red gain value and a blue gain value, and the red gain value and the blue gain value are any values ​​randomly selected within a preset red gain value range and a preset blue gain value range;

[0149] The image obtained by the inverse white balance transformation is converted into a Bayer arrangement RAW domain image, and the Bayer arrangement RAW domain image includes a red channel pixel value, a blue channel pixel value and two green channel pixel values;

[0150] Extracting red channel pixel values, blue channel pixel values ​​and two green channel pixel values ​​from the Bayer arrangement RAW domain image;

[0151] Averaging the two green channel pixel values ​​to obtain the green channel average pixel value;

[0152] The red channel pixel value, the blue channel pixel value and the green channel average pixel value are spliced ​​according to the RGB channel format to obtain a three-channel RAW domain image;

[0153] The three-channel RAW domain image is subjected to noise addition processing to obtain a degraded RAW image, wherein the noise addition processing adopts a brightness adjustment parameter.

[0154] Step 503, obtain the image multi-task processing network and the mirror backbone network to be trained, wherein the image multi-task processing network includes a backbone network, a neck and a detection head, and the backbone network and the mirror backbone network form a twin network, and then execute step 504.

[0155] Figure 6 is a schematic diagram of data flow in an image multi-task processing network according to an exemplary embodiment, wherein the following steps 504 to 509 are combined with Figure 6 The following is an explanation.

[0156] Step 504: input the RAW image into the backbone network, obtain a RAW feature map associated with the RAW image from the backbone network, obtain object detection prediction data associated with the RAW image from the detection head, and input the RAW feature map into the classification module, obtain classification prediction data through the classification module, and then execute step 505.

[0157] In the exemplary embodiment, for multiple tasks, the feature values ​​extracted from the three-channel RAW image are effectively downsampled by the backbone network to obtain the feature map of the multiple: the classification task directly uses the feature map obtained after 32 times downsampling, and obtains the probabilities of different categories through global average pooling and convolution layer. The detection task can adopt an anchor-free target detection method and key point detection method. The detection and key point detection tasks are completed by predicting the center point of the object, the width and height of the object, and the offset of the key point.

[0158] In an illustrative embodiment, the RAW image is extracted through a backbone network and feature maps are downsampled by 8, 16, or 32 times, and are transposed convolutionally performed on the neck and added and fused with feature maps of corresponding sizes to finally obtain a feature map downsampled by 4 times. The 4-fold feature map is used to predict the center point position of an object (such as a person or face) (including the position on the feature map and the offset between the feature map and the actual position) and the width and height of the bounding box of the object through the structure of the head.

[0159] In an exemplary embodiment, in key point detection, the offset between the key point of the face and the center point of the face category is directly regressed, and the position of the key point of the face is regressed through the coordinates of the center point and the offset.

[0160] Step 505: input the RGB image into the mirror backbone network, obtain the RGB feature map associated with the RGB image from the mirror backbone network, and then execute step 506.

[0161] Step 506 , stitching the RGB feature map and the RAW feature map to obtain a stitched feature map, step 507 .

[0162] Step 507: input the spliced ​​feature map into the automatic encoding module, and obtain the degradation prediction parameters through the automatic encoding module, wherein the degradation prediction parameters are used to degenerate the RGB image into the RAW image, and then execute step 508.

[0163] Step 508 : construct a loss function, which includes the loss between degradation prediction parameters and degradation parameters, the loss between classification prediction data and classification label values, and the loss between object detection prediction data and object detection label values, and then execute step 509 .

[0164] The loss between the object detection prediction data and the object detection label value includes at least one of the loss between the target detection prediction data and the target detection label value and the loss between the key point detection prediction data and the key point detection label value.

[0165] In an exemplary embodiment, the loss function is as follows:

[0166]

[0167] in, is the total loss; A regularization loss is used to decouple unnecessary dependencies between the outputs of different tasks; It is the overall loss function for target detection and key point detection tasks; It is the loss function of the classification task, which is implemented using the cross entropy function; is the loss function for the key point task pair, implemented using MSE (Mean Square Error); is the loss function of the supervision task of the degradation parameter in image degradation, which is implemented using MSE. w1, w2, w3, w4, and w5 are the weights of each item in the loss function, which are set according to the task of the specific item.

[0168] in, This is achieved by minimizing the absolute value of the cosine similarity.

[0169]

[0170] in, and They represent the tangents of the self-supervised task and the target detection task after the degradation operation of the streamline diagram. k represents the kth task, which refers to the self-supervised learning task after the degradation operation, l represents the lth task, which refers to the target detection task, and cosθ k,l is the cosine value of the angle between the self-supervised learning task and the target detection task. E represents the encoder task, which represents the backbone network feature extraction task. D represents the decoding task, which represents the target detection task and the self-supervised task respectively. represents a self-supervised task, Represents the object detection task.

[0171] Step 509: using the back propagation algorithm, update the parameters of the image multi-task processing network according to the gradient of the difference loss.

[0172] At this point, the training of the image multi-task processing network is completed. The process from step 501 to step 509 is repeated. Different degradation parameters can be used to degenerate the same RGB image into RAW images of different results, and the image multi-task processing network is trained again. The same degradation parameters can also be used to degenerate different RGB images into RAW images and the image multi-task processing network is trained again. Different degradation parameters can also be used to degenerate different RGB images into RAW images and the image multi-task processing network is trained again.

[0173] The process from step 501 to step 509 is repeatedly performed using different degradation parameters and / or different RGB images until the difference loss is lower than a preset loss threshold or reaches a set number of iterations, thereby obtaining a RAW image multi-task processing network.

[0174] Step 510 , obtaining a RAW original image from the camera, the RAW original image being a single-channel image arranged in a Bayer format, and then executing step 511 .

[0175] In the exemplary embodiment, the RAW image I∈R is directly acquired from the camera. H×W×1 , H and W are the height and width of the RAW original image, and R is the pixel value range based on the bit depth of the image sensor, for example:

[0176] 12-bit depth: value range is from 0 to 4095 (2 12 -1)

[0177] 14-bit depth: value range is from 0 to 16383 (2 14 -1)

[0178] 16-bit depth: value range is from 0 to 65535 (2 16 –1)

[0179] Step 511 , convert the RAW original image into an image in a red, green and blue three-color channel matrix arrangement format by using a space-to-depth transformation plus interpolation method, and then execute step 512 .

[0180] In the exemplary embodiment, step 511 is a demosaicing process. The image I is a single-channel image matrix arranged in Bayer format. Here, the space-to-depth transformation plus interpolation method is used to transform it into a matrix of three color channels to obtain an image I′∈R H / 2×W / 2×3 . The 3 color channels are arranged in red, green, and blue, that is, the image is arranged in a matrix format of red, green, and blue color channels.

[0181] Step 512 , normalize the image in the matrix arrangement format of the three color channels of red, green and blue to obtain a normalized image, and then execute step 513 .

[0182] In the exemplary embodiment, step 512 is an image preprocessing process. Since the data from the image sensor is generally 12 bits, it needs to be normalized. Here, the above RAW image is divided by 4095 to transform the data into the range of [0, 1].

[0183] Step 513: input the normalized image into the RAW image multi-task processing network to obtain a multi-task processing result.

[0184] In an illustrative embodiment, in the classification task, the probability of each category is obtained through a Softmax function, and finally the maximum value and its index are calculated in the channel direction to obtain the classification result and confidence; in the target detection task, the real center point position is obtained by obtaining the center point position and offset of each object in the feature map, and the real target frame information is obtained together with the width and height of the shape frame; in the key point detection task, the center point position of the object obtained by target detection and the offset of the predicted coordinate values ​​of different key points are used to calculate the position of each key point.

[0185] Figure 7 FIG. 1 is a schematic diagram showing the logical structure of a RAW image multi-task processing network acquisition device according to an exemplary embodiment. Figure 7 As shown, the RAW image multi-task processing network acquisition device mainly includes a network acquisition module 701, an RGB feature map acquisition module 702, a RAW feature map acquisition module 703, a splicing module 704, a prediction parameter acquisition module 705, a loss construction module 706, and a training module 707.

[0186] The network acquisition module 701 is configured to execute acquisition of an image multi-task processing network and a mirror backbone network to be trained, wherein the image multi-task processing network includes a backbone network, and a twin network is formed by the backbone network and the mirror backbone network;

[0187] The RGB feature map acquisition module 702 is configured to input the RGB image into the mirror backbone network and obtain the RGB feature map associated with the RGB image from the mirror backbone network;

[0188] A RAW feature map acquisition module 703 is configured to input the RAW image obtained by degrading the RGB image into the backbone network, and obtain a RAW feature map associated with the RAW image from the backbone network;

[0189] The stitching module 704 is configured to stitch the RGB feature map and the RAW feature map to obtain a stitched feature map;

[0190] The prediction parameter acquisition module 705 is configured to input the spliced ​​feature map into the automatic encoding module, and obtain the degradation prediction parameter through the automatic encoding module, wherein the degradation prediction parameter is used to degrade the RGB image to the RAW image;

[0191] The loss construction module 706 is configured to execute construction of a loss function, where the loss function includes a loss between a degradation prediction parameter and a degradation parameter, wherein the degradation parameter is a parameter used to degrade an RGB image to a RAW image;

[0192] The training module 707 is configured to train the image multi-task processing network to be trained according to the loss function to obtain the RAW image multi-task processing network.

[0193] In an exemplary embodiment, the image multi-task processing network further includes a neck and a detection head, the neck is connected to the backbone network, and the detection head is connected to the neck;

[0194] The RAW image multi-task processing network acquisition device further comprises:

[0195] The object detection prediction data acquisition module is configured to obtain object detection prediction data associated with the RAW image from the detection head;

[0196] The classification prediction data acquisition module is configured to input the RAW feature map into the classification module and obtain the classification prediction data through the classification module;

[0197] A label value acquisition module is configured to execute acquisition of object detection label values ​​and classification label values ​​associated with the RGB image;

[0198] Among them, the loss function further includes the loss between the classification prediction data and the classification label value, and the loss between the object detection prediction data and the object detection label value.

[0199] In an exemplary embodiment, the image training set is a public dataset based on RGB format images.

[0200] In an exemplary embodiment, the RAW image multi-task processing network acquisition device further includes:

[0201] The image degradation module is configured to perform degradation of the RGB image to the RAW image.

[0202] In an exemplary embodiment, the image degradation module includes:

[0203] A tone mapping inverse transformation submodule, configured to perform a global tone mapping inverse transformation on the RGB image;

[0204] A gamma correction inverse transformation submodule is configured to perform a gamma correction inverse transformation on an image obtained by inverse transformation of global tone mapping, wherein the gamma correction inverse transformation uses a gamma transformation parameter;

[0205] A color correction inverse transformation submodule is configured to perform color correction inverse transformation on the image obtained by the gamma correction inverse transformation;

[0206] A white balance inverse transformation submodule is configured to perform a white balance inverse transformation on the image obtained by the color correction inverse transformation, wherein the white balance inverse transformation uses a red gain value and a blue gain value;

[0207] The arrangement mode conversion submodule is configured to convert the image obtained after the white balance inverse transformation into a Bayer arrangement RAW domain image;

[0208] The pixel value extraction submodule is configured to extract a red channel pixel value, a blue channel pixel value and two green channel pixel values ​​from the Bayer arrangement RAW domain image;

[0209] A green channel average pixel value obtaining submodule is configured to perform averaging of two green channel pixel values ​​to obtain a green channel average pixel value;

[0210] The three-channel RAW domain image acquisition submodule is configured to execute splicing of the red channel pixel value, the blue channel pixel value and the green channel average pixel value according to the RGB channel format to obtain the three-channel RAW domain image;

[0211] The noise processing submodule is configured to perform noise processing on the three-channel RAW domain image to obtain a degraded RAW image, wherein the noise processing adopts a brightness adjustment parameter.

[0212] In an exemplary embodiment, the degradation parameters include: a brightness adjustment parameter, a red gain value, a blue gain value, a color correction inverse transformation parameter, and a gamma transformation parameter;

[0213] The degradation prediction parameters include: brightness adjustment prediction parameters, red gain prediction values, blue gain prediction values, and gamma transformation prediction parameters;

[0214] The loss between the degradation prediction parameters and the degradation parameters includes: the loss between the brightness adjustment prediction parameters and the brightness adjustment parameters, the loss between the red gain prediction value and the red gain value, the loss between the blue gain prediction value and the blue gain value, and the loss between the gamma transformation prediction parameters and the gamma transformation parameters.

[0215] In the exemplary embodiment, the training module 707 is further configured to perform:

[0216] Using the back propagation algorithm, the parameters of the image multi-task processing network are updated according to the gradient of the difference loss until the difference loss is lower than the preset loss threshold or reaches the set number of iterations, and the RAW image multi-task processing network is obtained.

[0217] In an exemplary embodiment, the auto-encoding module includes a global average pooling layer and a fully connected layer.

[0218] In an illustrative embodiment, the object detection prediction data includes at least one of target detection prediction data and key point detection prediction data, and the object detection label value includes at least one of a target detection label value and a key point detection label value, wherein the target detection prediction data corresponds to the target detection label value, and the key point detection prediction data corresponds to the key point detection label value;

[0219] The classification prediction data is the scene classification prediction data, and the classification label value is the scene classification label value;

[0220] The loss between object detection prediction data and object detection label values ​​includes:

[0221] At least one of the loss between target detection prediction data and target detection label value, and the loss between key point detection prediction data and key point detection label value.

[0222] In an exemplary embodiment, the classification module includes a global average pooling layer, a convolutional layer, and a fully connected layer.

[0223] In an exemplary embodiment, the RAW image multi-task processing network acquisition device of the embodiment of the present disclosure further includes:

[0224] A RAW image acquisition module is configured to acquire a RAW image from a camera, where the RAW image is a single-channel image arranged in a Bayer format;

[0225] An image conversion module is configured to convert the RAW original image into an image in a red, green, and blue three-channel matrix arrangement format by performing a space-to-depth transformation plus interpolation method;

[0226] The image normalization module is configured to perform normalization processing on an image in a matrix arrangement format of red, green and blue color channels to obtain a normalized image;

[0227] The multi-task processing result acquisition module is configured to input the normalized image into the RAW image multi-task processing network to obtain the multi-task processing result.

[0228] The RAW image multi-task processing network acquisition device of the disclosed embodiment utilizes the backbone network part in the image multi-task processing network to form a twin network, and uses a self-supervised learning method to train the backbone network for the degradation parameter part of the RGB image in the public image training set degenerated to the RAW image, thereby realizing the feature extraction of the RAW image by the backbone network, so that the RAW image multi-task processing network obtained after training can directly extract features for the RAW image, thereby solving the problem that the existing image multi-task processing network needs to use the RGB image after ISP processing as input, and advances the input data acquisition of the image multi-task processing network to the RAW image stage, saving the complicated ISP parameter debugging process and numerous ISP processing modules, improving the processing speed of the system, and better meeting the requirements of the real-time processing system.

[0229] All the above optional technical solutions can be arbitrarily combined to form optional embodiments of the present disclosure, and will not be described in detail here.

[0230] Regarding the RAW image multi-task processing network acquisition device in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the RAW image multi-task processing network acquisition method, and will not be elaborated here.

[0231] It should be noted that: the above embodiment only illustrates the division of the above functional modules. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0232] Figure 8 It is a structural diagram of an electronic device provided by an embodiment of the present disclosure. In some embodiments, the electronic device is a server. The electronic device 800 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 801 and one or more memories 802, wherein the memory 802 stores at least one program code, and the at least one program code is loaded and executed by the processor 801 to implement the RAW image multi-tasking network acquisition method provided by the above-mentioned various embodiments. Of course, the electronic device 800 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 800 may also include other components for realizing device functions, which will not be repeated here.

[0233] In an exemplary embodiment, a computer-readable storage medium including at least one instruction is also provided, such as a memory including at least one instruction. The at least one instruction can be executed by a processor in a computer device to complete the RAW image multi-tasking processing network acquisition method in the above embodiment.

[0234] Optionally, the above-mentioned computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may include ROM (Read-Only Memory), RAM (Random-Access Memory), CD-ROM (Compact Disc Read-Only Memory), magnetic tape, floppy disk and optical data storage device, etc.

[0235] In an exemplary embodiment, a computer program product is also provided, including one or more instructions, which can be executed by a processor of a computer device to implement the RAW image multi-tasking processing network acquisition method provided by the above-mentioned embodiments.

[0236] The above description is only a preferred embodiment of the present disclosure and is not intended to limit the present disclosure. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present disclosure shall be included in the scope of protection of the present disclosure.

Claims

1. A RAW image multi-task processing network acquisition method, comprising: Acquire an image multi-task processing network and a mirror image backbone network to be trained, wherein the image multi-task processing network includes a backbone network, and a twin network is formed by the backbone network and the mirror image backbone network; Inputting an RGB image into the mirror backbone network, and obtaining an RGB feature map associated with the RGB image from the mirror backbone network; Inputting a RAW image obtained by degrading the RGB image into the backbone network, and obtaining a RAW feature map associated with the RAW image from the backbone network; Splicing the RGB feature map and the RAW feature map to obtain a spliced ​​feature map; Inputting the splicing feature map into an automatic encoding module, and obtaining degradation prediction parameters through the automatic encoding module, wherein the degradation prediction parameters are used to degenerate the RGB image into a RAW image; Constructing a loss function, wherein the loss function includes a loss between the degradation prediction parameter and a degradation parameter, wherein the degradation parameter is a parameter used by the RGB image to degenerate to the RAW image; The image multi-task processing network to be trained is trained according to the loss function to obtain the RAW image multi-task processing network.

2. The RAW image multi-task processing network acquisition method according to claim 1, characterized in that: The image multi-task processing network also includes a neck and a detection head, wherein the neck is connected to the backbone network, and the detection head is connected to the neck; The RAW image multi-task processing network acquisition method further comprises: Obtaining object detection prediction data associated with the RAW image from the detection head; Inputting the RAW feature map into a classification module, and obtaining classification prediction data through the classification module; Obtaining an object detection label value and a classification label value associated with the RGB image; The loss function further includes the loss between the classification prediction data and the classification label value, and the loss between the object detection prediction data and the object detection label value.

3. The RAW image multi-task processing network acquisition method according to claim 1, characterized in that: The degradation parameters include: brightness adjustment parameters, red gain value, blue gain value, color correction inverse transformation parameters, and gamma transformation parameters; The degradation prediction parameters include: brightness adjustment prediction parameters, red gain prediction values, blue gain prediction values, and gamma transformation prediction parameters; The loss between the degradation prediction parameter and the degradation parameter includes: the loss between the brightness adjustment prediction parameter and the brightness adjustment parameter, the loss between the red gain prediction value and the red gain value, the loss between the blue gain prediction value and the blue gain value, and the loss between the gamma transformation prediction parameter and the gamma transformation parameter.

4. The RAW image multi-task processing network acquisition method according to claim 1, characterized in that: The step of training the image multi-task processing network to be trained according to the loss function to obtain the RAW image multi-task processing network comprises: Using a back propagation algorithm, the parameters of the image multi-task processing network are updated according to the gradient of the difference loss until the difference loss is lower than a preset loss threshold or reaches a set number of iterations, thereby obtaining the RAW image multi-task processing network.

5. The RAW image multi-task processing network acquisition method according to claim 1, characterized in that: The auto-encoding module includes a global average pooling layer and a fully connected layer.

6. The RAW image multi-task processing network acquisition method according to claim 2, characterized in that: The object detection prediction data includes at least one of target detection prediction data and key point detection prediction data, and the object detection label value includes at least one of target detection label value and key point detection label value, wherein the target detection prediction data corresponds to the target detection label value, and the key point detection prediction data corresponds to the key point detection label value; The classification prediction data is scene classification prediction data, and the classification label value is a scene classification label value; The loss between the object detection prediction data and the object detection label value includes: At least one of the loss between the target detection prediction data and the target detection label value, and the loss between the key point detection prediction data and the key point detection label value.

7. The RAW image multi-task processing network acquisition method according to claim 2, characterized in that: The classification module includes a global average pooling layer, a convolutional layer and a fully connected layer.

8. The RAW image multi-task processing network acquisition method according to any one of claims 1 to 7, characterized in that: After obtaining the RAW image multi-task processing network, the method further comprises: Acquire a RAW original image from a camera, wherein the RAW original image is a single-channel image arranged in a Bayer format; The RAW original image is converted into an image in a matrix arrangement format of red, green and blue color channels by using a space-to-depth transformation plus interpolation method; Normalizing the image in the matrix arrangement format of red, green and blue color channels to obtain a normalized image; The normalized image is input into the RAW image multi-task processing network to obtain a multi-task processing result.

9. A RAW image multi-task processing network acquisition device, characterized in that: include: A network acquisition module is configured to execute acquisition of an image multi-task processing network and a mirror backbone network to be trained, wherein the image multi-task processing network includes a backbone network, and a twin network is formed by the backbone network and the mirror backbone network; An RGB feature map acquisition module is configured to input an RGB image into the mirror backbone network and obtain an RGB feature map associated with the RGB image from the mirror backbone network; A RAW feature map acquisition module is configured to input a RAW image obtained by degrading the RGB image into the backbone network, and obtain a RAW feature map associated with the RAW image from the backbone network; A stitching module is configured to stitch the RGB feature map and the RAW feature map to obtain a stitching feature map; A prediction parameter acquisition module is configured to input the splicing feature map into an automatic encoding module, and obtain a degradation prediction parameter through the automatic encoding module, wherein the degradation prediction parameter is used to degenerate the RGB image to a RAW image; A loss construction module is configured to execute construction of a loss function, wherein the loss function includes a loss between the degradation prediction parameter and a degradation parameter, wherein the degradation parameter is a parameter used by the RGB image to degenerate to the RAW image; The training module is configured to train the image multi-task processing network to be trained according to the loss function to obtain the RAW image multi-task processing network.

10. An electronic device, characterized in that: include: processor; a memory for storing executable instructions for the processor; The processor is configured to execute the executable instructions to implement the RAW image multi-tasking processing network acquisition method as described in any one of claims 1 to 8.

Images