Image enhancement method, and training method and device of image enhancement network
By using a brightness curve prediction model and a perception-driven iterative optimization method, the problems of information distortion and loss in traditional image signal processing are solved, achieving the optimal image enhancement effect and improving the image clarity and information integrity of vehicle environment perception.
Patent Information
- Application Number
- CN202410799843.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-06-19
- Publication Date
- 2024-11-22
AI Technical Summary
Traditional image signal processing methods suffer from information distortion and loss in vehicle environment perception, which affects the algorithm performance of downstream perception tasks.
Image enhancement is achieved by using a brightness curve prediction model, and the model is iteratively optimized using perception results and perception labels to realize end-to-end image enhancement and improve image quality.
It achieves the optimal image quality enhancement effect, improves image clarity while maintaining information integrity, and reduces the demand for computing and storage resources.
Smart Images

Figure CN119006306A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the technical field of intelligent driving, and in particular to an image enhancement method, and a training method and device for an image enhancement network. Background Art
[0002] In recent years, with the rapid development of artificial intelligence, machine vision and other fields, assisted driving and autonomous driving have become important research areas in academia and industry. Environmental perception technology is one of the key technologies in assisted driving and autonomous driving systems, and it is also the most basic module. Environmental perception technology perceives and analyzes the acquired visual data, so that the vehicle can understand and identify roads, obstacles, traffic signals, etc.
[0003] Among them, image signal processing of the RAW image output by the sensor is a key link in whether the vehicle environment perception module can obtain high-quality visual data. Summary of the invention
[0004] Generally, the image quality enhancement results corresponding to traditional image signal processing have information distortion and loss problems, which restricts the algorithm performance of downstream perception tasks at the source of the data. In order to solve the above technical problems, the present disclosure provides an image enhancement method, a training method and a device for an image enhancement network, which can solve the problem that traditional image signal processing has information distortion and thus affects downstream perception tasks.
[0005] In a first aspect of the present disclosure, a method for training an image enhancement network is provided, comprising: determining a first image sample to be enhanced and a perceptual label corresponding to the first image sample; processing the first image sample based on an initial brightness curve prediction model to obtain a plurality of predicted brightness curve mapping diagrams corresponding to the first image sample; performing image enhancement processing on the first image sample based on the plurality of predicted brightness curve mapping diagrams to obtain a predicted enhanced image; perceptually processing the predicted enhanced image based on an initial environmental perception model to obtain a perceptual result corresponding to the predicted enhanced image; iteratively training the initial brightness curve prediction model and the initial environmental perception model based on the perceptual result and the perceptual label corresponding to the first image sample to obtain a target brightness curve prediction model; and obtaining a target image enhancement network based on the target brightness curve prediction model.
[0006] A second aspect of the present disclosure provides an image enhancement method, comprising: determining an image to be processed; processing the image to be processed based on a target brightness curve prediction model in a target image enhancement network to obtain a plurality of brightness curve mapping graphs corresponding to the image to be processed; performing image enhancement processing on the image to be processed based on the plurality of brightness curve mapping graphs to obtain an enhanced image; wherein the target image enhancement network is trained based on the training method of the image enhancement network provided in the first aspect above.
[0007] According to a third aspect of the present disclosure, a training device for an image enhancement network is provided, comprising: a sample determination module, used to determine a first image sample to be enhanced, and a perception label corresponding to the first image sample; a processing module, used to process the first image sample based on an initial brightness curve prediction model, and obtain a plurality of predicted brightness curve mapping diagrams corresponding to the first image sample; an enhancement module, used to perform image enhancement processing on the first image sample based on the plurality of predicted brightness curve mapping diagrams, and obtain a predicted enhanced image; a perception module, used to perform perception processing on the predicted enhanced image based on an initial environmental perception model, and obtain a perception result corresponding to the predicted enhanced image; a training module, used to iteratively train the initial brightness curve prediction model and the initial environmental perception model based on the perception result and the perception label corresponding to the first image sample, and obtain a target brightness curve prediction model; and a network determination module, used to obtain a target image enhancement network based on the target brightness curve prediction model.
[0008] A fourth aspect of the present disclosure provides an image enhancement device, comprising: an image determination module, used to determine an image to be processed; a first processing module, used to process the image to be processed based on a target brightness curve prediction model in a target image enhancement network, to obtain a plurality of brightness curve mapping graphs corresponding to the image to be processed; a second processing module, used to perform image enhancement processing on the image to be processed based on the plurality of brightness curve mapping graphs, to obtain an enhanced image; wherein the target image enhancement network is trained based on the training method of the image enhancement network provided in the first aspect above.
[0009] A fifth aspect of the present disclosure provides a computer-readable storage medium storing a computer program for executing the image enhancement network training method provided in the first aspect or the image enhancement method provided in the second aspect.
[0010] The sixth aspect of the present disclosure provides an electronic device, which includes: a processor; a memory for storing processor executable instructions; the processor is used to read the executable instructions from the memory and execute the instructions to implement the image enhancement network training method provided in the first aspect above, or the image enhancement method provided in the second aspect above.
[0011] The seventh aspect of the present disclosure provides a computer program product. When the instructions in the computer program product are executed by a processor, the training method of the image enhancement network provided in the first aspect or the image enhancement method provided in the second aspect is executed.
[0012] Based on the training method of the image enhancement network provided by the present invention, a predicted enhanced image is obtained through a brightness curve prediction model, and the predicted enhanced image is used as the input of an environmental perception model to obtain an output perception result; then, the brightness curve prediction model and the environmental perception model are optimized simultaneously using the perception results and perception labels, so that end-to-end image enhancement can be achieved in a perception-driven manner, achieving the optimal perceived image quality enhancement effect. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 It is a schematic diagram of a scenario applicable to the training method of the image enhancement network provided by an exemplary embodiment of the present disclosure.
[0014] Figure 2 A flowchart of a method for training an image enhancement network provided by an exemplary embodiment of the present disclosure.
[0015] Figure 3 A flowchart of a method for training an image enhancement network provided by another exemplary embodiment of the present disclosure.
[0016] Figure 4 A flowchart of a method for training an image enhancement network provided by yet another exemplary embodiment of the present disclosure.
[0017] Figure 5 A flowchart of a method for training an image enhancement network provided by yet another exemplary embodiment of the present disclosure.
[0018] Figure 6 A flowchart of a method for training an image enhancement network provided by yet another exemplary embodiment of the present disclosure.
[0019] Figure 7 A flowchart of a method for training an image enhancement network provided by yet another exemplary embodiment of the present disclosure.
[0020] Figure 8 A flowchart of an image enhancement method provided by an exemplary embodiment of the present disclosure.
[0021] Fig. 9 A schematic diagram of the structure of a training device for an image enhancement network provided by an exemplary embodiment of the present disclosure.
[0022] Fig.10 A schematic diagram of the structure of a training device for an image enhancement network provided by another exemplary embodiment of the present disclosure.
[0023] Fig.11 A schematic diagram of the structure of a training device for an image enhancement network provided by yet another exemplary embodiment of the present disclosure.
[0024] Fig.12 A schematic diagram of the structure of a training device for an image enhancement network provided by yet another exemplary embodiment of the present disclosure.
[0025] Fig.13 A schematic diagram of the structure of a training device for an image enhancement network provided by yet another exemplary embodiment of the present disclosure.
[0026] Fig.14 A schematic diagram of the structure of an image enhancement device provided by an exemplary embodiment of the present disclosure.
[0027] Fig.15 The present invention provides a schematic structural diagram of an electronic device according to an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0028] To explain the present disclosure, example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, rather than all of the embodiments. It should be understood that the present disclosure is not limited to the example embodiments.
[0029] It should be noted that the relative arrangement of components and steps, the numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present disclosure unless specifically stated otherwise.
[0030] Application Overview
[0031] The target image enhancement network obtained by using the image enhancement network training method provided by the embodiment of the present disclosure can be applied to, for example, intelligent driving scenarios of vehicles and any other feasible scenarios.
[0032] like Figure 1As shown, at least a plurality of image sensors 11 are provided on the vehicle 10, and the image output by the image sensor 11 is the original image without any processing, that is, the RAW domain image. The training method of the image enhancement network provided in the embodiment of the present disclosure is based on the image enhancement performed on the RAW domain image, and the training of the image enhancement network. After the training of the image enhancement network is completed, the trained image enhancement network can be used to perform image enhancement processing on the image to be processed to obtain an enhanced image. Furthermore, the enhanced image can be used as input data for vehicle environment perception, and it can be input into the vehicle environment perception model to obtain the corresponding environment perception result. It should be noted that Figure 1 This is only an example, and the embodiments of the present disclosure are Figure 1 There is no limit to the number and position of the multiple image sensors 11. In actual use, the image sensor 11 may be an image sensor corresponding to the vehicle's forward wide-angle camera, the image sensor 11 may be an image sensor corresponding to the vehicle's forward narrow-angle camera, the image sensor 11 may also be an image sensor corresponding to a side view camera, etc.
[0033] Generally, there are three methods for image sensor signal processing: the first is the image signal processing method of traditional vehicle-mounted cameras. The image quality enhancement results obtained by this method have information distortion and loss problems, which restricts the algorithm performance of downstream perception tasks at the source of the data. The second is the differentiable image signal processing method. This method optimizes the tuning cost by determining multiple sets of offline hyperparameters, but this method still relies on the traditional image processing link framework, and the number of offline hyperparameters determined by this method cannot meet the differentiated needs of massive intelligent driving data. The third is a supervised image enhancement method, which requires the collection of large-scale paired data, has a high data collection cost, and consumes a lot of data storage resources.
[0034] In order to solve the above problems, the embodiments of the present disclosure provide a training method for an image enhancement network, in which a predicted enhanced image is obtained through a brightness curve prediction model, and the predicted enhanced image is used as the input of an environmental perception model to obtain an output perception result; then, the brightness curve prediction model and the environmental perception model are optimized simultaneously using the perception results and perception labels, so that end-to-end image enhancement can be achieved in a perception-driven manner, thereby achieving the optimal perceived image quality enhancement effect.
[0035] Exemplary Methods
[0036] Figure 2 A flowchart of a training method for an image enhancement network provided in an embodiment of the present disclosure is provided. This embodiment can be applied to electronic devices, such as servers, such as Figure 2 As shown, the method includes the following steps S201-S206.
[0037] Step S201: Determine a first image sample to be enhanced and a perceptual label corresponding to the first image sample.
[0038] For example, Figure 1 As shown, the RAW domain image sample collected by the image sensor 11 can be obtained first, and then the RAW domain image sample can be preprocessed, for example, the RAW domain image sample can be de-mosaiced, and the preprocessed image sample is a three-channel image, that is, an RGB image. Then, the RGB image can be used as the first image sample to be enhanced.
[0039] Exemplarily, if the downstream perception task is target object detection, such as pedestrian detection, other vehicle detection, lane detection, etc., then the perception label corresponding to the first image sample is the object detection result, such as a real pedestrian detection frame, a real lane detection frame, etc. For another example, if the downstream perception task is target object recognition, such as pedestrian recognition, vehicle recognition, etc., then the perception label corresponding to the first image sample is the object recognition result, such as a real vehicle category label, a real pedestrian category label. For another example, if the downstream perception task is scene depth recognition, then the perception label corresponding to the first image sample is a real depth map.
[0040] It should be noted that the perception label corresponding to the first image sample in the embodiment of the present disclosure can be determined according to the specific type of the downstream environmental perception task in actual application, and the embodiment of the present disclosure does not limit this.
[0041] Step S202: Process the first image sample based on the initial brightness curve prediction model to obtain a plurality of predicted brightness curve mapping graphs corresponding to the first image sample.
[0042] Exemplarily, the initial brightness curve prediction model is an untrained neural network model for predicting brightness curve mapping graphs. After the first image sample to be enhanced is input into the initial brightness curve prediction model, multiple predicted brightness curve mapping graphs corresponding to the first image sample can be output. Exemplarily, the initial brightness curve prediction model can include a feature extraction layer and a nonlinear transformation layer: the feature extraction layer is used to extract features from the first image sample to obtain multiple initial feature graphs; and the nonlinear transformation layer is used to perform nonlinear transformation on the multiple initial feature graphs to obtain predicted brightness curve mapping graphs corresponding to each initial feature graph.
[0043] Exemplarily, the predicted brightness curve mapping diagram is used to map the pixel brightness values in the first range to the second range, thereby achieving brightness value enhancement. For example, the pixel brightness value of [0,1] is mapped to the range [-1,1] to stretch the brightness of the pixel. In some examples, if the size of the first image sample or the initial feature map is W*H, and the predicted brightness curve mapping diagram is used to map the brightness of each pixel to the range of [-1,1], then the predicted brightness curve mapping diagram is a mapping diagram with a size of W*H and brightness values between -1 and 1.
[0044] Furthermore, in the embodiment of the present disclosure, the predicted brightness curve mapping diagram obtained by the initial brightness curve prediction model can ensure that the brightness stretching meets the requirement of monotony and no overflow.
[0045] Step S203: performing image enhancement processing on the first image sample based on the multiple predicted brightness curve mappings to obtain a predicted enhanced image.
[0046] Exemplarily, the first image sample can be converted from the first color space to the second color space to obtain the second image sample, and then the original brightness component of the second image sample can be enhanced based on the predicted brightness curve mapping map corresponding to each preset channel in multiple predicted brightness curve mapping maps to obtain a predicted enhanced brightness component, and finally, based on the predicted enhanced brightness component and the first image sample, a predicted enhanced image can be obtained.
[0047] Step S204: Perform perception processing on the predicted enhanced image based on the initial environment perception model to obtain a perception result corresponding to the predicted enhanced image.
[0048] Exemplarily, the initial environmental perception model is an untrained neural network model for environmental perception, and after the predicted enhanced image is input into the initial environmental perception model, the perception result corresponding to the predicted enhanced image can be obtained. For example, the initial environmental perception model is an initial pedestrian detection model, and after the predicted enhanced image is input into the initial pedestrian detection model, the pedestrian detection result in the predicted enhanced image can be obtained. For another example, the initial environmental perception model is an initial object classification model, and after the predicted enhanced image is input into the initial object classification model, the type of each object in the predicted enhanced image can be obtained. For another example, the initial environmental perception model is an initial scene depth estimation model, and after the predicted enhanced image is input into the initial scene depth estimation model, the predicted depth map corresponding to the predicted enhanced image can be obtained.
[0049] It should be noted that the type of the initial environment perception model and the specific implementation method of the initial environment perception model in the embodiments of the present disclosure can be determined according to actual applications, and the embodiments of the present disclosure do not limit this.
[0050] Step S205: based on the perception result and the perception label corresponding to the first image sample, iteratively train the initial brightness curve prediction model and the initial environment perception model to obtain a target brightness curve prediction model.
[0051] For example, the loss of the initial environment perception model can be determined based on the perception result output by the initial environment perception model and the perception label corresponding to the first image sample, and then the loss is used to iteratively train the initial brightness curve prediction model and the initial environment perception model at the same time to obtain the target brightness curve prediction model. In this way, the brightness curve prediction model in the image enhancement network can be optimized through the loss function of the perception task, thereby realizing perception-guided image enhancement and achieving the purpose of improving the image enhancement effect.
[0052] Step S206: Obtain a target image enhancement network based on the target brightness curve prediction model.
[0053] For example, the target image enhancement network may be composed of a target brightness curve prediction model and a brightness enhancement algorithm; the input of the target image enhancement network is the image to be processed, and the output of the target image enhancement network is the enhanced image. For example, the image to be processed is input into the target brightness curve prediction model to obtain a plurality of brightness curve mapping graphs; then, the brightness enhancement algorithm is used to perform brightness enhancement processing on the plurality of brightness curve mapping graphs and the image to be processed to obtain an enhanced image.
[0054] It should be noted that after the iterative training is completed in the above step S205, the target environment perception model will also be obtained. Then, in the subsequent application process, the enhanced image can be applied to the target environment perception model, and the enhanced image can also be applied to other trained environment perception models. The present disclosed embodiment does not limit this.
[0055] The training method of the image enhancement network provided in the embodiment of the present disclosure obtains a predicted enhanced image through a brightness curve prediction model, and uses the predicted enhanced image as the input of an environmental perception model to obtain an output perception result; then, the brightness curve prediction model and the environmental perception model are simultaneously optimized using the perception result and the perception label, so that end-to-end image enhancement can be achieved in a perception-driven manner, achieving the optimal perceived image quality enhancement effect.
[0056] like Figure 3 As shown in the above Figure 2 Based on the illustrated embodiment, step S205 may include the following steps S2051 and S2052.
[0057] Step S2051: Determine a loss value of the initial environment perception model based on the perception result and the perception label corresponding to the first image sample.
[0058] Exemplarily, the predicted enhanced image can be input into the initial environment perception model to obtain the perception result, and then the loss of the initial environment perception model is determined based on the perception result, the perception label and the loss function of the initial environment perception model. Then, the initial brightness curve prediction model and the initial environment perception model are end-to-end optimized by the loss of the initial environment perception model. Among them, the embodiment of the present disclosure does not limit the specific implementation method of the loss function of the initial environment perception model.
[0059] In some examples, the loss value Loss of the initial environment perception model can be determined by the following formula (1): cv :
[0060] Loss cv =loss cv (g cv (f isp (I rgb )),gt) Formula (1);
[0061] Among them, I rgb is the first image sample, f isp (I rgb ) is the predicted enhanced image, g cv (·) is the initial environment perception model, g cv (f isp (I rgb )) is the perception result output by the initial environment perception model, loss cv is the loss function of the initial environment perception model, and gt is the perception label corresponding to the first image sample, that is, the true value of the downstream perception task.
[0062] Step S2052: Iteratively update the initial environment perception model and the initial brightness curve prediction model using the loss value to obtain a target brightness curve prediction model.
[0063] Exemplarily, the loss value of the initial environment perception model can be used to iteratively update the initial environment perception model and the initial brightness curve prediction model at the same time, and the target brightness curve prediction model is obtained after the iteration is completed.
[0064] The training method of the image enhancement network provided by the embodiment of the present disclosure obtains a predicted enhanced image through an initial brightness curve prediction model, and then uses the predicted enhanced image as the input of the environment perception model to obtain the output perception result. Furthermore, the model loss is determined using the perception result and the perception label, and the cascaded initial brightness curve prediction model and the initial environment perception model are iteratively updated simultaneously using the loss, so as to provide a purely perception-oriented image enhancement solution, thereby achieving the optimal perception quality enhancement effect.
[0065] like Figure 4 As shown in the above Figure 3 Based on the illustrated embodiment, step S2052 may include the following steps S21 to S23.
[0066] Step S21, using the loss value to iteratively update the weights of the initial environment perception model, and back-propagating the loss value to the initial brightness curve prediction model during the iterative update process.
[0067] Exemplarily, the initial brightness curve prediction model and the initial environment perception model can be cascaded in the initialization stage, that is, the predicted enhanced image obtained based on the initial brightness curve prediction model is used as the input of the initial environment perception model. Then, the weights of the two models are initialized using the Xavier initialization method and the He initialization method respectively. Of course, other initialization methods can also be used to initialize the weights of the two models, and the present disclosure does not limit this. Furthermore, in the model training stage, the weights of the initial environment perception model can be iteratively updated using the loss value of the initial environment perception model, and the loss value is back-propagated to the initial brightness curve prediction model during the iterative update process to update the weights of the initial brightness curve prediction model. For example, if the environment perception model is an object classification model, the gradient is back-propagated from the classifier of the object classification model to the brightness curve prediction model; wherein the gradient is the rate of change of the loss function relative to the model parameters, and in the classification task, the gradient can be used to update the parameters of the classification model, so that the loss value is continuously reduced.
[0068] Step S22: Iteratively update the weights of the initial brightness curve prediction model using the loss value.
[0069] Exemplarily, the weights of the initial brightness curve prediction model refer to the values of each neural network parameter in the initial brightness curve prediction model, and the loss value obtained by back propagation can be used to iteratively update the parameter values in the initial brightness curve prediction model to adjust the behavior of the initial brightness curve prediction model.
[0070] Step S23: In response to the updated environment perception model and the updated brightness curve prediction model both satisfying the convergence condition, a target brightness curve prediction model is obtained.
[0071] Exemplarily, the updated environmental perception model and the updated brightness curve prediction model both meet the convergence conditions, which can be achieved in the following three ways: the first is that the loss of the output of the updated environmental perception model is less than a preset value. The second is that the change in the weight of the environmental perception model between two iterations is less than the first preset value and / or the change in the weight of the brightness curve prediction model is less than the second preset value, wherein the first preset value and the second preset value may be the same or different. The third is that the number of iterations reaches a preset number. When the updated environmental perception model and the updated brightness curve prediction model both meet the convergence conditions, a target brightness curve prediction model is obtained, which is the trained brightness curve prediction model.
[0072] The training method of the image enhancement network provided in the embodiment of the present disclosure realizes end-to-end image enhancement in a perception-driven manner by back-propagating the loss value of the environmental perception model to the brightness curve prediction model during the iterative update process. Compared with the image enhancement method in the related art, this method does not require additional manpower and computing storage resources, and is a picture quality enhancement solution designed completely for perception tasks, thereby achieving better image enhancement effects.
[0073] like Figure 5 As shown in the above Figure 2 Based on the illustrated embodiment, step S203 may include the following steps S2031a to S2032a.
[0074] Step S2031a: perform defogging processing on the first image sample based on the initial first defogging model to obtain a defogged first image sample.
[0075] Exemplarily, after determining the first image sample to be enhanced, the first image sample can be defogged based on the initial first defogging model to obtain the first image sample after defogging. Among them, the first defogging model can be a defogging model corresponding to a dark channel prior algorithm. The principle of the dark channel prior algorithm is that there is at least one area in an image with very low pixel values. This area is called a dark channel. The dark channel prior algorithm then uses this principle to remove haze from the image by analyzing the dark channel. Of course, the first defogging model in the embodiment of the present disclosure can also be a defogging model corresponding to other defogging algorithms, and the embodiment of the present disclosure does not limit this.
[0076] In some embodiments, the first image sample after defogging can be obtained based on the following formula (2):
[0077]
[0078] Among them, I rgb is the first image sample to be enhanced, PreDehaze(·) is the first dehazing model, is the first image sample after dehazing.
[0079] In some embodiments, based on the perception results and the perception labels corresponding to the first image samples, the initial brightness curve prediction model, the initial environment perception model and the initial first defogging model can be iteratively trained to obtain the target brightness curve prediction model and the target first defogging model; further, based on the target brightness curve prediction model and the target first defogging model, a target image enhancement network is obtained.
[0080] It can be understood that the specific implementation method of obtaining the target image enhancement network based on the target brightness curve prediction model and the target first defogging model can refer to the implementation method of the aforementioned steps S205-S206, which will not be repeated here.
[0081] In some embodiments, based on the perception results and the perception labels corresponding to the first image samples, the initial brightness curve prediction model, the initial environment perception model and the initial first defogging model are iteratively trained to obtain a target brightness curve prediction model and a target first defogging model, including: determining a loss value of the initial environment perception model based on the perception results and the perception labels corresponding to the first image samples; and iteratively updating the initial environment perception model, the initial brightness curve prediction model and the initial first defogging model using the loss value to obtain a target brightness curve prediction model and a target first defogging model.
[0082] It can be understood that the specific implementation method of obtaining the target brightness curve prediction model and the target first defogging model can refer to the implementation method of the aforementioned steps S2051 and S2052, which will not be repeated here.
[0083] In some embodiments, the loss value is used to iteratively update the initial environment perception model, the initial brightness curve prediction model and the initial first defogging model to obtain the target brightness curve prediction model and the target first defogging model, including: iteratively updating the weights of the initial environment perception model using the loss value, and back-propagating the loss value to the initial brightness curve prediction model and the initial first defogging model during the iterative update process; iteratively updating the weights of the initial brightness curve prediction model and the weights of the initial first defogging model using the loss value; in response to the updated environment perception model, the updated brightness curve prediction model and the updated first defogging model all satisfying the convergence conditions, obtaining the target brightness curve prediction model and the target first defogging model.
[0084] For example, during the training process of the image enhancement network, it is necessary to continuously adjust the weights of the first defogging model so that the first defogging model converges. The weights of the first defogging model include but are not limited to the value of the defogging penalty factor.
[0085] It can be understood that the specific implementation method of obtaining the target brightness curve prediction model and the target first defogging model can refer to the implementation method of the aforementioned steps S21-S23, which will not be repeated here.
[0086] Step S2032a: performing image enhancement processing on the first image sample after defogging based on a plurality of predicted brightness curve mappings to obtain a predicted enhanced image.
[0087] For example, the first image sample after defogging can be converted from the first color space to the second color space to obtain the second image sample after defogging, and then the original brightness component of the second image sample after defogging is subjected to image enhancement processing based on the predicted brightness curve mapping map corresponding to each preset channel in the plurality of predicted brightness curve mapping maps to obtain the predicted enhanced brightness component, and finally the predicted enhanced image is obtained based on the predicted enhanced brightness component and the first image sample after defogging. For example, the predicted enhanced brightness component, the hue component in the first image sample after defogging, and the saturation component in the first image sample after defogging are recombined to obtain the predicted enhanced image.
[0088] The training method of the image enhancement network provided in the embodiment of the present disclosure first performs defogging on the first image sample and then performs image enhancement on the defogging first image sample to obtain a predicted enhanced image. Since the defogging is performed on the original RAW image, the defogging image information is more complete and the clarity of the predicted enhanced image is also improved.
[0089] like Figure 6 As shown in the above Figure 2 Based on the illustrated embodiment, step S203 may include the following steps S2031b to S2033b.
[0090] Step S2031b: performing defogging processing on the first image sample based on the initial first defogging model to obtain a defogged first image sample.
[0091] It can be understood that the specific implementation method of performing defogging processing on the first image sample based on the initial first defogging model to obtain the defogged first image sample can refer to the implementation method of the aforementioned step S2031a, which will not be repeated here.
[0092] Step S2032b: performing image enhancement processing on the first image sample after defogging based on multiple predicted brightness curve mappings to obtain a predicted intermediate image.
[0093] Exemplarily, the first image sample after dehazing can be converted from the first color space to the second color space to obtain the second image sample after dehazing, and then the original brightness component of the second image sample after dehazing is subjected to image enhancement processing based on the predicted brightness curve mapping map corresponding to each preset channel in multiple predicted brightness curve mapping maps to obtain the predicted enhanced brightness component, and finally, based on the predicted enhanced brightness component and the first image sample after dehazing, a predicted intermediate image is obtained.
[0094] Step S2033b: Defogging the predicted intermediate image based on the initial second defogging model to obtain a predicted enhanced image.
[0095] Exemplarily, after determining the predicted intermediate image, the predicted intermediate image can be defogged based on the initial second defogging model to obtain a predicted enhanced image. The second defogging model can be a defogging model corresponding to a dark channel prior algorithm. Of course, the second defogging model in the present disclosure can also be a defogging model corresponding to other defogging algorithms, and the embodiments of the present disclosure are not limited to this. In addition, the implementation method of the first defogging model and the implementation method of the second defogging model can be the same or different.
[0096] In some embodiments, the predicted enhanced image can be obtained based on the above formula (2). In this case, I rgb To predict the intermediate image, PreDehaze(·) is the second dehazing model. To predict strong images.
[0097] In some embodiments, based on the perception results and the perception labels corresponding to the first image samples, the initial brightness curve prediction model, the initial environment perception model, the initial first defogging model and the initial second defogging model can be iteratively trained to obtain the target brightness curve prediction model, the target first defogging model and the target second defogging model; further, based on the target brightness curve prediction model, the target first defogging model and the target second defogging model, a target image enhancement network is obtained.
[0098] It can be understood that the specific implementation method of obtaining the target image enhancement network based on the target brightness curve prediction model, the target first defogging model and the target second defogging model can refer to the implementation method of the aforementioned step S206, which will not be repeated here.
[0099] In some embodiments, based on the perception results and the perception labels corresponding to the first image samples, the initial brightness curve prediction model, the initial environment perception model, the initial first defogging model and the initial second defogging model are iteratively trained to obtain the target brightness curve prediction model, the target first defogging model and the target second defogging model, including: determining the loss value of the initial environment perception model based on the perception results and the perception labels corresponding to the first image samples; and iteratively updating the initial environment perception model, the initial brightness curve prediction model, the initial first defogging model and the initial second defogging model using the loss value to obtain the target brightness curve prediction model, the target first defogging model and the target second defogging model.
[0100] In some embodiments, the loss value is used to iteratively update the initial environment perception model, the initial brightness curve prediction model, the initial first defogging model and the initial second defogging model to obtain the target brightness curve prediction model, the target first defogging model and the target second defogging model, including: iteratively updating the weights of the initial environment perception model using the loss value, and back-propagating the loss value to the initial brightness curve prediction model, the initial first defogging model and the initial second defogging model during the iterative update process; iteratively updating the weights of the initial brightness curve prediction model, the initial first defogging model and the initial second defogging model using the loss value; in response to the updated environment perception model, the updated brightness curve prediction model, the updated first defogging model and the updated second defogging model all satisfying the convergence conditions, obtaining the target brightness curve prediction model, the target first defogging model and the target second defogging model.
[0101] Exemplarily, during the training process of the image enhancement network, it is necessary to continuously adjust the weights of the first defogging model and the weights of the second defogging model so that the first defogging model and the second defogging model converge. The weights of the first defogging model and the second defogging model include but are not limited to the value of the defogging penalty factor. In some examples, if the first defogging model and the second defogging model are implemented in the same way, the defogging granularity of the first defogging model may be greater than the defogging granularity of the second defogging model; wherein the defogging granularity can be limited to a preset range and then fine-tuned, and the defogging granularity includes but is not limited to a filter kernel and a defogging penalty factor. For example, the defogging penalty factor of the first defogging model may be 0.5, and the penalty factor of the second defogging model may be 0.7.
[0102] The training method of the image enhancement network provided by the embodiment of the present disclosure first performs a first defogging process on the first image sample, then performs an image enhancement process on the defogging first image sample to obtain a predicted intermediate image, and finally performs a second defogging process on the predicted intermediate image to obtain a predicted enhanced image. In this way, the defogging module can be split into two stages: RAW domain pre-defogging and defogging post-processing, thereby greatly reducing the complexity of image defogging driven by perception, and ensuring that the enhanced image under perception guidance not only has the advantage of information integrity, but also has better picture quality clarity than traditional image signal processing.
[0103] In some embodiments, the first image sample is processed based on the initial brightness curve prediction model to obtain multiple predicted brightness curve mapping diagrams corresponding to the first image sample, including: performing feature extraction on the first image sample based on the feature extraction layer in the initial brightness curve prediction model to obtain image features of the first image sample in multiple preset channels; performing nonlinear transformation on the image features of multiple preset channels based on the activation function layer in the initial brightness curve prediction model to obtain predicted brightness curve mapping diagrams corresponding to each of the multiple preset channels.
[0104] Exemplarily, the initial brightness curve prediction model includes at least a feature extraction layer, which can be used to extract features from the first image sample to obtain image features of the first image sample in multiple preset channels. It should be noted that the embodiment of the present disclosure does not limit the specific method of feature extraction. For example, a convolutional neural network (CNN) can be used to extract features from the first image sample.
[0105] For example, the convolution kernel size of the feature extraction layer can be set to 3*3, and the first image sample is first downsampled by the convolution kernel to obtain the downsampled features. The downsampled features are then upsampled to obtain the final image features. Among them, the number of channels corresponding to the multiple preset channels is a hyperparameter in the initial brightness curve prediction model, and its value is related to the image sensor module. In the actual application process, the number of channels corresponding to the multiple preset channels can be set based on the relevant characteristics of the sensor module and the empirical value. The embodiments of the present disclosure do not limit this. The following embodiments take the number of channels of the multiple preset channels as 12 as an example for illustrative explanation.
[0106] Exemplarily, the initial brightness curve prediction model includes at least an activation function layer, and the activation function in the activation function layer can be used to map the image features of multiple preset channels to the corresponding interval range, and obtain the predicted brightness curve mapping diagrams corresponding to the multiple preset channels. For example, in order to ensure that the brightness stretching meets the requirement of monotony and no overflow, the tanh (·) activation function can be used to map the image features of multiple preset channels to the interval [-1, 1], and generate the predicted brightness curve stretching diagram C corresponding to each pixel of the channel number k i ,i=1,…,k;wherein, k is the number of channels of the plurality of preset channels. The predicted brightness curve stretching graph C i It is a multi-segment quadratic monotonic curve stretching diagram of the image in the [0, 1] interval. This can ensure the stability of brightness stretching and prevent pixel value overexposure.
[0107] The disclosed embodiment extracts features from a first image sample, performs a nonlinear transformation on the extracted image features, obtains a corresponding predicted brightness curve mapping map, and performs image enhancement on the first image sample based on the predicted brightness curve mapping map, thereby enhancing image clarity by enhancing image brightness. At the same time, since the predicted brightness curve mapping map is obtained by performing a nonlinear transformation on image features, using the predicted brightness curve mapping map for brightness enhancement has a good enhancement effect.
[0108] like Figure 7 As shown in the above Figure 2 Based on the illustrated embodiment, step S203 may include the following steps S2031c to S2033c.
[0109] Step S2031c: convert the first image sample from the first color space to the second color space to obtain a second image sample.
[0110] Exemplarily, the first image sample may be converted from the RGB color space to the HSV color space to obtain a second image sample, where the second image sample is an image of the first image sample in the HSV color space.
[0111] Step S2032c: performing image enhancement processing on the original brightness component of the second image sample based on the predicted brightness curve mapping maps corresponding to each preset channel in the plurality of predicted brightness curve mapping maps to obtain a predicted enhanced brightness component.
[0112] Exemplarily, after converting the first image sample from the RGB color space to the HSV color space, a second image sample is obtained, and the second image sample consists of a hue H component, a saturation S component, and a brightness V component. Then, the brightness V component can be image enhanced using the predicted brightness curve mappings corresponding to each preset channel in the multiple predicted brightness curve mappings to obtain a predicted enhanced brightness component. For example, the predicted brightness curve mappings corresponding to the multiple preset channels include predicted brightness curve mappings corresponding to 12 preset channels, which are respectively the predicted brightness curve mappings corresponding to the first preset channel to the predicted brightness curve mappings corresponding to the twelfth preset channel. The brightness component V can be firstly enhanced using the predicted brightness curve mapping corresponding to the first preset channel to obtain a first predicted enhancement result; then the first predicted enhancement result is enhanced using the predicted brightness curve mapping corresponding to the second preset channel to obtain a second predicted enhancement result, and so on, and finally the eleventh predicted enhancement result is enhanced using the predicted brightness curve mapping corresponding to the twelfth preset channel to obtain the final predicted enhanced brightness component.
[0113] Step S2033c: obtain a predicted enhanced image based on the predicted enhanced brightness component and the first image sample.
[0114] Exemplarily, after performing image enhancement processing on the second image sample to obtain a predicted enhanced brightness component, the predicted enhanced brightness component and the hue component and saturation component corresponding to the first image sample in the HSV color space can be recombined to obtain a predicted enhanced image in the HSV color space, and finally the predicted enhanced image in the HSV color space is converted to the RGB color space to obtain the final predicted enhanced image.
[0115] The training method of the image enhancement network provided by the embodiment of the present disclosure processes the original brightness component of the first image sample in the second color space through the predicted brightness curve mapping diagram corresponding to each preset channel in multiple preset channels to obtain an enhanced brightness component, thereby obtaining a predicted enhanced image based on the enhanced brightness component. In this way, the image clarity can be enhanced by enhancing the image brightness; since other image information except the image brightness does not change during the image enhancement process, the image clarity can be improved while maintaining the image information intact.
[0116] In some embodiments, image enhancement processing is performed on the original brightness component of the second image sample based on the predicted brightness curve mapping map corresponding to each preset channel in multiple predicted brightness curve mapping maps to obtain the predicted enhanced brightness component, including: performing pixel-by-pixel brightness stretching processing on the original brightness component of the second image sample based on the predicted brightness curve mapping map corresponding to the first preset channel in the multiple predicted brightness curve mapping maps to obtain the predicted brightness component corresponding to the first preset channel; performing pixel-by-pixel brightness stretching processing on the predicted brightness component corresponding to the previous preset channel of each second preset channel based on the predicted brightness curve mapping map corresponding to each second preset channel in at least one second preset channel in the multiple predicted brightness curve mapping maps to obtain the predicted enhanced brightness component; wherein the predicted brightness component corresponding to the previous preset channel of the second preset channel includes at least the predicted brightness component corresponding to the first preset channel.
[0117] For example, the plurality of predicted brightness curve mapping graphs include predicted brightness curve mapping graphs corresponding to 12 preset channels, which are respectively predicted brightness curve mapping graph C1 corresponding to a first preset channel and predicted brightness curve mapping graphs C2 to C3 corresponding to 11 second preset channels. 12 . Then, based on the predicted brightness curve mapping map C1 corresponding to the first preset channel, the original brightness component V0 of the second image sample is subjected to pixel-by-pixel brightness stretching processing to obtain the predicted brightness component V1 corresponding to the first preset channel. Further, based on the predicted brightness curve mapping map C2 corresponding to the second preset channel, the predicted brightness component V1 corresponding to the first preset channel is subjected to pixel-by-pixel brightness stretching processing to obtain the predicted brightness component V2 corresponding to the second preset channel; then, based on the predicted brightness curve mapping map C3 corresponding to the second preset channel, the predicted brightness component V2 corresponding to the second preset channel is subjected to pixel-by-pixel brightness stretching processing to obtain the predicted brightness component V3 corresponding to the second preset channel; then, based on the predicted brightness curve mapping map C4 corresponding to the second preset channel, the predicted brightness component V3 corresponding to the second preset channel is subjected to pixel-by-pixel brightness stretching processing to obtain the predicted brightness component V4 corresponding to the second preset channel, and so on. Finally, based on the predicted brightness curve mapping map C 12 The predicted brightness component V corresponding to the second preset channel 11 Perform pixel-by-pixel brightness stretching to obtain the predicted enhanced brightness component V 12 .
[0118] In some examples, the predicted brightness curve map C corresponding to each preset channel can be used i , i=1,…,k performs k rounds of pixel-by-pixel curve stretching on the original brightness component V0 in the HSV color space to obtain the predicted brightness component V with enhanced clarity k , the predicted enhanced brightness component V can be determined by the following formula (3): k :
[0119] V i =V i-1 +C i *V i-1 *(V i-1 -1), i=1,…,k Formula (3);
[0120] Wherein, k is the number of channels of the plurality of preset channels.
[0121] The disclosed embodiment performs multiple rounds of pixel-by-pixel curve stretching on the original brightness component corresponding to the first image sample in the second color space through the predicted brightness curve mapping diagram corresponding to each preset channel, so as to obtain the enhanced brightness component corresponding to the first image sample. Since multiple rounds of pixel-level curve stretching are performed, a better image clarity enhancement effect can be obtained.
[0122] Figure 8 The present invention provides a flowchart of an image enhancement method. This embodiment can be applied to electronic devices, such as a computing unit in a vehicle, such as Figure 8 As shown, the method includes the following steps S801-S803.
[0123] Step S801: Determine an image to be processed.
[0124] For example, Figure 1 As shown, the RAW domain image captured by the image sensor 11 can be obtained first, and then the RAW domain image can be preprocessed, for example, the RAW domain image can be de-mosaiced, and the preprocessed image is a three-channel image, that is, an RGB image. Then, the RGB image can be used as the image to be processed.
[0125] Step S802: Process the image to be processed based on the target brightness curve prediction model in the target image enhancement network to obtain multiple brightness curve mapping graphs corresponding to the image to be processed.
[0126] It can be understood that the specific implementation method of obtaining the multiple brightness curve mapping diagrams corresponding to the image to be processed can refer to the implementation method of the aforementioned step S202, which will not be repeated here.
[0127] Step S803, performing image enhancement processing on the image to be processed based on the multiple brightness curve mapping graphs to obtain an enhanced image;
[0128] Wherein, the target image enhancement network is trained based on the above-mentioned training method.
[0129] It can be understood that the specific implementation method of performing image enhancement processing on the image to be processed based on multiple brightness curve mapping graphs to obtain the enhanced image can refer to the implementation method of the aforementioned step S203, which will not be repeated here.
[0130] It should be noted that after the iterative training is completed in the above step S205, the target environment perception model will also be obtained. Then, during the application process, the enhanced image obtained in step S803 can be applied to the target environment perception model, and the enhanced image obtained in step S803 can also be applied to other trained environment perception models. The present disclosed embodiment does not limit this.
[0131] The image enhancement method provided by the embodiment of the present disclosure performs image enhancement processing on the processed image by using multiple brightness curve mapping diagrams obtained by the target brightness curve prediction model in the target image enhancement network to obtain an enhanced image; wherein, since the target image enhancement network optimizes the brightness curve prediction model and the environmental perception model simultaneously using the perception results and the perception labels during the training process, the target image enhancement network can realize end-to-end image enhancement in a perception-driven manner to achieve the optimal perceived image quality enhancement effect. At the same time, since other image information except the image brightness does not change during the image enhancement process, the clarity of the image can be improved while maintaining the image information intact.
[0132] In some embodiments, image enhancement processing is performed on the image to be processed based on multiple brightness curve mapping maps to obtain an enhanced image, including: defogging the image to be processed based on a target first defogging model to obtain a defogged image to be processed; image enhancement processing is performed on the defogged image to be processed based on multiple brightness curve mapping maps to obtain an enhanced image.
[0133] It can be understood that the specific implementation method of performing image enhancement processing on the defogged image to be processed based on multiple brightness curve mapping graphs to obtain the enhanced image can refer to the implementation method of the aforementioned steps S2031a to S2032a, which will not be repeated here.
[0134] In some embodiments, image enhancement processing is performed on the image to be processed based on multiple brightness curve mapping maps to obtain an enhanced image, including: defogging the image to be processed based on a target first defogging model to obtain a defogged image to be processed; image enhancement processing is performed on the defogged image to be processed based on multiple brightness curve mapping maps to obtain an intermediate image; defogging the intermediate image based on a target second defogging model to obtain an enhanced image.
[0135] It can be understood that the specific implementation method of performing defogging processing on the intermediate image based on the target second defogging model to obtain the enhanced image can refer to the implementation method of the aforementioned steps S2031b to S2033b, which will not be repeated here.
[0136] In some embodiments, a target brightness curve prediction model in a target image enhancement network is used to process an image to be processed to obtain multiple brightness curve mapping diagrams corresponding to the image to be processed, including: extracting features of the image to be processed based on a feature extraction layer in the target brightness curve prediction model to obtain image features of the image to be processed in multiple preset channels; performing nonlinear transformation on the image features of multiple preset channels based on an activation function layer in the target brightness curve prediction model to obtain brightness curve mapping diagrams corresponding to each of the multiple preset channels.
[0137] It can be understood that the specific implementation methods of performing feature extraction on the processed image based on the feature extraction layer in the target brightness curve prediction model, and performing nonlinear transformation on the image features of multiple preset channels based on the activation function layer in the target brightness curve prediction model can refer to the aforementioned implementation methods of performing feature extraction on the first image sample based on the feature extraction layer in the initial brightness curve prediction model, and performing nonlinear transformation on the image features of multiple preset channels based on the activation function layer in the initial brightness curve prediction model, and will not be repeated here.
[0138] In some embodiments, image enhancement processing is performed on an image to be processed based on multiple brightness curve mapping maps to obtain an enhanced image, including: converting the image to be processed from a first color space to a second color space to obtain a converted image; performing image enhancement processing on the original brightness component of the converted image based on the brightness curve mapping maps corresponding to each preset channel in the multiple brightness curve mapping maps to obtain an enhanced brightness component; and obtaining an enhanced image based on the enhanced brightness component and the image to be processed.
[0139] It can be understood that the specific implementation method of obtaining the enhanced image based on the enhanced brightness component and the image to be processed can refer to the implementation method of the aforementioned steps S2031c to S2033c, which will not be repeated here.
[0140] In some embodiments, image enhancement processing is performed on the original brightness component of the converted image based on the brightness curve mapping map corresponding to each preset channel in multiple brightness curve mapping maps to obtain an enhanced brightness component, including: performing pixel-by-pixel brightness stretching processing on the original brightness component of the converted image based on the brightness curve mapping map corresponding to the first preset channel in the multiple brightness curve mapping maps to obtain the brightness component corresponding to the first preset channel; performing pixel-by-pixel brightness stretching processing on the brightness component corresponding to the previous preset channel of each second preset channel based on the brightness curve mapping map corresponding to each second preset channel in at least one second preset channel in the multiple brightness curve mapping maps to obtain an enhanced brightness component; wherein the brightness component corresponding to the previous preset channel of the second preset channel includes at least the brightness component corresponding to the first preset channel.
[0141] It can be understood that the specific implementation methods of obtaining the brightness component corresponding to the first preset channel and obtaining the enhanced brightness component can refer to the aforementioned implementation methods of obtaining the predicted brightness component corresponding to the first preset channel and obtaining the predicted enhanced brightness component, respectively, and will not be repeated here.
[0142] Exemplary Devices
[0143] Fig. 9 A training device for an image enhancement network provided in an embodiment of the present disclosure, such as Fig. 9 As shown, the image enhancement network training device 90 includes a sample determination module 91, a processing module 92, an enhancement module 93, a perception module 94, a training module 95 and a network determination module 96.
[0144] A sample determination module 91, used to determine a first image sample to be enhanced and a perceptual label corresponding to the first image sample;
[0145] A processing module 92, configured to process the first image sample based on the initial brightness curve prediction model to obtain a plurality of predicted brightness curve mapping graphs corresponding to the first image sample;
[0146] An enhancement module 93, configured to perform image enhancement processing on the first image sample based on a plurality of predicted brightness curve mapping graphs to obtain a predicted enhanced image;
[0147] A perception module 94 is used to perform perception processing on the predicted enhanced image based on the initial environment perception model to obtain a perception result corresponding to the predicted enhanced image;
[0148] A training module 95, configured to iteratively train an initial brightness curve prediction model and an initial environment perception model based on the perception result and the perception label corresponding to the first image sample to obtain a target brightness curve prediction model;
[0149] The network determination module 96 is used to obtain a target image enhancement network based on a target brightness curve prediction model.
[0150] In some embodiments, Fig.10 As shown, the training module 95 includes a loss determination unit 951 and a model updating unit 952.
[0151] A loss determination unit 951, configured to determine a loss value of the initial environment perception model based on the perception result and the perception label corresponding to the first image sample;
[0152] The model updating unit 952 is used to iteratively update the initial environment perception model and the initial brightness curve prediction model using the loss value to obtain a target brightness curve prediction model.
[0153] In some embodiments, the model updating unit 952 is specifically used to iteratively update the weights of the initial environmental perception model using the loss value, and back-propagate the loss value to the initial brightness curve prediction model during the iterative update process; iteratively update the weights of the initial brightness curve prediction model using the loss value; in response to the updated environmental perception model and the updated brightness curve prediction model both satisfying the convergence conditions, obtain the target brightness curve prediction model.
[0154] In some embodiments, Fig.11 As shown, the enhancement module 93 includes a first defogging unit 931 and an image enhancement unit 932 .
[0155] A first defogging unit 931 is used to perform defogging processing on the first image sample based on an initial first defogging model to obtain a defogged first image sample;
[0156] The image enhancement unit 932 is used to perform image enhancement processing on the first image sample after defogging based on multiple predicted brightness curve mapping diagrams to obtain a predicted enhanced image.
[0157] In some embodiments, Fig.12 As shown, the enhancement module 93 includes a first defogging unit 931 , an image enhancement unit 932 and a second defogging unit 933 .
[0158] A first defogging unit 931 is used to perform defogging processing on the first image sample based on an initial first defogging model to obtain a defogged first image sample;
[0159] An image enhancement unit 932 is used to perform image enhancement processing on the first image sample after defogging based on multiple predicted brightness curve mappings to obtain a predicted intermediate image;
[0160] The second defogging unit 933 is used to perform defogging processing on the predicted intermediate image based on the initial second defogging model to obtain a predicted enhanced image.
[0161] In some embodiments, the processing module 92 is specifically used to perform feature extraction on the first image sample based on the feature extraction layer in the initial brightness curve prediction model to obtain image features of the first image sample in multiple preset channels; based on the activation function layer in the initial brightness curve prediction model, perform nonlinear transformation on the image features of multiple preset channels respectively to obtain predicted brightness curve mapping diagrams corresponding to each of the multiple preset channels.
[0162] In some embodiments, Fig.13 As shown, the enhancement module 93 includes a spatial conversion unit 934, a brightness enhancement unit 935 and a component combination unit 936.
[0163] A space conversion unit 934, configured to convert the first image sample from the first color space to the second color space to obtain a second image sample;
[0164] A brightness enhancement unit 935 is used to perform image enhancement processing on the original brightness component of the second image sample based on the predicted brightness curve mapping map corresponding to each preset channel in the multiple predicted brightness curve mapping maps to obtain a predicted enhanced brightness component;
[0165] The component combining unit 936 is configured to obtain a predicted enhanced image based on the predicted enhanced brightness component and the first image sample.
[0166] In some embodiments, the brightness enhancement unit 935 is specifically used to perform pixel-by-pixel brightness stretching processing on the original brightness component of the second image sample based on the predicted brightness curve mapping map corresponding to the first preset channel in multiple predicted brightness curve mapping maps to obtain the predicted brightness component corresponding to the first preset channel; based on the predicted brightness curve mapping map corresponding to each second preset channel in at least one second preset channel in the multiple predicted brightness curve mapping maps, the predicted brightness component corresponding to the previous preset channel of each second preset channel is sequentially performed pixel-by-pixel brightness stretching processing to obtain the predicted enhanced brightness component; wherein the predicted brightness component corresponding to the previous preset channel of the second preset channel includes at least the predicted brightness component corresponding to the first preset channel.
[0167] The beneficial technical effects corresponding to the exemplary embodiment of the training device 90 for the above-mentioned image enhancement network can be found in the corresponding beneficial technical effects of the above-mentioned exemplary method part, which will not be repeated here.
[0168] Fig.14 An image enhancement device provided in an embodiment of the present disclosure, such as Fig.14 As shown, the image enhancement device 140 includes an image determination module 141 , a first processing module 142 and a second processing module 143 .
[0169] An image determination module 141 is used to determine an image to be processed;
[0170] A first processing module 142 is used to process the image to be processed based on the target brightness curve prediction model in the target image enhancement network to obtain a plurality of brightness curve mapping diagrams corresponding to the image to be processed;
[0171] A second processing module 143 is used to perform image enhancement processing on the image to be processed based on the multiple brightness curve mapping diagrams to obtain an enhanced image;
[0172] Wherein, the target image enhancement network is trained based on the above-mentioned training method.
[0173] In some embodiments, the second processing module 143 is specifically used to perform defogging processing on the image to be processed based on the target first defogging model to obtain the defogged image to be processed; and perform image enhancement processing on the defogged image to be processed based on multiple brightness curve mapping graphs to obtain an enhanced image.
[0174] In some embodiments, the second processing module 143 is specifically used to perform defogging on the image to be processed based on the target first defogging model to obtain the defogged image to be processed; perform image enhancement processing on the defogged image to be processed based on multiple brightness curve mapping graphs to obtain an intermediate image; and perform defogging on the intermediate image based on the target second defogging model to obtain an enhanced image.
[0175] In some embodiments, the first processing module 142 is specifically used to perform feature extraction on the image to be processed based on the feature extraction layer in the target brightness curve prediction model to obtain image features of the image to be processed in multiple preset channels; based on the activation function layer in the target brightness curve prediction model, perform nonlinear transformation on the image features of multiple preset channels respectively to obtain brightness curve mapping diagrams corresponding to each of the multiple preset channels.
[0176] In some embodiments, the second processing module 143 is used to convert the image to be processed from the first color space to the second color space to obtain a converted image; perform image enhancement processing on the original brightness component of the converted image based on the brightness curve mapping map corresponding to each preset channel in multiple brightness curve mapping maps to obtain an enhanced brightness component; based on the enhanced brightness component and the image to be processed, obtain an enhanced image.
[0177] In some embodiments, the second processing module 143 is specifically used to convert the image to be processed from the first color space to the second color space to obtain a converted image; based on the brightness curve mapping map corresponding to the first preset channel in multiple brightness curve mapping maps, the original brightness component of the converted image is subjected to pixel-by-pixel brightness stretching processing to obtain the brightness component corresponding to the first preset channel; based on the brightness curve mapping map corresponding to each second preset channel in at least one second preset channel in the multiple brightness curve mapping maps, the brightness component corresponding to the previous preset channel of each second preset channel is subjected to pixel-by-pixel brightness stretching processing in sequence to obtain an enhanced brightness component; wherein the brightness component corresponding to the previous preset channel of the second preset channel includes at least the brightness component corresponding to the first preset channel; based on the enhanced brightness component and the image to be processed, an enhanced image is obtained.
[0178] The beneficial technical effects corresponding to the exemplary embodiment of the above-mentioned image enhancement device 140 can be found in the corresponding beneficial technical effects of the above-mentioned exemplary method part, which will not be repeated here.
[0179] Exemplary Electronic Devices
[0180] Fig.15 A structural diagram of an electronic device 150 provided in an embodiment of the present disclosure includes at least one processor 151 and a memory 152 .
[0181] The processor 151 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 150 to perform desired functions.
[0182] The memory 152 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 151 may execute one or more computer program instructions to implement the training method of the image enhancement network and / or the image enhancement method and / or other desired functions of the various embodiments of the present disclosure described above.
[0183] In one example, the electronic device 150 may further include: an input device 153 and an output device 154 , and these components are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0184] The input device 153 may also include, for example, a keyboard, a mouse, etc.
[0185] The output device 154 can output various information to the outside, and may include, for example, a display, a speaker, a printer, a communication network and its connected remote output device, etc.
[0186] Of course, to simplify, Fig.15 Only some of the components related to the present disclosure in the electronic device 150 are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application situations, the electronic device 150 may also include any other appropriate components.
[0187] Exemplary computer program products and computer-readable storage media
[0188] In addition to the above-mentioned methods and devices, embodiments of the present disclosure may also provide a computer program product, including computer program instructions, which, when executed by a processor, enable the processor to execute the steps in the training method of the image enhancement network and / or the steps in the image enhancement method of various embodiments of the present disclosure described in the above-mentioned "Exemplary Method" section.
[0189] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the disclosed embodiments, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0190] In addition, an embodiment of the present disclosure may also be a computer-readable storage medium having computer program instructions stored thereon. When the computer program instructions are executed by a processor, the processor executes the steps in the training method of the image enhancement network and / or the steps in the image enhancement method of the various embodiments of the present disclosure described in the above “Exemplary Method” section.
[0191] Computer readable storage media can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium is, for example, but not limited to, a system, device or device including electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0192] The basic principles of the present disclosure are described above in conjunction with specific embodiments. However, the advantages, strengths, effects, etc. mentioned in the present disclosure are only examples and not limitations, and cannot be considered as necessary for each embodiment of the present disclosure. In addition, the specific details disclosed above are only for the purpose of illustration and ease of understanding, rather than limitation, and the above details do not limit the present disclosure to being implemented by adopting the above specific details.
[0193] Those skilled in the art may make various changes and modifications to the present disclosure without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the present disclosure claims and their equivalents, the present disclosure is also intended to include these modifications and variations.
Claims
1. A training method for an image enhancement network, comprising: Determining a first image sample to be enhanced and a perceptual label corresponding to the first image sample; Processing the first image sample based on an initial brightness curve prediction model to obtain a plurality of predicted brightness curve mapping graphs corresponding to the first image sample; Performing image enhancement processing on the first image sample based on the multiple predicted brightness curve mappings to obtain a predicted enhanced image; Performing perception processing on the predicted enhanced image based on the initial environment perception model to obtain a perception result corresponding to the predicted enhanced image; Iteratively training the initial brightness curve prediction model and the initial environment perception model based on the perception result and the perception label corresponding to the first image sample to obtain a target brightness curve prediction model; Based on the target brightness curve prediction model, a target image enhancement network is obtained.
2. The method according to claim 1, wherein: The iterative training of the initial brightness curve prediction model and the initial environment perception model based on the perception result and the perception label corresponding to the first image sample to obtain a target brightness curve prediction model includes: Determining a loss value of the initial environment perception model based on the perception result and the perception label corresponding to the first image sample; The initial environment perception model and the initial brightness curve prediction model are iteratively updated using the loss value to obtain a target brightness curve prediction model.
3. The method according to claim 2, wherein the iterative updating of the initial environment perception model and the initial brightness curve prediction model using the loss value to obtain a target brightness curve prediction model comprises: Iteratively updating the weights of the initial environment perception model using the loss value, and back-propagating the loss value to the initial brightness curve prediction model during the iterative updating process; Iteratively updating the weights of the initial brightness curve prediction model using the loss value; In response to both the updated environment perception model and the updated brightness curve prediction model satisfying convergence conditions, the target brightness curve prediction model is obtained.
4. The method according to any one of claims 1 to 3, wherein: The performing image enhancement processing on the first image sample based on the multiple predicted brightness curve mappings to obtain a predicted enhanced image includes: Performing defogging processing on the first image sample based on the initial first defogging model to obtain a defogged first image sample; Image enhancement processing is performed on the defogged first image sample based on the multiple predicted brightness curve maps to obtain the predicted enhanced image.
5. The method according to any one of claims 1 to 3, wherein: The performing image enhancement processing on the first image sample based on the multiple predicted brightness curve mappings to obtain a predicted enhanced image includes: Performing defogging processing on the first image sample based on the initial first defogging model to obtain a defogged first image sample; Performing image enhancement processing on the first image sample after defogging based on the multiple predicted brightness curve mappings to obtain a predicted intermediate image; The predicted intermediate image is defogged based on the initial second defogging model to obtain the predicted enhanced image.
6. The method according to claim 1, wherein: The processing of the first image sample based on the initial brightness curve prediction model to obtain a plurality of predicted brightness curve mapping graphs corresponding to the first image sample includes: Performing feature extraction on the first image sample based on the feature extraction layer in the initial brightness curve prediction model to obtain image features of the first image sample in multiple preset channels; Based on the activation function layer in the initial brightness curve prediction model, nonlinear transformation is performed on the image features of the multiple preset channels respectively to obtain predicted brightness curve mapping diagrams corresponding to the multiple preset channels respectively.
7. The method according to claim 1, wherein: The performing image enhancement processing on the first image sample based on the multiple predicted brightness curve mappings to obtain a predicted enhanced image includes: Converting the first image sample from a first color space to a second color space to obtain a second image sample; Performing image enhancement processing on the original brightness component of the second image sample based on the predicted brightness curve mapping map corresponding to each preset channel in the plurality of predicted brightness curve mapping maps to obtain the predicted enhanced brightness component; A predicted enhanced image is obtained based on the predicted enhanced luminance component and the first image samples.
8. The method according to claim 7, wherein: The performing image enhancement processing on the original brightness component of the second image sample based on the predicted brightness curve mapping map corresponding to each preset channel in the plurality of predicted brightness curve mapping maps to obtain the predicted enhanced brightness component includes: Performing pixel-by-pixel brightness stretching processing on the original brightness component of the second image sample based on the predicted brightness curve mapping map corresponding to the first preset channel in the multiple predicted brightness curve mapping maps to obtain a predicted brightness component corresponding to the first preset channel; Based on the predicted brightness curve mapping maps corresponding to each of the second preset channels in at least one of the multiple predicted brightness curve mapping maps, sequentially performing pixel-by-pixel brightness stretching processing on the predicted brightness components corresponding to the previous preset channels of each of the second preset channels, to obtain the predicted enhanced brightness components; The predicted brightness component corresponding to the previous preset channel of the second preset channel at least includes the predicted brightness component corresponding to the first preset channel.
9. An image enhancement method, comprising: Determine the image to be processed; Processing the image to be processed based on the target brightness curve prediction model in the target image enhancement network to obtain a plurality of brightness curve mapping graphs corresponding to the image to be processed; Performing image enhancement processing on the image to be processed based on the multiple brightness curve mapping graphs to obtain an enhanced image; Wherein, the target image enhancement network is trained based on the method described in any one of claims 1 to 8.
10. A training device for an image enhancement network, comprising: A sample determination module, used to determine a first image sample to be enhanced and a perceptual label corresponding to the first image sample; A processing module, configured to process the first image sample based on an initial brightness curve prediction model to obtain a plurality of predicted brightness curve mapping graphs corresponding to the first image sample; an enhancement module, configured to perform image enhancement processing on the first image sample based on the multiple predicted brightness curve mapping graphs to obtain a predicted enhanced image; A perception module, used for performing perception processing on the predicted enhanced image based on an initial environment perception model to obtain a perception result corresponding to the predicted enhanced image; A training module, configured to iteratively train the initial brightness curve prediction model and the initial environment perception model based on the perception result and the perception label corresponding to the first image sample to obtain a target brightness curve prediction model; The network determination module is used to obtain a target image enhancement network based on the target brightness curve prediction model.
11. An image enhancement device, comprising: An image determination module, used for determining an image to be processed; A first processing module, used for processing the image to be processed based on the target brightness curve prediction model in the target image enhancement network to obtain a plurality of brightness curve mapping graphs corresponding to the image to be processed; A second processing module, configured to perform image enhancement processing on the image to be processed based on the multiple brightness curve mapping diagrams to obtain an enhanced image; Wherein, the target image enhancement network is trained based on the method described in any one of claims 1 to 8.
12. A computer-readable storage medium storing a computer program, wherein the computer program is used to execute the image enhancement network training method described in any one of claims 1 to 8, or the image enhancement method described in claim 9.
13. An electronic device, comprising: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the instructions to implement the image enhancement network training method described in any one of claims 1 to 8, or the image enhancement method described in claim 9.
