Image processing method and device, and storage medium
The method uses a neural network-based image quality evaluation model to optimize ISP parameters, addressing the variability in ISP parameter debugging and improving image quality in autonomous driving cameras.
Patent Information
- Application Number
- JP2024525638
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-10-29
- Filing Date
- 2022-08-22
- Publication Date
- 2024-11-06
- Estimated Expiration
- 2042-08-22
AI Technical Summary
The debugging of Image Signal Processor (ISP) parameters in autonomous driving cameras is dependent on environmental conditions and human expertise, leading to inconsistent and suboptimal image quality.
An image processing method that utilizes a pre-trained neural network-based image quality evaluation model to automatically adjust ISP parameters based on objective and subjective quality metrics, optimizing image quality for different scene types.
Ensures consistent and improved image quality by automatically adjusting ISP parameters, enhancing the performance of autonomous driving systems across varying environmental conditions.
Smart Images

Figure 2024541043000001_ABST
Abstract
Description
[Technical field]
[0001] The present application relates to the field of artificial intelligence technology, in particular to image processing methods and devices and storage media. [Background technology]
[0002] In the field of autonomous driving, the quality of the images output by a camera depends on the Image Signal Processor (ISP) technology in addition to the camera module, processes, etc.
[0003] In the prior art, as shown in Fig. 1, the imaging process includes: a landscape object is photographed by using a lens 11, the generated optical image is projected onto the surface of a photosensitive sensor 12, then the optical image is converted into an analog electric signal by photoelectric conversion, the noise of the analog electric signal is removed, then the processed analog electric signal is converted into a digital image signal by an analog-to-digital converter 13, the digital image signal is input to a digital signal processing chip (Digital Signal Processor, DSP) 14 for processing by the DSP 14, then data in YUV (or RGB) format is output and sent to a central processing unit (central processing unit, CPU) for processing through an I / O interface, and finally the processed data is converted into an image to be displayed on the screen of an electronic device. The structural framework of the DSP 14 includes an image signal processor (image signal processor, ISP), an image decoder (JPEG encoder, JPEG), and a USB device controller (USB).
[0004] The ISP is configured to perform post-processing on the data output by the photosensitive sensor, and its main functions include black level compensation, lens shading correction, bad pixel correction, color interpolation, noise reduction, white balance correction, color correction, gamma correction, color space conversion (e.g., RGB to YUV conversion), chroma noise reduction and edge enhancement, as well as color and contrast enhancement in YUV space, automatic exposure control, etc. Scene details can be better restored in various optical conditions by using the ISP. ISB technology greatly determines the imaging quality of the camera.
[0005] At present, the image quality of cameras in the field of autonomous driving depends on the ISP parameters, but the debugging effect of ISP parameters depends on the debugging environment, parameter labeling method, debugging personnel experience, etc., and the debugging effect is low. Summary of the Invention
[0006] In view of this, an image processing method and apparatus, as well as a storage medium, are proposed to automatically adjust ISP parameters based on the target evaluation result output by the image quality evaluation model, so as to ensure the debugging effect of ISP parameters and improve the subsequent image quality obtained after ISP processing.
[0007] According to a first aspect, an embodiment of the present application provides an image processing method, the method comprising: processing the captured raw image based on the ISP parameters to obtain an image of interest; Invoking a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, where the image quality assessment model is a model obtained by training a neural network using a sample image, and the target assessment result indicates the image quality of the evaluated target image; Adjusting ISP parameters based on the target evaluation results; Includes.
[0008] In this implementation, the captured original image is processed based on the ISP parameters to obtain a target image. The image quality assessment model is invoked based on the target image to output a target assessment result indicating the image quality. The ISP parameters are automatically adjusted based on the target assessment result. This ensures the debugging effect of the ISP parameters, so that the subsequent image quality obtained after ISP processing is improved.
[0009] In a possible implementation, the object assessment result includes objective and subjective quality information of the object image.
[0010] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0011] In this implementation, the target evaluation result includes the objective quality information and subjective quality information of the target image, so that the parameters output by the image quality evaluation model can indicate the evaluated quality status of the target image in terms of objective indicators and / or subjective aesthetic feeling, which further ensures the evaluation effect of the image quality evaluation model.
[0012] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer, and invoking the pre-trained image quality assessment model based on the target image to output and obtain the target assessment result includes: Inputting a target image into a scene classification layer to output and obtain a scene classification result; inputting the target image and scene classification results into an objective scoring layer to output and obtain objective quality information, and inputting the target image and scene classification results into a subjective scoring layer to output and obtain subjective quality information; Includes.
[0013] In this implementation, a scene classification layer is introduced into the image quality assessment model. The electronic device inputs the target image into the scene classification layer to output and obtain the scene classification result, inputs the target image and the scene classification result into the objective scoring layer to output and obtain the objective quality information, and inputs the target image and the scene classification result into the subjective scoring layer to output and obtain the subjective quality information. This provides a multi-task image processing method, so that the ISP parameters can be subsequently automatically optimized for different scene types, and the image quality of each scene is improved.
[0014] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0015] In this implementation, the objective quality information can include objective score values corresponding to the specified dimension, and the subjective quality information can include subjective score values corresponding to the specified dimension, so that the output parameters of the image quality assessment model are enriched and the assessment effect of the image quality assessment model is guaranteed.
[0016] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise, and adjusting the ISP parameters based on the subject evaluation results includes: determining, for each specified dimension, a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimensions based on the target score values corresponding to the specified dimensions and the preset parameter thresholds; Including, The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0017] In this implementation, a target score value corresponding to the designated dimension is determined according to the objective score value and the subjective score value corresponding to the designated dimension, and an ISP parameter corresponding to the designated dimension is adjusted according to the target score value corresponding to the designated dimension and a preset parameter threshold, so that the ISP parameter can be automatically optimized for different scene types and automatically adapted to all-weather scenes during autonomous driving, and can meet the image quality requirements of autonomous driving perception algorithms.
[0018] In another possible implementation, prior to invoking the pre-trained image quality assessment model based on the target image to output and obtain the target assessment result, the method further includes obtaining an image quality assessment model, the image quality assessment model being obtained by training based on at least one sample data group, each sample data group including a sample image and a pre-labeled accurate assessment result.
[0019] In this implementation, an image quality assessment model is obtained by training based on at least one sample data group, where each sample data group includes sample images and pre-labeled accurate assessment results, which is helpful for subsequently outputting target assessment results based on the trained image quality assessment model, thereby providing a prerequisite for automatically adjusting ISP parameters and debugging ISP parameters.
[0020] In another possible implementation, prior to obtaining the image quality assessment model, the method further comprises: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, inputting sample images into an original parameter model, which is a multi-task neural network model, to obtain a training result; comparing the training results with the accurate evaluation results to obtain a computational loss indicative of the error between the training results and the accurate evaluation results; Obtaining an image quality assessment model by training based on the computational losses corresponding to each of the at least one sample data group; Further includes.
[0021] In this implementation, a possible solution is provided for training an image quality assessment model based on a training sample set. The image quality assessment model is obtained by training based on the computation loss corresponding to at least one sample data group, so that the evaluation effect of the trained image quality assessment model is guaranteed.
[0022] In another possible implementation, after invoking the image quality assessment model based on the target image to output and obtain the target assessment result, the method further comprises: Performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; performing screening on the training sample set based on the label information to obtain a screened training sample set; training an image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model; Further includes.
[0023] In this implementation, an automatic labeling model of objective indicators is provided. The objective label value of the target image in a specified dimension can be automatically labeled, and the training sample set is screened based on the label information. This optimizes the quality of the samples in the training sample set, improves the stability of the image quality assessment model, and enhances the accuracy of the subsequent image processing method.
[0024] According to a second aspect, an embodiment of the present application provides a method for image quality assessment, the method comprising: processing the captured raw image based on the ISP parameters to obtain an image of the target; Invoking the pre-trained image quality assessment model based on the target image to output and obtain a target assessment result; The image quality assessment model is a model obtained by training a neural network using sample images, and the target assessment result indicates the image quality of the assessed target image.
[0025] In a possible implementation, the object assessment result includes objective and subjective quality information of the object image.
[0026] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0027] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer, and invoking the pre-trained image quality assessment model based on the target image to output and obtain the target assessment result includes: Inputting a target image into a scene classification layer to output and obtain a scene classification result; inputting the target image and scene classification results into an objective scoring layer to output and obtain objective quality information, and inputting the target image and scene classification results into a subjective scoring layer to output and obtain subjective quality information; Includes.
[0028] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0029] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise, and the method further comprises: determining, for each specified dimension, a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimensions based on the target score values corresponding to the specified dimensions and the preset parameter thresholds; Including, The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0030] In another possible implementation, prior to invoking the pre-trained image quality assessment model based on the target image to output and obtain the target assessment result, the method further includes obtaining an image quality assessment model, the image quality assessment model being obtained by training based on at least one sample data group, each sample data group including a sample image and a pre-labeled accurate assessment result.
[0031] In another possible implementation, prior to obtaining the image quality assessment model, the method further comprises: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, inputting sample images into an original parameter model, which is a multi-task neural network model, to obtain a training result; comparing the training results with the accurate evaluation results to obtain a computational loss indicative of the error between the training results and the accurate evaluation results; Obtaining an image quality assessment model by training based on the computational losses corresponding to each of the at least one sample data group; Further includes.
[0032] In another possible implementation, after invoking the image quality assessment model based on the target image to output and obtain the target assessment result, the method further comprises: Performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; performing screening on the training sample set based on the label information to obtain a screened training sample set; training an image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model; Further includes.
[0033] According to a third aspect, an embodiment of the present application provides a method for automatically improving image quality, the method comprising: processing the captured raw image based on the ISP parameters to obtain an image of interest; Invoking a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, where the image quality assessment model is a model obtained by training a neural network using a sample image, and the target assessment result indicates the image quality of the evaluated target image; adjusting the ISP parameters based on the subject evaluation results; acquiring a second image; and processing the captured first image based on the adjusted ISP parameters. Includes.
[0034] In a possible implementation, the object assessment result includes objective and subjective quality information of the object image.
[0035] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0036] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer, and invoking the pre-trained image quality assessment model based on the target image to output and obtain the target assessment result includes: Inputting a target image into a scene classification layer to output and obtain a scene classification result; inputting the target image and scene classification results into an objective scoring layer to output and obtain objective quality information, and inputting the target image and scene classification results into a subjective scoring layer to output and obtain subjective quality information; Includes.
[0037] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0038] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise, and adjusting the ISP parameters based on the subject evaluation results includes: determining, for each specified dimension, a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimensions based on the target score values corresponding to the specified dimensions and the preset parameter thresholds; Including, The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0039] In another possible implementation, prior to invoking the pre-trained image quality assessment model based on the target image to output and obtain the target assessment result, the method further includes obtaining an image quality assessment model, the image quality assessment model being obtained by training based on at least one sample data group, each sample data group including a sample image and a pre-labeled accurate assessment result.
[0040] In another possible implementation, prior to obtaining the image quality assessment model, the method further comprises: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, inputting sample images into an original parameter model, which is a multi-task neural network model, to obtain a training result; comparing the training results with the accurate evaluation results to obtain a computational loss indicative of the error between the training results and the accurate evaluation results; Obtaining an image quality assessment model by training based on the computational losses corresponding to each of the at least one sample data group; Further includes.
[0041] In another possible implementation, after invoking the image quality assessment model based on the target image to output and obtain the target assessment result, the method further comprises: Performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; performing screening on the training sample set based on the label information to obtain a screened training sample set; training an image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model; Further includes.
[0042] According to a fourth aspect, an embodiment of the present application provides an objective indicator labeling method, the method comprising: processing the captured raw image based on the ISP parameters to obtain an image of interest; Performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; performing screening on the training sample set based on the label information to obtain a screened training sample set; Training a pre-trained image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model, where the image quality assessment model is a neural network model used to assess image quality; Includes.
[0043] In a possible implementation, before training the pre-trained image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model, the method includes: Invoking an image quality assessment model based on the target image to output and obtain a target assessment result, the image quality assessment model being a model obtained by training a neural network using a sample image, the target assessment result indicating the image quality of the evaluated target image; Adjusting ISP parameters based on the target evaluation results; Further includes.
[0044] In another possible implementation, the object assessment result includes objective and subjective quality information of the object image.
[0045] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0046] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer, and invoking the image quality assessment model based on the target image to output and obtain the target assessment result includes: Inputting a target image into a scene classification layer to output and obtain a scene classification result; inputting the target image and scene classification results into an objective scoring layer to output and obtain objective quality information, and inputting the target image and scene classification results into a subjective scoring layer to output and obtain subjective quality information; Includes.
[0047] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0048] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise, and adjusting the ISP parameters based on the subject evaluation results includes: determining, for each specified dimension, a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimensions based on the target score values corresponding to the specified dimensions and the preset parameter thresholds; Including, The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0049] In another possible implementation, prior to invoking the pre-trained image quality assessment model based on the target image to output and obtain the target assessment result, the method further includes obtaining an image quality assessment model, the image quality assessment model being obtained by training based on at least one sample data group, each sample data group including a sample image and a pre-labeled accurate assessment result.
[0050] In another possible implementation, prior to obtaining the image quality assessment model, the method further comprises: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, inputting sample images into an original parameter model, which is a multi-task neural network model, to obtain a training result; comparing the training results with the accurate evaluation results to obtain a computational loss indicative of the error between the training results and the accurate evaluation results; Obtaining an image quality assessment model by training based on the computational losses corresponding to each of the at least one sample data group; Further includes.
[0051] According to a fifth aspect, an embodiment of the present application provides a camera, the camera including an ISP and a processor.
[0052] The processor is configured to perform a method according to the first aspect or any one of the possible implementations of the first aspect, or to perform a method according to the second aspect or any one of the possible implementations of the second aspect, or to perform a method according to the third aspect or any one of the possible implementations of the third aspect, or to perform a method according to the fourth aspect or any one of the possible implementations of the fourth aspect.
[0053] According to a sixth aspect, an embodiment of the present application provides a vehicle, the vehicle comprising: A processor; a memory configured to store instructions to be executed by the processor; Includes.
[0054] When executing the instructions, the processor is configured to perform a method according to the first aspect or any one of the possible implementations of the first aspect, or to perform a method according to the second aspect or any one of the possible implementations of the second aspect, or to perform a method according to the third aspect or any one of the possible implementations of the third aspect, or to perform a method according to the fourth aspect or any one of the possible implementations of the fourth aspect.
[0055] According to a sixth aspect, an embodiment of the present application provides an image processing apparatus, comprising: a processing unit configured to process the captured raw image based on the ISP parameters to obtain a target image; an output unit configured to call a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, the image quality assessment model being a model obtained by training a neural network using sample images, the target assessment result indicating the image quality of the evaluated target image; an adjustment unit configured to adjust the ISP parameters based on the target evaluation result; Includes.
[0056] The target evaluation result includes objective quality information and subjective quality information of the target image.
[0057] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0058] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer. Input the target image into a scene classification layer to output and obtain a scene classification result; Input the target image and scene classification results to an objective scoring layer to output and obtain objective quality information, and input the target image and scene classification results to a subjective scoring layer to output and obtain subjective quality information. It is further configured as follows.
[0059] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0060] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise. For each specified dimension, determining a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimension based on the target score value corresponding to the specified dimension and the preset parameter threshold value. It is further structured as follows: The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0061] In another possible implementation, the apparatus further comprises an acquisition unit.
[0062] The obtaining unit is configured to obtain an image quality assessment model, which is obtained by training based on at least one sample data group, each sample data group including sample images and pre-labeled accurate assessment results.
[0063] In another possible implementation, the device further comprises a training unit, the training unit comprising: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, input the sample images to an original parameter model, which is a multi-task neural network model, to obtain a training result; Comparing the training results with the accurate evaluation results to obtain a computational loss that indicates the error between the training results and the accurate evaluation results; Obtain an image quality assessment model by training based on the computational losses corresponding to each of at least one sample data group. It is configured as follows.
[0064] In another possible implementation, the device further comprises a retraining unit, the retraining unit comprising: Performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; Perform screening on the training sample set based on the label information to obtain a screened training sample set; Train an image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model. It is configured as follows.
[0065] According to an eighth aspect, an embodiment of the present application provides an image quality assessment device, the device comprising: a processing unit configured to process the captured raw image based on the ISP parameters to obtain a target image; an output unit configured to invoke the pre-trained image quality assessment model based on the target image to output and obtain a target assessment result; The image quality assessment model is a model obtained by training a neural network using sample images, and the target assessment result indicates the image quality of the assessed target image.
[0066] In a possible implementation, the object assessment result includes objective and subjective quality information of the object image.
[0067] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0068] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer. Input the target image into a scene classification layer to output and obtain a scene classification result; The target image and scene classification results are input to an objective scoring layer to output and obtain objective quality information, and the target image and scene classification results are input to a subjective scoring layer to output and obtain subjective quality information. It is further configured as follows.
[0069] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0070] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise. The apparatus further includes an adjustment unit. The adjustment unit includes: For each specified dimension, determining a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimension based on the target score value corresponding to the specified dimension and the preset parameter threshold value. It is configured as follows: The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0071] In another possible implementation, the device further includes an acquisition unit, which is configured to acquire an image quality assessment model, the image quality assessment model being acquired by training based on at least one sample data group, each sample data group including a sample image and a pre-labeled accurate assessment result.
[0072] In another possible implementation, the device further comprises a training unit, the training unit comprising: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, input the sample images to an original parameter model, which is a multi-task neural network model, to obtain a training result; Comparing the training results with the accurate evaluation results to obtain a computational loss that indicates the error between the training results and the accurate evaluation results; Obtain an image quality assessment model by training based on the computational losses corresponding to each of at least one sample data group. It is configured as follows.
[0073] In another possible implementation, the device further comprises a retraining unit, the retraining unit comprising: Performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; Perform screening on the training sample set based on the label information to obtain a screened training sample set; Train an image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model. It is configured as follows.
[0074] According to a ninth aspect, an embodiment of the present application provides an apparatus for automatically improving image quality, the apparatus comprising: a first processing unit configured to process the captured raw image based on the ISP parameters to obtain a target image; an output unit configured to call a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, the image quality assessment model being a model obtained by training a neural network using sample images, the target assessment result indicating the image quality of the evaluated target image; an adjusting unit configured to adjust the ISP parameters based on the target evaluation result; a second processing unit configured to process the captured first image based on the adjusted ISP parameters to obtain a second image; Includes.
[0075] In a possible implementation, the object assessment result includes objective and subjective quality information of the object image.
[0076] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0077] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer. Input the target image into a scene classification layer to output and obtain a scene classification result; The target image and scene classification results are input to an objective scoring layer to output and obtain objective quality information, and the target image and scene classification results are input to a subjective scoring layer to output and obtain subjective quality information. It is further configured as follows.
[0078] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0079] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise. For each specified dimension, determining a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimension based on the target score value corresponding to the specified dimension and the preset parameter threshold value. It is further structured as follows: The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0080] In another possible implementation, the device further includes an acquisition unit, which is further configured to acquire an image quality assessment model, the image quality assessment model being acquired by training based on at least one sample data group, each sample data group including a sample image and a pre-labeled accurate assessment result.
[0081] In another possible implementation, the device further comprises a training unit, the training unit comprising: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, input the sample images to an original parameter model, which is a multi-task neural network model, to obtain a training result; Comparing the training results with the accurate evaluation results to obtain a computational loss that indicates the error between the training results and the accurate evaluation results; Obtain an image quality assessment model by training based on the computational losses corresponding to each of at least one sample data group. It is configured as follows.
[0082] In another possible implementation, the device further comprises a retraining unit, the retraining unit comprising: Performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; Perform screening on the training sample set based on the label information to obtain a screened training sample set; Train an image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model. It is configured as follows.
[0083] According to a tenth aspect, an embodiment of the present application provides an objective indicator labeling apparatus, the apparatus comprising: a processing unit configured to process the captured raw image based on the ISP parameters to obtain a target image; A labeling unit configured to perform automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; a screening unit configured to perform screening on the training sample set based on the label information to obtain a screened training sample set; a retraining unit configured to train a pre-trained image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model, the image quality assessment model being a neural network model used to evaluate image quality; and Includes.
[0084] In a possible implementation, the apparatus comprises: an output unit configured to call an image quality assessment model based on the target image to output and obtain a target assessment result, the image quality assessment model being a model obtained by training a neural network using sample images, the target assessment result indicating the image quality of the evaluated target image; an adjustment unit configured to adjust the ISP parameters based on the target evaluation result; Further includes.
[0085] In another possible implementation, the object assessment result includes objective and subjective quality information of the object image.
[0086] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0087] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer. Input the target image into a scene classification layer to output and obtain a scene classification result; The target image and scene classification results are input to an objective scoring layer to output and obtain objective quality information, and the target image and scene classification results are input to a subjective scoring layer to output and obtain subjective quality information. It is further configured as follows.
[0088] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0089] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise. For each specified dimension, determining a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimension based on the target score value corresponding to the specified dimension and the preset parameter threshold value. It is further structured as follows: The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0090] In another possible implementation, the apparatus further comprises an acquisition unit.
[0091] The obtaining unit is configured to obtain an image quality assessment model, which is obtained by training based on at least one sample data group, each sample data group including sample images and pre-labeled accurate assessment results.
[0092] In another possible implementation, the device further comprises a training unit, the training unit comprising: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, input the sample images to an original parameter model, which is a multi-task neural network model, to obtain a training result; Comparing the training results with the accurate evaluation results to obtain a computational loss that indicates the error between the training results and the accurate evaluation results; Obtain an image quality assessment model by training based on the computational losses corresponding to each of at least one sample data group. It is configured as follows.
[0093] According to an eleventh aspect, an embodiment of the present application provides a non-volatile computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause a method according to the first aspect or any one of its possible implementations to be performed, or a method according to the second aspect or any one of its possible implementations to be performed, or a method according to the third aspect or any one of its possible implementations to be performed, or a method according to the fourth aspect or any one of its possible implementations to be performed.
[0094] According to a twelfth aspect, an embodiment of the present application provides a computer program product. The computer program product includes a computer readable code, or a non-volatile computer readable storage medium carrying the computer readable code. When the computer readable code is executed on an electronic device, a processor in the electronic device executes a method according to the first aspect or any one of its possible implementations, executes a method according to the second aspect or any one of its possible implementations, executes a method according to the third aspect or any one of its possible implementations, or executes a method according to the fourth aspect or any one of its possible implementations.
[0095] The accompanying drawings, which are incorporated in and constitute a part of this specification, together with the description illustrate exemplary embodiments, features, and aspects of the present application and are intended to explain the principles of the present application. [Brief description of the drawings]
[0096] [Figure 1] FIG. 1 is a schematic diagram of an imaging process in the prior art. [Diagram 2] FIG. 2 is a schematic diagram of a structure of an electronic device according to an exemplary embodiment of the present application. [Diagram 3] 1 is a flowchart of an image processing method according to an exemplary embodiment of the present application. [Figure 4] 4 is a flowchart of an image processing method according to another exemplary embodiment of the present application. [Diagram 5] FIG. 2 is a schematic diagram of the structure of an image quality assessment model according to an exemplary embodiment of the present application; [Figure 6] 1 is a flowchart of a process for training an image quality assessment model according to an exemplary embodiment of the present application. [Figure 7] 4 is a flowchart of an image processing method according to another exemplary embodiment of the present application. [Figure 8] 4 is a flowchart of an image processing method according to another exemplary embodiment of the present application. [Figure 9]FIG. 1 is a block diagram of an image processing device according to an exemplary embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0097] Various exemplary embodiments, features, and aspects of the present application are described in detail below with reference to the accompanying drawings, in which the same reference numerals indicate elements having the same or similar functions. While various aspects of the embodiments are illustrated in the accompanying drawings, the accompanying drawings are not necessarily drawn to scale unless otherwise specified.
[0098] The specific word "exemplary" herein means "serving as an example, embodiment, or illustration." Any embodiment described as an "exemplary" is not necessarily being described as superior or better than other embodiments.
[0099] In addition, numerous specific details are provided in the following specific implementations to better describe the present application. Those skilled in the art will appreciate that the present application can be practiced without some of the specific details. In some instances, methods, schemes, elements, and circuits that are well known to those skilled in the art have not been described in detail so as to emphasize the subject matter of the present application.
[0100] First, the application scenario of this application is described.
[0101] The method and apparatus provided in the embodiment of the present application may be applied in the fields of intelligent vehicles, new energy vehicles, etc. The following only uses an example in which the method provided in the embodiment of the present application is executed by an electronic device for description. Figure 2 is a schematic diagram of the structure of an electronic device according to an exemplary embodiment of the present application. The electronic device can be applied to an Autonomous Driving (AD) or Advanced Driver Assistance System (ADAS) embedded platform.
[0102] The electronic device may include a vehicle 21. The vehicle 21 may be a vehicle equipped with wireless communication function. An in-vehicle terminal, an in-vehicle module, an in-vehicle unit, a chip (system), or other parts or components of the vehicle 21 may be provided with wireless communication function. In the embodiment of the present application, the vehicle 21 may be in an automatic driving state. In other words, the vehicle 21 drives completely autonomously and does not need to be controlled by a driver or only needs a little control by the driver.
[0103] The camera 22 is disposed in the vehicle 21. The camera 22 is configured to optimize the ISP parameters before delivery of the electronic device by using a pre-trained image quality evaluation model. Optionally, the camera 22 includes a lens, an ISP, and a processor. The lens is configured to capture an original image. The ISP is configured to perform post-processing on the original image, and has main functions including black level compensation, lens correction, bad pixel correction, color interpolation, noise removal, white balance correction, color correction, gamma correction, color space conversion (e.g., RGB to YUV conversion), chroma noise removal, edge enhancement, and color and contrast enhancement in YUV space, automatic exposure control, etc. The processor is a newly added processor configured to enhance image quality. The processor is configured to automatically adjust the ISP parameters based on the target evaluation result output by the image quality evaluation model to ensure the debugging effect of the ISP parameters to improve the subsequent image quality obtained after the ISP processing.
[0104] In an embodiment of the present application, the ISP is configured to process the captured original image to obtain a target image, the processor is configured to obtain the target image, invoke a pre-trained image quality assessment model based on the target image, and output a target assessment result, the image quality assessment model being a model obtained by training a neural network using a sample image, the target assessment result indicating the image quality of the evaluated target image, and adjust the ISP parameters based on the target assessment result.
[0105] At least one sensor 23 may further be disposed on the vehicle 21, such as an on-board radar (e.g., millimeter wave radar, LiDAR, or other ultrasonic radar), a rain sensor, a vehicle attitude sensor (e.g., a gyroscope), an inertial measurement unit (IMU), and a global navigation satellite system (GNSS). The sensors 23 may be disposed on one vehicle 21 or on multiple vehicles 21.
[0106] The vehicle 21 may further be provided with an autonomous driving system 24. The autonomous driving system 24 may be configured to generate an autonomous driving policy used to respond to road conditions based on data collected by the sensors, and to realize autonomous driving of the vehicle 21 according to the generated policy.
[0107] The vehicle 21 may further be arranged with a human-machine interface (HMI) 25. The human-machine interface 25 may be configured to broadcast the current road conditions and the policies used by the automated driving system 24 of the vehicle 21 to inform the concerned driver and passengers by using visual icons or in the form of audio broadcasts.
[0108] In a possible implementation, the electronic device of the embodiment of the present application may further include a server 20. The server 20 may be located in the vehicle 21 as an on-board computing unit, or may be located in the cloud, may be a physical device, or may be a visual device, such as a virtual machine or a container, and may be equipped with wireless communication capabilities. For example, the server 20 is a virtual device provided after resource pooling of servers (spatially separated) in multiple locations. A chip (system) or other parts or components of the server 20 may be provided with wireless communication capabilities. The server 20 may communicate with the vehicle 21 through a wireless communication connection, for example by using mobile communication technologies such as 2G / 3G / 4G / 5G, or through wireless communication such as Wi-Fi, Bluetooth, frequency modulation (FM), radio modem, or satellite communication. For example, in a test, the server 20 may be carried in the vehicle 21 and communicate with the vehicle 21 through wireless communication. Through communication between the vehicles 21 and the server 20, the server 20 can collect data collected by sensors located within one or more vehicles 21 or on the road or elsewhere for calculations and return the calculation results to the corresponding vehicle 21.
[0109] The following describes the image processing method provided in the embodiments of the present application by using some exemplary embodiments.
[0110] 3 is a flowchart of an image processing method according to an exemplary embodiment of the present application. In this embodiment, an example in which the method is applied to the electronic device shown in FIG. 2 is used for explanation. The method includes several steps:
[0111] Step 301: Process the captured raw image based on the ISP parameters to obtain a target image.
[0112] Optionally, the electronic device capturing the original image includes capturing an optical image by using a camera and obtaining a digital image signal, i.e. the original image, through processing performed by a photosensitive sensor and an analog-to-digital converter.
[0113] Optionally, the electronic device captures optical images in real time or at preset time intervals by using a camera, where the preset time intervals are set by default or defined and set by a user, which is not limited in the embodiment of the present application.
[0114] Optionally, the electronic device processes the captured original image by using the ISP, wherein the electronic device performs at least one of the following processing operations by using the ISP: black level compensation, lens correction, bad pixel correction, color interpolation, noise removal, white balance correction, color correction, gamma correction, color space conversion (e.g., RGB to YUV conversion), chroma noise removal, edge enhancement, color and contrast ratio enhancement in YUV color space, and automatic exposure control.
[0115] Optionally, the image format of the source image is Bayer format, and the image format of the target image is YUV or RGB format.
[0116] The ISP parameters are image signal processing parameters. Optionally, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise.
[0117] The image quality of the target image is related to the ISP parameters. Optionally, the image quality of the target image indicates the performance of the target image with respect to objective metrics and / or subjective aesthetic perception.
[0118] Optionally, the image quality of the target image indicates an assessed objective quality status and / or an assessed subjective quality status of the target image in a specified dimension, for example the specified dimension includes at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0119] Step 302: Invoke a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, where the image quality assessment model is a model obtained by training a neural network using a sample image, and the target assessment result indicates the image quality of the evaluated target image.
[0120] The electronic device obtains a pre-trained image quality assessment model, inputs a target image into the image quality assessment model, and outputs and obtains a target assessment result.
[0121] There are two possible implementations for the electronic device to obtain a pre-trained image quality assessment model, including but not limited to the following:
[0122] In a possible implementation, the electronic device pre-trains and stores the image quality assessment model.The electronic device retrieves the image quality assessment model stored in the electronic device.
[0123] In another possible implementation, the server pre-trains and stores the image quality assessment model, and the electronic device retrieves the image quality assessment model from the server.
[0124] The image quality assessment model is a model obtained by training a neural network using sample images and accurate assessment results. In other words, the image quality assessment model is determined based on the sample images and accurate assessment results. The accurate assessment results include pre-labeled assessment results of the image quality of the sample images. For example, the neural network is a multi-task Convolutional Neural Network (CNN). The type of the neural network is not limited in the embodiment of the present application.
[0125] The image quality assessment model is used to convert the input target image into a target assessment result, which indicates the image quality of the assessed target image.
[0126] The image quality assessment model indicates the correlation between the target image and the target assessment result.
[0127] The image quality assessment model is a preset mathematical model, and the image quality assessment model includes a model coefficient between a target image and a target assessment result. The model coefficient may be a fixed value, or may be a value that is dynamically modified over time, or may be a value that is dynamically modified according to a usage scenario.
[0128] In a possible implementation, the image quality assessment model may be a convolutional neural network capable of assessing the image quality of the target image.
[0129] It should be noted that, for the training process of the image quality assessment model, please refer to the relevant details in the following embodiments, and the details will not be described here.
[0130] Optionally, the target assessment result comprises objective quality information and / or subjective quality information of the target image, the objective quality information indicating the objective quality information of the target image in terms of objective indicators and the subjective quality information indicating an assessed quality status of the target image in terms of subjective aesthetic sensations.
[0131] Optionally, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0132] Step 303: Adjust the ISP parameters based on the object evaluation results.
[0133] The electronic device adjusts the ISP parameters based on the target evaluation results to continuously improve the image quality of the output image.
[0134] Optionally, the electronic device processes the captured first image based on the adjusted ISP parameters to obtain a second image, the image quality of the second image being higher than the image quality of a third image, the second image being an image processed based on the adjusted ISP parameters and the third image being an image processed based on the ISP parameters obtained before the adjustment.
[0135] Optionally, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise.
[0136] Optionally, the electronic device adjusting the ISP parameters based on the target evaluation results includes, for each specified dimension, determining a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension, and adjusting the ISP parameters corresponding to the specified dimension based on the target score value corresponding to the specified dimension and a preset parameter threshold, where the preset parameter threshold is a preset parameter threshold of the specified dimension for the scene type indicated by the scene classification result.
[0137] In addition, for how the electronic device adjusts the ISP parameters according to the target evaluation results for different ISP parameter types, please refer to the related details in the following embodiments, and the details will not be described here.
[0138] In summary, the captured original image is processed based on the ISP parameters to obtain a target image. The image quality assessment model is invoked based on the target image to output a target assessment result indicating the image quality. The ISP parameters are automatically adjusted based on the target assessment result. This ensures the debugging effect of the ISP parameters, so that the subsequent image quality obtained after ISP processing is improved.
[0139] In a possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer. The scene classification layer is used to perform scene classification on the target image to output and obtain a scene classification result. The objective scoring layer is used to perform objective scoring on the target image based on the scene classification result to obtain objective quality information. The subjective scoring layer is used to perform subjective scoring on the target image based on the scene classification result to obtain subjective quality information. In a possible implementation, as shown in FIG. 4, the image processing method includes, but is not limited to, the following steps: 1. Capture an original image; 2. Perform ISP optimization on the original image to obtain an optimized target image; 3. Perform scene classification on the target image, i.e., input the target image into the scene classification layer of the image quality assessment model, and output and obtain a scene classification result; 4. Perform image quality assessment based on the target image and the scene classification result, i.e., input the target image and the scene classification result into the objective scoring layer to output and obtain objective quality information, and input the target image and the scene classification result into the subjective scoring layer to output and obtain subjective quality information. 5. Automatically adjust ISP parameters according to objective quality information and subjective quality information. The method may further include: 6. When the ISP acquires the target image, automatically label the objective indicators of the target image to obtain label information, and then optimize the training sample set of the image quality assessment model according to the label information. This improves the stability and accuracy of the image quality assessment model.
[0140] For example, the image quality assessment model is a multi-task CNN model. In the example, as shown in FIG. 5, the image quality assessment model includes a CNN encoder 51, a scene classification decoder 52, an objective scoring decoder 53, and a subjective scoring decoder 54. The CNN encoder 51 includes a pooling layer and a convolution layer of a classful network, and performs downsampling by using three convolution layers. The CNN encoder 51 is used to extract image features from a target image and input the extracted image features to a scene classification decoder 52. The scene classification decoder 52 is also called a scene classification layer, which is a fully connected layer with a softmax activation function, and is used to perform scene classification on the target image based on the input image features, and output and obtain a scene classification result. The objective scoring decoder 53 is also called an objective scoring layer, which is used to perform objective scoring on the target image based on the input scene classification result, and obtain objective quality information. Based on scale invariance, the objective scoring decoder 53 uses a scaling layer internally to achieve accurate scoring on objective indicators. The subjective scoring decoder 54, also called a subjective scoring layer, is used to perform subjective scoring on the target image based on the input scene classification result to obtain subjective quality information. The subjective scoring decoder 54 performs upsampling by using three transposed convolution layers based on the encoding features.
[0141] It should be noted that, before the electronic device obtains the image quality assessment model, the electronic device needs to train a training sample set to obtain the image quality assessment model. The following describes the training process of the image quality assessment model.
[0142] In a possible implementation, as shown in FIG. 6, the training process of the image quality assessment model includes the following steps:
[0143] Step 601: Obtain a training sample set including at least one sample data group.
[0144] The image quality assessment model is obtained by training based on at least one sample data group, where each sample data group includes sample images and pre-labeled accurate assessment results.
[0145] The sample images are images in an image database, and the accurate assessment results are pre-labeled assessment results of the image quality of the sample images.
[0146] Step 602: For at least one sample data group, input the sample images into an original parameter model, which is a multi-task neural network model, to obtain a training result.
[0147] Optionally, the original parameter model is established based on a neural network model, for example, the original parameter model is established based on a CNN model.
[0148] For example, for each sample data group, the electronic device generates an input-output pair corresponding to the sample data group, the input parameters of the input-output pair are sample images in the sample data group, and the target parameters are accurate evaluation results of the sample data group, and the electronic device inputs the input parameters into the original parameter model to obtain a training result.
[0149] Optionally, the input-output pairs are represented by using a property vector.
[0150] Step 603: The training results are compared with the correct evaluation results to obtain a computational loss indicating the error between the training results and the correct evaluation results.
[0151] In a possible implementation, the computational loss is the sum of computational losses corresponding to the scene class, the subjective indicator, and the objective indicator, respectively. For each sample data group, the training result includes a scene classification prediction result, subjective quality prediction information, and objective quality prediction information, and the accurate evaluation result includes an accurate scene classification result, accurate subjective quality information, and accurate objective quality information. The accurate subjective quality information includes an accurate subjective score value corresponding to the specified dimension, and the accurate objective quality information includes an accurate objective score value corresponding to the specified dimension.
[0152] Optionally, cross entropy is used as the computation loss corresponding to the scene class. class The formula for calculating (p,q) is:
number
[0153] where p is the scene classification prediction result, q is the correct scene classification result, and c is the set of scene types.
[0154] Optionally, the sum of the cross entropy of the loss of the objective measure and the confidence is used as the computational loss corresponding to the objective measure. box The formula for calculating (p,q) is:
number
[0155] where p is the objective quality prediction information, q is the accurate objective quality information, x, y, m and n denote the brightness dimension, color dimension, contrast ratio dimension and sharpness and noise dimension, respectively, i denotes a single scene, the value of i is a positive integer, and I denotes a set of scene types.
[0156] Optionally, an empirical mode decomposition-based loss (EMD-based loss) is used as the computation loss corresponding to the subjective metric, i.e., a score histogram is generated for the prediction of sample images based on the probability distribution of human ratings on the sample images. (outside 1) The formula to calculate TIFF2024541043000004.tif9170 is:
number
[0157] where p is the subjective quality prediction information, p with a hat is the exact subjective quality information, k is a set of specified indices, r is the power of the root, and r is a fixed value set in advance. p (k) indicates the cumulative probability of the predicted subjective score. The larger the predicted subjective score, the smaller the CDF p The value of (k) is large. In the prediction process, a softmax function is used to ensure that each independent probability is greater than 0.
[0158] Step 604: Obtain an image quality assessment model by training based on the computational losses corresponding to each of the at least one sample data group.
[0159] Optionally, the electronic device determines a gradient direction of the image quality assessment model based on the computational loss and a backpropagation algorithm, and forward updates model parameters in the image quality assessment model layer by layer from the output layer of the image quality assessment model.
[0160] Image quality assessment is performed on the target image based on the image quality assessment model trained in the above embodiment. Figure 7 is a flowchart of an image processing method according to another exemplary embodiment of the present application. In this embodiment, an example in which the method is applied to the electronic device shown in Figure 1 is used for explanation. The method includes several steps:
[0161] Step 701: The captured raw image is processed based on the ISP parameters to obtain a target image.
[0162] In addition, for the process in which the electronic device processes the captured original image based on the ISP parameters to obtain the target image, please refer to the relevant details in the above embodiment, and the details will not be described again here.
[0163] Step 702: Obtain an image quality assessment model obtained by training based on at least one sample data group, where each sample data group includes sample images and pre-labeled accurate assessment results.
[0164] It should be noted that, for the training process of the image quality assessment model, please refer to the relevant details in the above embodiment, and the details will not be described again here.
[0165] Step 703: Input the target image into the scene classification layer, and output and obtain the scene classification result.
[0166] The electronic device inputs a target image into a scene classification layer, and outputs and obtains a scene classification result, wherein the scene classification result of the target image uniquely indicates a scene type corresponding to the target image within a scene type set.
[0167] Optionally, the scene classification layer is used to determine the scene type corresponding to the target image based on the color temperature and photosensibility (ISO) of the target image. The color temperature can have seven values, for example, D75, D65, D50, TL84, A and H. The curve fitting the color temperature can match the color temperature in the all-weather traffic scene of the autonomous driving. The photosensitivity can have three ranges, for example, low photosensitivity lower than ISO800, intermediate photosensitivity from ISO800 to ISO6400, and high photosensitivity higher than ISO6400. Alternatively, ISO can have 16 ranges, such as ISO100, ISO200, ISO400, ISO800, ISO1600, ISO3200, ISO6400, ISO12800, ISO25600, ISO51200, ISO102400, ISO204800, ISO409600, ISO819200, IOS1638400, and ISO3276800, which basically cover all-weather lighting conditions in autonomous driving traffic scenes. Based on two parameters, namely color temperature and photosensitivity, the requirements of autonomous driving all-weather scenes are basically covered.
[0168] In a possible implementation, the scene type set includes five scene types: scene type 1 "midday", scene type 2 "morning and afternoon", scene type 3 "early morning and evening", scene type 4 "underground and tunnel", and scene type 5 "night". The scene type classification method is not limited to the embodiment of the present application.
[0169] Step 704: The target image and scene classification result are input into an objective scoring layer to output and obtain objective quality information, and the target image and scene classification result are input into a subjective scoring layer to output and obtain subjective quality information.
[0170] Optionally, the objective quality information comprises objective score values corresponding to specified dimensions, the specified dimensions comprising at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0171] The objective scoring layer is used to perform objective indicator scoring for white balance of the target image based on the scene classification result to obtain an objective score value for brightness dimension, and / or perform objective indicator scoring for color reproduction of the target image to obtain an objective score value for color dimension, and / or perform objective indicator scoring for texture detail of the target image to obtain an objective score value for contrast ratio dimension, and / or perform objective indicator scoring for sharpness and signal-to-noise ratio of the target image to obtain an objective score value for sharpness and noise dimension.
[0172] Optionally, the objective scoring layer includes a deep object detection neural network and is used to perform objective indicator scoring on the image quality of the target image based on the scene classification result to obtain objective quality information, so that the image quality level can be quantitatively analyzed based on the objective indicators of the descriptive dimensions of the image quality of the target image.
[0173] Optionally, the subjective quality information includes subjective score values corresponding to the designated dimensions, the designated dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension. The subjective scores are given based on subjective aesthetics. The subjective aesthetics is an important factor of aesthetic quality in hue, contrast ratio, and image denoising, and a multi-layer Laplace filter can be used to find the hue, contrast ratio, or image denoising parameters that are approximately aesthetically optimal settings.
[0174] The subjective scoring layer is used to perform subjective index scoring for white balance of the target image based on the scene classification result to obtain a subjective score value for brightness dimension, and / or perform subjective index scoring for color reproduction of the target image to obtain a subjective score value for color dimension, and / or perform subjective index scoring for texture details of the target image to obtain a subjective score value for contrast ratio dimension, and / or perform subjective index scoring for sharpness and signal-to-noise ratio of the target image to obtain a subjective score value for sharpness and noise dimension.
[0175] Optionally, a subjective scoring layer, including a deep object detection neural network and a semantic segmentation neural network, is used to perform subjective indicator scoring on the image quality of the target image based on the scene classification result to obtain subjective quality information, so that the distribution of human evaluation comments on the image can be predicted from direct perception (technical perspective) and attractiveness (aesthetic perspective).
[0176] In a possible implementation, the target image is any sample image in the training sample set. When outputting the target image, the ISP obtains label information and automatically labels the objective indicators of the target image, so as to subsequently optimize the training sample set of the image quality assessment model based on the label information.
[0177] Optionally, the target image is automatically labeled to obtain label information, the label information including objective label values corresponding to the specified dimensions, the training sample set is screened based on the label information to obtain a screened training sample set, and the image quality assessment model is trained based on the screened training sample set to obtain an updated image quality assessment model.
[0178] Optionally, screening the training sample set based on the label information to obtain a screened training sample set includes deleting a target image from the training sample set to obtain a screened training sample set when an objective label value corresponding to a specified dimension satisfies a pre-set deletion condition.
[0179] For example, the preset deletion condition includes that the absolute value of the difference between the objective label value corresponding to the specified dimension and the pre-labeled accurate objective score value is greater than a first preset threshold. The first preset threshold is set by default or defined and set by a user, which is not limited in the embodiment of the present application.
[0180] For example, the preset deletion condition includes that the objective label value corresponding to the specified dimension is greater than a second preset threshold value, and the second preset threshold value is set by default or defined and set by a user, which is not limited in the embodiment of the present application.
[0181] Optionally, at least one candidate sample data group is obtained. Each candidate sample data group includes a candidate sample image and a pre-labeled accurate evaluation result. The at least one candidate sample data group is added to the screened training sample set, and the image quality assessment model is trained based on the training sample set obtained after adding the at least one candidate sample data group, so as to obtain an updated image quality assessment model. This method can continuously increase the amount of samples in the training sample set, optimize sample quality, and enhance the stability and accuracy of the image quality assessment model.
[0182] Optionally, the label information includes objective label values corresponding to designated dimensions, the designated dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0183] Optionally, the specified dimension includes a brightness dimension. The objective label value corresponding to the brightness dimension includes a brightness deviation value. The brightness deviation value indicates the exposure state. The mean value and variance of the target image relative to the grayscale image are calculated separately for different scene types. When abnormal brightness exists, the mean value is out of the mean point interval and the variance is also relatively small. By calculating the mean value and variance of the grayscale image, the brightness deviation value of the target image is determined.
[0184] For example, the mean value and deviation of the target image relative to the grayscale image are calculated. When abnormal brightness exists, the mean value deviates from the mean point (which may be assumed to be 128), and the deviation is also relatively small. Whether the target image is overexposed or underexposed is determined by calculating the mean value and deviation of the grayscale image. For example, the second preset threshold is 1. A brightness deviation value less than or equal to 1 indicates that the brightness of the target image is normal, and the target image does not need to be deleted. A brightness deviation value greater than 1 indicates that the brightness of the target image is abnormal, and the target image is deleted from the training sample set. As another example, different thresholds are set for different scene types. A brightness deviation value greater than a third preset threshold indicates that the brightness of the target image is excessively high. A brightness deviation value less than a fourth preset threshold indicates that the brightness of the target image is excessively low. The third preset threshold is greater than the fourth preset threshold.
[0185] Optionally, the specified dimension includes a color dimension. The objective label value corresponding to the color dimension includes a color cast value. For example, the target image is an RGB image. The RGB image is transformed into the CIE Lab space, where L indicates the brightness of the image, a indicates the red / green component of the image, and b indicates the yellow / blue component of the image. Generally, for an image with a color cast, the average values of components a and b are far from the origin and the deviation is small. The color cast value of the target image is determined by calculating the average values and deviations of components a and b of the target image.
[0186] Optionally, the specified dimensions include a contrast ratio dimension. The objective label values corresponding to the contrast ratio dimension include a gamma value.
[0187] Optionally, the specified dimensions include sharpness and noise dimensions. Objective label values corresponding to the sharpness and noise dimensions include Brenner gradient and peak signal-to-noise ratio.
[0188] Optionally, after the image quality assessment model is invoked based on the target image to output and obtain the target assessment result, the method further includes the electronic device adding the target image and the target assessment result to a training sample set to obtain an updated training sample set, and training the image quality assessment model based on the updated training sample set to obtain an updated image quality assessment model.
[0189] Step 705: Adjust the ISP parameters based on the objective quality information and the subjective quality information.
[0190] Optionally, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise, and adjusting the ISP parameters based on the target evaluation results includes: for each specified dimension, determining a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension, and adjusting the specified ISP parameters based on the target score value corresponding to the specified dimension and a preset parameter threshold, the preset parameter threshold being a preset parameter threshold of the specified dimension for the scene type indicated by the scene classification result.
[0191] Optionally, the target score value corresponding to the specified dimension is determined based on the objective score value and the subjective score value corresponding to the specified dimension, a first weight value of the objective score value, and a second weight value of the subjective score value, where the first weight value of the objective score value and the second weight value of the subjective score value are set by default or defined and set by a user, which is not limited in the embodiment of the present application.
[0192] Optionally, the training sample set is updated based on the adjusted ISP parameters, and the image quality assessment model is trained based on the updated training sample set to obtain an updated image quality assessment model.
[0193] In summary, the multi-task image processing method in the field of autonomous driving proposed in this solution automatically optimizes ISP parameters in different scene types, improves image quality in each scene, and automatically adapts to all-weather scenes in autonomous driving, thereby meeting the image quality requirements of autonomous driving recognition algorithms. In addition, objective indicator automatic labeling is provided. The objective label value of the target image in a specified dimension can be automatically labeled, and the training sample set is screened based on the label information. In this way, the quality of the samples in the training sample set is improved, and the number of samples can be continuously increased to enhance the stability of the image quality assessment model, thereby improving the accuracy of the image processing model.
[0194] 8 is a flowchart of a method for automatically improving image quality according to an exemplary embodiment of the present application. In this embodiment, an example in which the method is applied to the electronic device shown in FIG. 1 is used for illustration. The method includes several steps:
[0195] Step 801: The captured original image is processed based on the ISP parameters to obtain a target image.
[0196] Step 802: Based on the target image, a pre-trained image quality assessment model is called to output and obtain a target assessment result, where the image quality assessment model is a model obtained by training a neural network using a sample image, and the target assessment result indicates the image quality of the evaluated target image.
[0197] Step 803: Adjust the ISP parameters based on the subject evaluation results.
[0198] In addition, for the process in which the electronic device adjusts the ISP parameters according to the target evaluation result output by the image quality evaluation model, please refer to the relevant details in the above embodiment, and the details will not be described again here.
[0199] Step 804: The captured first image is processed based on the adjusted ISP parameters to obtain a second image.
[0200] Optionally, the electronic device processes the first image based on the adjusted ISP parameters to obtain a second image, the image quality of the second image being higher than the image quality of a third image, the second image being an image processed based on the adjusted ISP parameters and the third image being an image processed based on the ISP parameters before the adjustment.
[0201] For the process in which the electronic device processes the captured first image based on the adjusted ISP parameters to obtain the second image, please refer to the relevant details of the process in which the captured original image is processed based on the ISP parameters to obtain the target image, and the details will not be described again here.
[0202] The following are apparatus embodiments of the present application, which can be used to implement the method embodiments of the present application. For details not disclosed in the apparatus embodiments of the present application, please refer to the method embodiments of the present application.
[0203] 9 is a block diagram of an image processing apparatus according to an exemplary embodiment of the present application. The apparatus may be implemented as all or part of an electronic device by software, hardware, or a combination of software and hardware. The apparatus may include a processing unit 910, an output unit 920, and an adjustment unit 930.
[0204] The processing unit 910 is configured to process the captured original image based on the ISP parameters to obtain a target image, the image quality of which is related to the ISP parameters.
[0205] The output unit 920 is configured to invoke the pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, where the image quality assessment model is a model obtained by training a neural network using sample images, and the target assessment result indicates the image quality of the evaluated target image.
[0206] The adjusting unit 930 is configured to adjust the ISP parameters based on the subject evaluation results.
[0207] In a possible implementation, the object assessment results include objective and subjective quality information of the object image.
[0208] The objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetics.
[0209] In another possible implementation, the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer, and the output unit 920 Input the target image into a scene classification layer to output and obtain a scene classification result; Input the target image and scene classification results to an objective scoring layer to output and obtain objective quality information, and input the target image and scene classification results to a subjective scoring layer to output and obtain subjective quality information. It is further configured as follows.
[0210] In other possible implementations, the objective quality information includes objective score values corresponding to specified dimensions, and the subjective quality information includes subjective score values corresponding to specified dimensions, the specified dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
[0211] In another possible implementation, the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise, and the adjustment unit 930 is For each specified dimension, determining a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimension based on the target score value corresponding to the specified dimension and the preset parameter threshold value. It is further structured as follows: The preset parameter threshold is a preset parameter threshold of a specified dimension for the scene type indicated by the scene classification result.
[0212] In another possible implementation, the apparatus further comprises an acquisition unit.
[0213] The obtaining unit is configured to obtain an image quality assessment model, which is obtained by training based on at least one sample data group, each sample data group including sample images and pre-labeled accurate assessment results.
[0214] In another possible implementation, the device further comprises a training unit, the training unit comprising: Obtaining a training sample set including at least one sample data group; For each of at least one sample data group, input the sample images to an original parameter model, which is a multi-task neural network model, to obtain a training result; Comparing the training results with the accurate evaluation results to obtain a computational loss that indicates the error between the training results and the accurate evaluation results; Obtain an image quality assessment model by training based on the computational losses corresponding to each of at least one sample data group. It is configured as follows.
[0215] In another possible implementation, the device further comprises a training unit, the retraining unit comprising: Performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; Perform screening on the training sample set based on the label information to obtain a screened training sample set; Train an image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model. It is configured as follows.
[0216] It should be noted that, when the device provided in the above embodiment implements the functions of the device, the division of the above functional modules is only used as an example for description. In practical application, the above functions may be assigned to different functional modules for implementation as necessary, that is, the internal structure of the device is divided into different functional modules to implement all or part of the above-mentioned functions. In addition, the device and method embodiments provided in the above embodiment belong to the same concept. For the specific implementation process thereof, please refer to the method embodiment. Details will not be described again here.
[0217] An embodiment of the present application provides a camera, the camera including an ISP and a processor, the processor configured to execute the above method executed by an electronic device.
[0218] An embodiment of the present application provides a vehicle, the vehicle including a processor and a memory configured to store instructions for execution by the processor, the method being performed by the electronic device when the processor is configured to execute the instructions.
[0219] An embodiment of the present application provides a computer program product including computer readable code or a non-volatile computer readable storage medium carrying the computer readable code, which when executed by a processor of an electronic device causes the processor of the electronic device to perform the above-mentioned method performed by the electronic device.
[0220] An embodiment of the present application provides a non-volatile computer-readable storage medium storing computer program instructions, which, when executed by a processor, implement the above-mentioned method performed by the electronic device.
[0221] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction execution device. A computer-readable storage medium can be, for example, but not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer-readable storage media include portable computer disks, hard disks, Random-Access Memory (RAM), Read-Only Memory (ROM), Electrically Programmable Read-Only Memory (EPROM, or Flash memory), Static Random-Access Memory (SRAM), portable Compact Disc Read-Only Memory (CD-ROM), Digital Video Disc (DVD), memory sticks, floppy disks, mechanically encoded devices such as punched cards or slotted structures that store instructions, and any suitable combination thereof.
[0222] The computer readable program instructions or code described herein can be downloaded by each computing / processing device from a computer readable storage medium or can be downloaded to an external computer or external storage device through a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, fiber optic transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer readable program instructions from the network and transfers the computer readable program instructions to a computer readable storage medium within each computing / processing device for storage.
[0223] The computer program instructions used to perform the operations herein may be assembly instructions, Instruction Set Architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, status setting data, or source or object code written in any combination of one or more programming languages. Programming languages include object-oriented programming languages, such as Smalltalk or C++, and traditional procedural programming languages, such as "C" or similar programming languages. The computer readable program instructions may be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. When calling a remote computer, the remote computer may be connected to the user's computer via any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., connected over the Internet using an Internet Service Provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, a Field-Programmable Gate Array (FPGA), or a Programmable Logic Array (PLA), is customized by using status information in computer-readable program instructions. The electronic circuitry may execute the computer-readable program instructions to implement various subject matter of the present application.
[0224] Various aspects of the present application are described herein with reference to flowcharts and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the present application. It should be understood that each block of the flowcharts and / or block diagrams, and combinations of blocks in the flowcharts and / or block diagrams, can be implemented by computer readable program instructions.
[0225] These computer-readable program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus such that, when the instructions are executed by the processor of the computer or other programmable data processing apparatus, the machine may appear to cause the apparatus to perform the functions / operations specified in one or more blocks in the flowcharts and / or block diagrams. These computer-readable program instructions may alternatively be stored in a computer-readable storage medium. These instructions enable a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, a computer-readable medium storing instructions includes an artifact including instructions implementing various aspects of the functions / operations specified in one or more blocks in the flowcharts and / or block diagrams.
[0226] The computer-readable program instructions may alternatively be loaded into a computer, other programmable data processing apparatus, or other device such that a series of operational steps are executed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process. Thus, the instructions executed on the computer, other programmable data processing apparatus, or other device perform the functions / operations identified in one or more blocks in the flowcharts and / or block diagrams.
[0227] The flowcharts and block diagrams in the accompanying drawings illustrate possible implementations of the system architecture, functionality, and operation of the apparatus, system, method, and computer program product according to several embodiments of the present application. Each block in the flowchart or block diagram in this context may include a module, program segment, or part of an instruction, which includes one or more executable instructions for implementing a particular logical function. In some alternative implementations, the functions marked in the blocks may alternatively be performed in a different order than that marked in the accompanying drawings. For example, two consecutive blocks may in fact be executed substantially simultaneously or may be executed in a reverse order depending on the functions involved.
[0228] It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented by hardware (e.g., a circuit or ASIC (Application-Specific Integrated Circuit)) that performs the corresponding function or operation, or may be implemented by a combination of hardware and software, e.g., firmware.
[0229] Although the present application has been described with reference to embodiments, in the process of implementing the claimed application, those skilled in the art may understand and implement other variations of the disclosed embodiments by considering the accompanying drawings, the disclosed content, and the appended claims. In the claims, "comprising" does not exclude other components or steps, and "a" or "one" does not exclude a plurality. A single processor or other unit may perform several functions recited in the claims. Although some means are recited in mutually different dependent claims, this does not mean that these means cannot be combined to produce better effects.
[0230] The above has described the embodiments of the present application. The above description is a non-exhaustive example and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements of the technology in the market, or to allow those skilled in the art to understand the embodiments disclosed herein.
[0231] This application claims priority to Chinese Patent Application No. 202111269694.5, filed with the State Intellectual Property Office of the People's Republic of China on October 29, 2021, for the invention entitled "IMAGE PROCESSING METHOD AND APPARATUS, AND STORAGE MEDIUM," and the prior Chinese patent application is hereby incorporated by reference in its entirety.
Claims
1. 1. An image processing method, comprising: processing the captured raw image based on the ISP parameters to obtain an image of the target; Invoking a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, the image quality assessment model being a model obtained by training a neural network using sample images, the target assessment result indicating the image quality of the evaluated target image; adjusting the ISP parameters based on the subject evaluation results; The method according to claim 1,
2. The object evaluation result includes objective quality information and subjective quality information of the object image; the objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetic sensations; The method of claim 1.
3. the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer; Invoking a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, inputting the target image into the scene classification layer to output and obtain a scene classification result; inputting the target image and the scene classification result into the objective scoring layer to output and obtain the objective quality information, and inputting the target image and the scene classification result into the subjective scoring layer to output and obtain the subjective quality information; having The method of claim 2.
4. The objective quality information includes objective score values corresponding to designated dimensions, and the subjective quality information includes subjective score values corresponding to the designated dimensions, the designated dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension. The method according to claim 2 or 3.
5. the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise; The step of adjusting the ISP parameters based on the subject evaluation results includes: determining, for each specified dimension, a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; adjusting the ISP parameters corresponding to the specified dimension based on the target score value and a preset parameter threshold value corresponding to the specified dimension; having the preset parameter threshold is a preset parameter threshold of the specified dimension for a scene type indicated by the scene classification result; The method according to claim 4.
6. Prior to invoking a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, the method further comprises: obtaining the image quality assessment model; The image quality assessment model is obtained by training based on at least one sample data group, each sample data group including the sample images and pre-labeled accurate assessment results.
6. The method according to any one of claims 1 to 5.
7. Prior to obtaining the image quality assessment model, the method further comprises: obtaining a training sample set including the at least one sample data group; For each of the at least one sample data group, inputting the sample images into an original parameter model, which is a multi-task neural network model, to obtain a training result; comparing the training results with the accurate evaluation results to obtain a computational loss indicative of an error between the training results and the accurate evaluation results; obtaining the image quality assessment model by training based on a computational loss corresponding to each of the at least one sample data group; Further comprising The method according to claim 6.
8. After invoking an image quality assessment model based on the target image to output and obtain a target assessment result as described above, the method further comprises: performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; performing screening on the training sample set based on the label information to obtain a screened training sample set; training the image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model; Further comprising The method of claim 7.
9. An image processing device, a processing unit configured to process the captured raw image based on the ISP parameters to obtain a target image; an output unit configured to call a pre-trained image quality assessment model based on the target image to output and obtain a target assessment result, the image quality assessment model being a model obtained by training a neural network using sample images, the target assessment result being indicative of an image quality of the evaluated target image; and an adjusting unit configured to adjust the ISP parameters based on the subject evaluation results; An apparatus having
10. The object evaluation result includes objective quality information and subjective quality information of the object image; the objective quality information indicates an assessed quality status of the target image in terms of objective indicators, and the subjective quality information indicates an assessed quality status of the target image in terms of subjective aesthetic sensations; 10. The apparatus of claim 9.
11. the image quality assessment model includes a scene classification layer, an objective scoring layer, and a subjective scoring layer; The output unit is inputting the target image into the scene classification layer to output and obtain a scene classification result; inputting the target image and the scene classification result into the objective scoring layer to output and obtain the objective quality information, and inputting the target image and the scene classification result into the subjective scoring layer to output and obtain the subjective quality information. [0023] 11. The apparatus of claim 10.
12. The objective quality information includes objective score values corresponding to designated dimensions, and the subjective quality information includes subjective score values corresponding to the designated dimensions, the designated dimensions including at least one of a brightness dimension, a color dimension, a contrast ratio dimension, and a sharpness and noise dimension.
12. Apparatus according to claim 10 or 11.
13. the ISP parameters include at least one of brightness, color, contrast ratio, sharpness, and noise; The adjustment unit includes: For each specified dimension, determining a target score value corresponding to the specified dimension based on the objective score value and the subjective score value corresponding to the specified dimension; Adjusting the ISP parameters corresponding to the specified dimension based on the target score value and a preset parameter threshold value corresponding to the specified dimension. It is further structured as follows: the preset parameter threshold is a preset parameter threshold of the specified dimension for a scene type indicated by the scene classification result; 13. The apparatus of claim 12.
14. The apparatus further comprises an acquisition unit, The acquisition unit is configured to acquire the image quality assessment model, and the image quality assessment model is acquired by training based on at least one sample data group, each sample data group including the sample images and pre-labeled accurate assessment results. Apparatus according to any one of claims 9 to 13.
15. The apparatus further comprises a training unit, the training unit comprising: obtaining a training sample set including the at least one group of sample data; For each of the at least one sample data group, inputting the sample images into an original parameter model, which is a multi-task neural network model, to obtain a training result; comparing the training results with the accurate evaluation results to obtain a computational loss indicative of an error between the training results and the accurate evaluation results; obtaining the image quality assessment model by training based on a computational loss corresponding to each of the at least one sample data group; It is configured as follows:
15. The apparatus of claim 14.
16. The apparatus further comprises a retraining unit, the retraining unit comprising: performing automatic labeling on the target image to obtain label information including objective label values corresponding to the specified dimensions; performing screening on the training sample set based on the label information to obtain a screened training sample set; training the image quality assessment model based on the screened training sample set to obtain an updated image quality assessment model. It is configured as follows:
16. The apparatus of claim 15.
17. An ISP and a processor, The processor is configured to execute a method according to any one of claims 1 to 8. camera.
18. A processor; a memory configured to store instructions to be executed by the processor; The processor, when executing the instructions, is configured to perform a method according to any one of claims 1 to 8. vehicle.
19. storing computer program instructions; The computer program instructions, when executed by a processor, perform the method according to any one of claims 1 to 8. A non-volatile computer-readable storage medium.
20. having computer readable code or a non-volatile computer readable storage medium carrying said computer readable code; When the computer readable code is executed on an electronic device, a processor within the electronic device performs the method of any one of claims 1 to 8. Computer program products.
Citation Information
Patent Citations
Image processing device, image processing method, and program
JP2016103807A
Predicting optimal values for parameters used in an operation of an image signal processor using machine learning
US20200372682A1
Image processing apparatus, image processing program, recording medium, and image processing method
WO2018216280A1
