Image signal processing method, device, equipment and storage medium

Abstract

The present application discloses an image signal processing method, device, equipment and storage medium, which relates to the field of machine learning technology, including: correcting the image signal to be processed output by the image sensor based on the image correction parameter to obtain the corresponding corrected image; performing image quality detection on the corrected image through the image quality detection model constructed based on the first convolutional neural network algorithm to obtain the corresponding image quality detection result; determining the adjustment factor for the image correction parameter based on the image quality detection result, and adjusting the image correction parameter using the adjustment factor, so as to perform image correction processing based on the image correction parameter obtained after the adjustment. The present application detects the corrected image through the image quality detection model, and then determines the adjustment factor for adjusting the image correction parameter based on the corresponding detection result, thereby realizing real-time self-optimization of image quality, quickly and accurately restoring the image, and improving the accuracy of image signal processing.

Description

Technical Field

[0001] The present invention relates to the field of machine learning technology, and in particular to an image signal processing method, device, equipment and storage medium. Background Art

[0002] At present, the biggest difficulty in debugging image parameters in image signal processing lies in multiple scenes. Each set of tuned image parameters may not be suitable for every usage scene, and manual debugging is difficult to traverse the changes in various complex scenes. The existing technical solutions generally use a combination of multiple ISP (Image Signal Process) parameters during development and the parameters of the current image to perform matching calculations to debug the image effect. Although it can solve the problems of manual debugging in complex scenes, the existing technology only adjusts the image parameters within the preset matching combination. The algorithm for changing the parameters is not flexible enough, and there is no corresponding judgment mechanism to judge the actual effect after parameter tuning. When used in actual scenes, it cannot achieve the optimal image effect in real time and accurately. Summary of the invention

[0003] In view of this, the purpose of the present invention is to provide an image signal processing method, device, equipment and storage medium, which can realize real-time self-optimization of image quality, quickly and accurately restore images, and improve the accuracy of image signal processing. The specific scheme is as follows:

[0004] In a first aspect, the present application discloses an image signal processing method, comprising:

[0005] Correcting the image signal to be processed output by the image sensor based on the image correction parameters to obtain a corresponding corrected image;

[0006] Performing image quality detection on the corrected image by using an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a corresponding image quality detection result;

[0007] An adjustment factor for the image correction parameter is determined based on the image quality detection result, and the image correction parameter is adjusted using the adjustment factor, so as to perform image correction processing based on the image correction parameter obtained after the adjustment.

[0008] Optionally, the correcting the image signal to be processed output by the image sensor based on the image correction parameter to obtain a corresponding corrected image includes:

[0009] The image signal to be processed output by the image sensor is input into the ISP module, and the image signal to be processed is corrected based on the image correction parameters in the ISP module to obtain a corrected image output by the ISP module.

[0010] Optionally, performing image quality detection on the corrected image by using an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a corresponding image quality detection result includes:

[0011] Inputting the corrected image into an image quality detection model constructed based on a first convolutional neural network algorithm to perform image quality detection, and obtaining an image quality detection result output by the image quality detection model that includes a quality score value corresponding to each image feature point in the corrected image;

[0012] Accordingly, determining the adjustment factor for the image correction parameter based on the image quality detection result includes:

[0013] The image quality detection result and the image correction parameters corresponding to the corrected image are input into a prediction model constructed based on a second convolutional neural network algorithm to determine an adjustment factor for the image correction parameters.

[0014] Optionally, after adjusting the image correction parameter by using the adjustment factor, the method further includes:

[0015] Determining a comprehensive quality value corresponding to the corrected image according to the image quality detection result;

[0016] Performing regression analysis on the comprehensive quality values ​​corresponding to the multiple frames of the corrected images to obtain corresponding regression analysis results;

[0017] Based on the regression analysis result, it is determined whether reinforcement learning is needed for the prediction model, and the prediction model is updated using the reinforcement model after reinforcement learning.

[0018] Optionally, determining whether reinforcement learning is required for the prediction model based on the regression analysis result includes:

[0019] If the regression analysis result indicates that the image quality effect corresponding to the corrected image does not meet the preset conditions, the prediction model is reinforced by using the image quality detection result, the comprehensive quality value, the image correction parameters corresponding to the corrected image, and the adjustment factor.

[0020] Optionally, performing image quality detection on the corrected image by using an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a corresponding image quality detection result includes:

[0021] Determining a static area image and a dynamic area image in the corrected image;

[0022] Image quality detection is performed on the static area image and the dynamic area image respectively by using an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a first detection result corresponding to the static area image and a second detection result corresponding to the dynamic area image.

[0023] Optionally, determining the static area image and the dynamic area image in the corrected image includes:

[0024] Determine dynamic objects and static objects in the corrected image, and obtain contour information corresponding to the dynamic objects;

[0025] The moving trajectory of the dynamic object is determined based on the contour information corresponding to the dynamic object in multiple consecutive frames, and the first area image corresponding to the contour information and / or the moving trajectory in the corrected image is determined as the dynamic area image, and the second area image corresponding to the static object in the corrected image is determined as the static area image.

[0026] In a second aspect, the present application discloses an image signal processing device, comprising:

[0027] An image correction processing module is used to perform correction processing on the image signal to be processed output by the image sensor based on the image correction parameters to obtain a corresponding corrected image;

[0028] An image quality detection module is used to perform image quality detection on the corrected image through an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a corresponding image quality detection result;

[0029] An adjustment factor determination module, configured to determine an adjustment factor for the image correction parameter based on the image quality detection result;

[0030] The correction parameter adjustment module is used to adjust the image correction parameter using the adjustment factor so as to perform image correction processing based on the image correction parameter obtained after the adjustment.

[0031] In a third aspect, the present application discloses an electronic device, comprising:

[0032] Memory, used to store computer programs;

[0033] The processor is used to execute the computer program to implement the steps of the above-disclosed image signal processing method.

[0034] In a fourth aspect, the present application discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, the steps of the aforementioned disclosed image signal processing method are implemented.

[0035] It can be seen that the present application provides an image signal processing method, including: performing image quality detection on the image signal to be processed output by the image sensor based on the image correction parameters to obtain a corresponding corrected image; performing image quality detection on the corrected image through an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a corresponding image quality detection result; determining an adjustment factor for the image correction parameter based on the image quality detection result, and adjusting the image correction parameter using the adjustment factor, so as to perform image correction processing based on the image correction parameter obtained after the adjustment. It can be seen that the present application performs image quality detection on the corrected image obtained by performing image quality detection on the image signal to be processed based on the image correction parameter through an image quality detection model constructed based on the first convolutional neural network algorithm, and then adjusts the image correction parameter using the adjustment factor for the image correction parameter determined based on the image quality detection result, so as to perform image correction processing based on the image correction parameter obtained after the adjustment, thereby achieving real-time self-optimization of image quality, fast and accurate image restoration, and improving the accuracy of image signal processing. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0037] Figure 1 A flow chart of an image signal processing method disclosed in this application;

[0038] Figure 2 A schematic diagram of image quality detection disclosed in this application;

[0039] Figure 3 A schematic diagram of image dynamic and static object separation quality detection disclosed in this application;

[0040] Figure 4 A flowchart of a specific image signal processing method disclosed in this application;

[0041] Figure 5 A schematic diagram of an overall framework of image signal processing disclosed in this application;

[0042] Figure 6 A schematic diagram of the structure of an image signal processing device disclosed in this application;

[0043] Figure 7 This is a structural diagram of an electronic device disclosed in this application. DETAILED DESCRIPTION

[0044] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0045] At present, the image signal processing parameter change algorithm in the prior art is not flexible enough, and there is no judgment mechanism for the actual effect of the parameters. It only relies on the combination of multiple ISP parameter combinations during development and the parameters of the current image for calculation, and cannot accurately achieve the actual image effect when used in actual scenes. To this end, the present application provides an image signal processing method that can achieve real-time self-optimization of image quality, quickly and accurately restore images, and improve the accuracy of image signal processing.

[0046] The embodiment of the present invention discloses a method for processing an image signal. Figure 1 As shown, the method includes:

[0047] Step S11: Correcting the image signal to be processed output by the image sensor based on the image correction parameters to obtain a corresponding corrected image.

[0048] In this embodiment, the above-mentioned image signal to be processed is an unprocessed image signal output by the image sensor after analog-to-digital conversion, and then the image signal to be processed is processed based on the image correction parameter, that is, the image signal to be processed output by the image sensor is input into the image signal processing module of each platform, and then the image signal processing module of each platform corrects the above-mentioned image signal to be processed based on the current image correction parameter to obtain a corrected image corresponding to the current image correction parameter.

[0049] Step S12: performing image quality detection on the corrected image by using an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a corresponding image quality detection result.

[0050] In this embodiment, after the image signal to be processed output by the image sensor is corrected based on the image correction parameters and the corresponding corrected image is obtained, the image quality detection of the corrected image can be performed using the image quality detection model constructed based on the first convolutional neural network algorithm to obtain the corresponding image quality detection result. It can be understood that the image quality detection model constructed based on the first convolutional neural network algorithm can perform quality detection on the corrected image, and then according to the corresponding image quality detection result, it can be known whether the image correction parameters used to correct the image signal to be processed are suitable for the current scene.

[0051] Specifically, the static area image and the dynamic area image in the corrected image are determined; the image quality detection model constructed based on the first convolutional neural network algorithm is used to perform image quality detection on the static area image and the dynamic area image to obtain a first detection result corresponding to the static area image and a second detection result corresponding to the dynamic area image. It can be understood that after determining the edge shape of the object, the image quality detection is performed by splitting the local image in the original corrected image, and the dynamic and static objects in the original corrected image are evaluated at present, that is, by distinguishing the dynamic and static areas, it is prevented from sacrificing too much optimization effect of the static area image in the process of optimizing the dynamic area image, or when optimizing the static area image, the image effect of the dynamic area image is excessively affected, and the local noise relative to the flat part of the corrected image can be further analyzed by the quality detection model, and the overall and local focus point brightness of the corrected image is calculated, and the noise, clarity, contrast, sharpness, smear, fog penetration, color difference, white balance, dynamic range, signal-to-noise ratio, focus clarity, brightness, etc. of the dynamic and static area images are evaluated by data. For example, the noise rating of a dynamic area image is 30, the clarity is 80, the contrast is 80, the ghosting is 80, etc., while the noise rating of a static area image is 80, the clarity is 90, the contrast is 90, the ghosting is 100, etc.

[0052] For example, Figure 2 As shown, when the image quality detection model performs image quality detection on the corrected image, the image data is analyzed and processed by the background point blank area clarity and noise analysis module, so as to analyze the probability distribution of noise points through the multivariate Gaussian distribution, and obtain the image analysis detection result corresponding to the background point blank area. The image data is input into the image dynamic and static object module for detection, and the image objects are classified and analyzed, and then the image object boundaries are analyzed, and the actual corrected image is analyzed, and then the clarity, edge and other quality detection results of each object in the image are judged according to the dynamic and static properties of the image. Among them, the image of the adjacent frame is used to judge whether the current image object has the moving attribute and then determine the dynamic and static attributes of the image.

[0053] Furthermore, the determining of the static area image and the dynamic area image in the corrected image may include: determining the dynamic object and the static object in the corrected image, and obtaining the contour information corresponding to the dynamic object; determining the moving trajectory of the dynamic object based on the contour information corresponding to the dynamic object in a plurality of consecutive frames, and determining the first area image corresponding to the contour information and / or the moving trajectory in the corrected image as the dynamic area image, and determining the second area image corresponding to the static object in the corrected image as the static area image. For example, see Figure 3The image dynamic and static object detection module confirms the object movement trajectory based on several consecutive frames of images, that is, the absolute value of the pixel difference between the corrected images is determined by the difference method between adjacent frames, so as to determine the dynamic objects and static objects. Then, the corrected image data is grayed out first, and then Gaussian filtered, and then the three-frame difference method is used to perform a difference operation on the two adjacent frames in the three frames and then perform a binarization process, and the corresponding binary image results after the binarization process are subjected to an AND operation. The relevant calculation formula is as follows:

[0054] R′=|I(n)-I(n-1)|∩|I(n+1)-I(n)|;

[0055] Wherein, R′ represents the result obtained by the three-frame difference method, I(n) represents the corrected image of the nth frame, I(n-1) represents the corrected image of the n-1th frame, and I(n+1) represents the corrected image of the n+1th frame.

[0056] Therefore, the result obtained by the three-frame difference method can capture the contour information of dynamic objects, and after obtaining the contour information of the moving object in a certain frame of the corrected image, the image of the dynamic object position in the original corrected image is reversely intercepted, which is slightly enlarged compared to the contour of the dynamic object, so as to evaluate the quality of the dynamic object image separately and record the dynamic evaluation result of the dynamic object image. Similarly, after judging other static objects, the quality of the static object image is evaluated and the static evaluation result of the static object image is recorded. It can be seen from this that in the image quality analysis, the object shape is reversely analyzed and restored according to the object type, and then the object movement is analyzed to judge the comprehensive quality evaluation of the dynamic and static area image parts in the entire corrected image, thereby reducing the error generated when determining the adjustment factor.

[0057] Step S13: determining an adjustment factor for the image correction parameter based on the image quality detection result, and adjusting the image correction parameter using the adjustment factor, so as to perform image correction processing based on the adjusted image correction parameter.

[0058] In this embodiment, an adjustment factor for the image correction parameter is determined based on the image quality detection result, that is, how to adjust the image correction parameter can be determined according to the image quality detection result, so the image correction parameter is adjusted using the determined adjustment factor, and then, image correction processing is performed on the next frame of image signals to be processed collected by the current image sensor device based on the adjusted image correction parameters to obtain a corrected image corresponding to the adjusted image correction parameters.

[0059] It can be seen that the embodiment of the present application performs image quality detection on the corrected image obtained by correcting the image signal to be processed based on the image correction parameters through the image quality detection model constructed based on the first convolutional neural network algorithm, and then adjusts the image correction parameters using the adjustment factors for the image correction parameters determined based on the image quality detection results, so as to perform image correction processing based on the adjusted image correction parameters, thereby achieving real-time self-optimization of image quality, quickly and accurately restoring the image, and improving the accuracy of image signal processing.

[0060] See also Figure 4 As shown, the embodiment of the present invention discloses a specific image signal processing method. Compared with the previous embodiment, this embodiment further illustrates and optimizes the technical solution.

[0061] Step S21: inputting the image signal to be processed output by the image sensor into the ISP module, and performing correction processing on the image signal to be processed based on the image correction parameters in the ISP module to obtain a corrected image output by the ISP module.

[0062] It should be pointed out that the above-mentioned ISP module can be the ISP module of each chip platform.

[0063] In this embodiment, after the image signal to be processed output by the image sensor is input into the ISP module, the ISP module performs correction processing on the image signal to be processed, and the correction processing is performed based on the image correction parameters in the ISP module, and then the ISP module outputs a corrected image corresponding to the image correction parameters, and the corrected image can be a YUV (a color encoding method) format image, that is, the ISP module outputs a frame of YUV image data corresponding to the image signal to be processed and the actual ISP image parameters corresponding to the current corrected image, such as the exposure time corresponding to the current corrected image. The relevant image data output by the ISP module can be provided to related embedded applications for processing on the one hand, and can be input into the image quality detection model constructed based on the first convolutional neural network algorithm for image quality detection on the other hand.

[0064] Step S22: Input the corrected image into an image quality detection model constructed based on the first convolutional neural network algorithm for image quality detection, and obtain an image quality detection result output by the image quality detection model that includes quality score values ​​corresponding to each image feature point in the corrected image.

[0065] It should be pointed out that the above-mentioned image feature points represent various parameters of the image, such as image noise, clarity, sharpness, etc., and when the quality score value corresponding to the feature point output by the image quality detection does not reach the ideal value, the image feature point is determined as an image missing point.

[0066] In this embodiment, after the ISP module outputs the corrected image, it can select the frame extraction optimization frequency to periodically extract frames according to the actual system performance, and input the frame information related to the corrected image into the image quality detection model constructed based on the first convolutional neural network algorithm for image quality detection, wherein the frame information may include the current corrected image and the corresponding actual ISP image parameters, etc., that is, the image quality detection model scores each image feature point in the corrected image, so as to obtain the quality score value corresponding to each image feature point in the corrected image output by the image quality detection model. For example, the image quality detection result output by the image quality detection model may include but is not limited to: the quality score values ​​of image feature points such as noise, dynamic range, clarity, sharpness, smear, color difference, brightness, white balance, and focus clarity of the dynamic area image; the quality score values ​​of image feature points such as noise, dynamic range, brightness, clarity, sharpness, color difference, white balance, and focus clarity of the static area image, the quality score values ​​of noise and clarity of the flat area image, etc.

[0067] Step S23: inputting the image quality detection result and the image correction parameters corresponding to the corrected image into a prediction model constructed based on a second convolutional neural network algorithm to determine an adjustment factor for the image correction parameters.

[0068] It should be pointed out that the above-mentioned prediction model is adapted to the ISP-related processing algorithms such as the 3A algorithm of each chip platform during the training process of the machine learning model. The 3A algorithm includes: AWB (Automatic White Balance) algorithm, AE (Auto Exposure) algorithm, and AF (Auto Focus) algorithm.

[0069] It can be understood that the prediction model constructed based on the second convolutional neural network algorithm analyzes the data output by the image quality detection model, and then comprehensively predicts the specific modules and specific parameter information that should be modified, and then synchronizes the predicted information to the above-mentioned ISP module, so that the above-mentioned ISP module can update the image correction parameters based on the information predicted by the prediction module. For example, the image quality detection model outputs that the dynamic area image drag evaluation of 80 in the current corrected image is relatively serious, and the noise evaluation is 30. The clarity score of 60 is not as expected, the edge sharpness is 40. The contrast is 40 is low, and the long frame and short frame exposure time of the current corrected image, such as 200ms for long frames and 10ms for short frames, and other actual ISP image parameter information are obtained. The ISP-related data and image quality score values ​​are input into the prediction model constructed based on the second convolutional neural network algorithm for processing. The above-mentioned prediction model can predict the data that should be adjusted for the current corrected image. The prediction model is divided into Analyze whether the quality score value corresponding to the corrected image reaches each calibration threshold, so as to judge whether it is necessary to turn on the wide dynamic mode, whether it is necessary to increase or decrease the exposure time, whether it is necessary to re-adjust the current Gamma curve according to the current scene light and other related ISP image parameters provided by the chip platform, that is, the image correction parameters of the ISP module, such as changing the chip platform AE exposure time and exposure gain to reduce dynamic blur and dynamic smear, etc. At the same time, strengthen the noise reduction module, such as 3DNR (3DDenoising), adjust the chip platform-related AEroute, DPCC (Defect Pixel Cluster Correction), 3DNR and other parameters, so as to make a new balance between noise, smear and sharpening.

[0070] Step S24: using the adjustment factor to adjust the image correction parameter, so as to perform image correction processing based on the adjusted image correction parameter.

[0071] For the specific content of the above step S24, reference may be made to the corresponding content disclosed in the above embodiments, which will not be repeated here.

[0072] In this embodiment, after the image correction parameters are adjusted using the adjustment factor so as to perform image correction processing based on the adjusted image correction parameters, the method may further include: determining the comprehensive quality value corresponding to the corrected image according to the image quality detection result; performing regression analysis on the comprehensive quality values ​​corresponding to multiple frames of the corrected images to obtain corresponding regression analysis results; determining whether it is necessary to perform reinforcement learning on the prediction model based on the regression analysis results, and updating the prediction model using the reinforcement model after reinforcement learning. If the regression analysis result indicates that the image quality effect corresponding to the corrected image does not meet the preset conditions, the prediction model is reinforced by using the image quality detection result, the comprehensive quality value, the image correction parameters corresponding to the corrected image, and the adjustment factor.

[0073] It can be understood that by regression analyzing the quality detection results of the first few frames of corrected images, it is possible to determine whether the adjustment factor determined by the current prediction model can enable the ISP module to accurately restore the image. That is, after adjusting the image correction parameters in the ISP module using the adjustment factor, the quality detection results of the first few frames of corrected images are regressed and analyzed to determine whether the prediction model needs reinforcement learning. It should be pointed out that when the image quality detection model outputs the image quality detection result corresponding to the corrected image, that is, the quality score values ​​of the image feature points such as noise, dynamic range, clarity, sharpness, smear, chromatic aberration, brightness, white balance, and focus clarity of the dynamic area image; after scoring the quality score values ​​of the image feature points such as noise, dynamic range, brightness, clarity, sharpness, chromatic aberration, white balance, and focus clarity of the static area image, the dynamic total score corresponding to the dynamic area image is calculated according to the image feature point score values ​​corresponding to the above-mentioned dynamic area image according to the weight ratio of each image feature point score value. Similarly, the static total score corresponding to the static area image is calculated, and then the dynamic total score and the static total score are used to calculate the comprehensive quality value of the corrected image according to the custom weights.

[0074] For example, the data corresponding to the corrected images of the eight frames before and after the regression analysis, the comprehensive quality value of the first frame of the corrected image evaluated after the image quality detection is 60, the image correction parameter corresponding to the ISP module used by the second frame of the corrected image is the adjustment factor determined by the prediction module for the first frame of the corrected image, and the image correction parameter corresponding to the ISP module is adjusted by the adjustment factor, and the second frame of the corrected image is the image reprocessed by adjusting the image correction parameter corresponding to the ISP module by the adjustment factor, and the corresponding comprehensive quality value is 65, similarly, the third frame is 70, the fourth frame is 69, the fifth frame is 67, the sixth frame is 62, the seventh frame is 61, and the eighth frame is 62. The comprehensive quality values ​​corresponding to the eight frames of the corrected images are subjected to regression analysis, and it is analyzed that the quality effects corresponding to the eight frames of the corrected images show a continuous downward trend, and the decline amplitude and the number of declines are greater than the preset comprehensive threshold, indicating that the adjustment factor determined by the prediction module fails to actually optimize the image signal to be processed in the current scene. Among them, the above-mentioned preset comprehensive threshold can represent the tolerance of the continuous decline amplitude and the number of declines. Therefore, it is necessary to perform reinforcement learning on the prediction model, that is, to perform reinforcement learning on the prediction model using the image quality detection result, the comprehensive quality value, the image correction parameter corresponding to the corrected image, and the adjustment factor.

[0075] For another example, five frames of images are collected continuously, where the comprehensive score corresponding to the first frame image I(n) is 40, the comprehensive score corresponding to I(n+1) is 70, and similarly, the comprehensive score corresponding to I(n+2) is 60, the comprehensive score corresponding to I(n+3) is 50, and the comprehensive score corresponding to I(n+4) is 40. According to the above comprehensive scores, the comprehensive optimization effect of these five consecutive frames of images shows a downward trend, that is, the current prediction model needs to be optimized offline, and then the image quality detection results corresponding to I(n) (noise 30, clarity 60, ghosting 8) are calculated. 0, contrast 60, brightness 80, etc.), the actual ISP image parameters corresponding to I(n), the adjustment factors corresponding to I(n) and the comprehensive score value 40, the image quality detection results corresponding to I(n+1) (noise 40, clarity 80, contrast 80, smear 90, etc.), the actual ISP image parameters corresponding to I(n+1) and the last adjustment factors and other parameters are input into the reinforcement model so that the prediction model can learn the numerical direction in which the single and combined parameters should be optimized. By training the local model according to the actual image data, it is more targeted. Alternatively, the results of the five image processings before and after are recorded at the same time, and the difference of the entire result is taken to determine whether the real overall effect optimization is generated. If the optimization effect increases less or decreases, it means that the adjustment factors determined by the current prediction model are used to adjust the image correction parameters of the ISP module and fail to achieve the actual image optimization effect. For example, if the overall quality of the five frames of images I(n)>I(n+1)>I(n+2)>I(n+3)>I(n+4) shows a downward trend, or the quality values ​​of some frames show a downward trend, the prediction model needs to be optimized and trained, so that the data sets corresponding to the five image processing are input into the offline learning model for training in the prediction direction. For example, the adjustment factors, quality detection results, and actual ISP image parameters determined by the prediction model corresponding to the five frames of corrected images are used as training data to perform reinforcement learning on the preset model. It can be seen that by training the prediction model through offline reinforcement learning, the offline self-update of the prediction model can be achieved, which is more suitable for the long-term self-optimization of image equipment for certain actual scenes, and the model training can be more targeted in actual scenes.

[0076] It can be seen that the embodiment of the present application performs image quality detection on the corrected image obtained by correcting the image signal to be processed based on the image correction parameters through the image quality detection model constructed based on the first convolutional neural network algorithm, and then determines the adjustment factor for the image correction parameter based on the prediction model constructed based on the second convolutional neural network algorithm based on the image quality detection result, and then uses the above adjustment factor to adjust the image correction parameter, so as to perform image correction processing based on the image correction parameter obtained after adjustment, that is, the prediction model is trained by the machine learning model, the image quality is detected in real time by the convolutional neural network model, and the image quality is self-optimized in real time by the prediction model, and the prediction model will be reinforced when used in actual use. Self-optimization is performed in the user's common scenes to achieve automatic processing of image type devices in complex scenes, so as to quickly and accurately restore the image and improve the accuracy of image signal processing. In addition, the embodiment of the present application combines image processing with actual image quality analysis, and uses relevant algorithm models to adjust parameters and regression analysis, so as to more targetedly perform image correction optimization for complex scenes.

[0077] like Figure 5As shown, first, the unprocessed image signal after analog-to-digital conversion is input by the sensor (image sensor) to the ISP module of each chip platform, and the image signal is corrected and processed by the ISP module, and the YUV format data (corrected image) is output, and the actual ISP-related real-time parameters are also attached, such as the exposure time of the current YUV format data, etc. The YUV format data can be provided to the embedded application for processing, or the frame extraction optimization frequency is selected according to the actual performance to extract the frame regularly, and the relevant frame information containing the current actual ISP image parameters is input into the algorithm model (the quality detection model constructed based on the first convolutional neural network algorithm) for image quality detection to evaluate the quality effect of the entire corrected image. And through the Dyna-2 algorithm framework, the parameter prediction and the image data analysis during the image quality detection are separated, and the combination of permanent memory and instantaneous memory is used to carry out the reinforcement learning of the model. That is, an algorithm model constructed based on a convolutional neural network algorithm is used to analyze the image quality of each feature point of the image, perform image quality evaluation, and obtain corresponding quality detection results. Then, an adjustment factor for adjusting the image correction parameters of the ISP module is determined based on the quality detection results through an ISP parameter prediction module, so as to use the adjustment factor to adjust the image correction parameters of the ISP module and improve the correction effect of the image. Then, it is also possible to determine whether reinforcement learning is required through regression analysis of the quality detection results of multiple frames of image data. If the ISP parameter prediction module is required, the relevant training data is input into the algorithm model reinforcement learning module to complete the reinforcement learning of the ISP parameter prediction model and update the prediction model using the model after reinforcement learning. Among them, the above-mentioned training data may include the adjustment factor determined by the ISP parameter prediction model corresponding to the corrected image, the quality detection result, the actual ISP image parameters and other related frame information.

[0078] Correspondingly, the present application also discloses an image signal processing device, see Figure 6 As shown, the device comprises:

[0079] The image correction processing module 11 is used to perform correction processing on the image signal to be processed output by the image sensor based on the image correction parameters to obtain a corresponding corrected image;

[0080] An image quality detection module 12 is used to perform image quality detection on the corrected image through an image quality detection model constructed based on a first convolutional neural network algorithm to obtain a corresponding image quality detection result;

[0081] An adjustment factor determination module 13, configured to determine an adjustment factor for the image correction parameter based on the image quality detection result;

[0082] The correction parameter adjustment module 14 is used to adjust the image correction parameter using the adjustment factor, so as to perform image correction processing based on the image correction parameter obtained after the adjustment.

[0083] As can be seen from the above, in this embodiment, an image quality detection model constructed based on the first convolutional neural network algorithm is used to perform image quality detection on the corrected image obtained by correcting the image signal to be processed based on the image correction parameters, and then the image correction parameters are adjusted using the adjustment factors for the image correction parameters determined based on the image quality detection results, so as to perform image correction processing based on the adjusted image correction parameters, thereby achieving real-time self-optimization of image quality, quickly and accurately restoring the image, and improving the accuracy of image signal processing.

[0084] In some specific embodiments, the image correction processing module 11 may specifically include:

[0085] The image correction unit is used to input the image signal to be processed output by the image sensor into the ISP module, and perform correction processing on the image signal to be processed based on the image correction parameters in the ISP module to obtain the corrected image output by the ISP module.

[0086] In some specific embodiments, the image quality detection module 12 may specifically include:

[0087] The first quality detection unit is used to input the corrected image into an image quality detection model constructed based on a first convolutional neural network algorithm to perform image quality detection, and obtain an image quality detection result output by the image quality detection model that includes a quality score value corresponding to each image feature point in the corrected image.

[0088] In some specific embodiments, the adjustment factor determination module 13 may specifically include:

[0089] An adjustment factor determination unit is used to input the image quality detection result and the image correction parameters corresponding to the corrected image into a prediction model constructed based on a second convolutional neural network algorithm to determine an adjustment factor for the image correction parameters.

[0090] In some specific embodiments, after adjusting the image correction parameter by using the adjustment factor, the method may specifically include:

[0091] A comprehensive quality determination unit, configured to determine a comprehensive quality value corresponding to the corrected image according to the image quality detection result;

[0092] A regression analysis unit, used for performing regression analysis on the comprehensive quality values ​​corresponding to the multiple frames of the corrected images to obtain corresponding regression analysis results;

[0093] A reinforcement learning unit is used to determine whether reinforcement learning is needed for the prediction model based on the regression analysis result, and to update the prediction model using the reinforcement model after reinforcement learning.

[0094] In some specific embodiments, the reinforcement learning unit may specifically include:

[0095] The reinforcement learning subunit is used to perform reinforcement learning on the prediction model using the image quality detection result, the comprehensive quality value, the image correction parameters corresponding to the corrected image and the adjustment factor when the regression analysis result indicates that the image quality effect corresponding to the corrected image does not meet the preset conditions.

[0096] In some specific embodiments, the image quality detection module 12 may specifically include:

[0097] A dynamic and static image determination unit, used to determine a static area image and a dynamic area image in the corrected image;

[0098] The second quality detection unit is used to perform image quality detection on the static area image and the dynamic area image respectively through an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a first detection result corresponding to the static area image and a second detection result corresponding to the dynamic area image.

[0099] In some specific embodiments, the dynamic and static image determination unit may specifically include:

[0100] a dynamic and static object determination subunit, used to determine dynamic objects and static objects in the corrected image;

[0101] A contour information acquisition subunit, used to acquire contour information corresponding to the dynamic object;

[0102] A movement trajectory determination subunit is used to determine the movement trajectory of the dynamic object based on the contour information corresponding to the dynamic object in multiple consecutive frames, and determine the first area image corresponding to the contour information and / or the movement trajectory in the corrected image as the dynamic area image, and determine the second area image corresponding to the static object in the corrected image as the static area image.

[0103] Furthermore, an embodiment of the present application also provides an electronic device. Figure 7This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content in the diagram cannot be regarded as any limitation on the scope of use of the present application.

[0104] Figure 7 The present invention provides a schematic diagram of the structure of an electronic device 20 provided in an embodiment of the present application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 is used to store a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the image signal processing method disclosed in any of the above embodiments. In addition, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0105] In this embodiment, the power supply 23 is used to provide working voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and the external device, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 25 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0106] In addition, the memory 22, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon can include an operating system 221, a computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0107] The operating system 221 is used to manage and control the hardware devices and computer programs 222 on the electronic device 20, and may be Windows Server, Netware, Unix, Linux, etc. In addition to the computer program that can be used to complete the image signal processing method performed by the electronic device 20 disclosed in any of the aforementioned embodiments, the computer program 222 may further include computer programs that can be used to complete other specific tasks.

[0108] Furthermore, an embodiment of the present application also discloses a storage medium, in which a computer program is stored. When the computer program is loaded and executed by a processor, the steps of the image signal processing method disclosed in any of the aforementioned embodiments are implemented.

[0109] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar areas between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method area description.

[0110] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the presence of other identical elements in the process, method, article or device including the elements.

[0111] The above is a detailed introduction to an image signal processing method, device, equipment and storage medium provided by the present invention. Specific examples are used in this article to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the method of the present invention and its core idea. At the same time, for a person skilled in the art, according to the idea of ​​the present invention, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present invention.

Claims

1. A method for processing an image signal, characterized in that: include: Correcting the image signal to be processed output by the image sensor based on the image correction parameters to obtain a corresponding corrected image; Performing image quality detection on the corrected image by using an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a corresponding image quality detection result; Determining an adjustment factor for the image correction parameter based on the image quality detection result, and adjusting the image correction parameter using the adjustment factor, so as to perform image correction processing based on the image correction parameter obtained after the adjustment; The image quality detection model constructed based on the first convolutional neural network algorithm is used to perform image quality detection on the corrected image to obtain a corresponding image quality detection result, including: Inputting the corrected image into an image quality detection model constructed based on a first convolutional neural network algorithm to perform image quality detection, and obtaining an image quality detection result output by the image quality detection model that includes a quality score value corresponding to each image feature point in the corrected image; Accordingly, determining the adjustment factor for the image correction parameter based on the image quality detection result includes: Inputting the image quality detection result and the image correction parameters corresponding to the corrected image into a prediction model constructed based on a second convolutional neural network algorithm to determine an adjustment factor for the image correction parameters; After the image correction parameter is adjusted by using the adjustment factor, the method further includes: Determining a comprehensive quality value corresponding to the corrected image according to the image quality detection result; Performing regression analysis on the comprehensive quality values ​​corresponding to the multiple frames of the corrected images to obtain corresponding regression analysis results; Based on the regression analysis result, it is determined whether reinforcement learning is needed for the prediction model, and the prediction model is updated using the reinforcement model after reinforcement learning.

2. The image signal processing method according to claim 1, characterized in that: The method of performing correction processing on the image signal to be processed output by the image sensor based on the image correction parameter to obtain a corresponding corrected image includes: The image signal to be processed output by the image sensor is input into the ISP module, and the image signal to be processed is corrected based on the image correction parameters in the ISP module to obtain a corrected image output by the ISP module.

3. The image signal processing method according to claim 1, characterized in that: The determining whether it is necessary to perform reinforcement learning on the prediction model based on the regression analysis result includes: If the regression analysis result indicates that the image quality effect corresponding to the corrected image does not meet the preset conditions, the prediction model is reinforced by using the image quality detection result, the comprehensive quality value, the image correction parameters corresponding to the corrected image, and the adjustment factor.

4. The image signal processing method according to any one of claims 1 to 3, characterized in that: The image quality detection model constructed based on the first convolutional neural network algorithm is used to perform image quality detection on the corrected image to obtain a corresponding image quality detection result, including: Determining a static area image and a dynamic area image in the corrected image; Image quality detection is performed on the static area image and the dynamic area image respectively by using an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a first detection result corresponding to the static area image and a second detection result corresponding to the dynamic area image.

5. The image signal processing method according to claim 4, characterized in that: The determining of the static area image and the dynamic area image in the corrected image comprises: Determine dynamic objects and static objects in the corrected image, and obtain contour information corresponding to the dynamic objects; The moving trajectory of the dynamic object is determined based on the contour information corresponding to the dynamic object in multiple consecutive frames, and the first area image corresponding to the contour information and / or the moving trajectory in the corrected image is determined as the dynamic area image, and the second area image corresponding to the static object in the corrected image is determined as the static area image.

6. An image signal processing device, characterized in that: include: An image correction processing module is used to perform correction processing on the image signal to be processed output by the image sensor based on the image correction parameters to obtain a corresponding corrected image; An image quality detection module is used to perform image quality detection on the corrected image through an image quality detection model constructed based on the first convolutional neural network algorithm to obtain a corresponding image quality detection result; An adjustment factor determination module, configured to determine an adjustment factor for the image correction parameter based on the image quality detection result; A correction parameter adjustment module, used to adjust the image correction parameter using the adjustment factor, so as to perform image correction processing based on the image correction parameter obtained after the adjustment; The image quality detection module specifically includes a first quality detection unit, which is used to input the corrected image into an image quality detection model constructed based on a first convolutional neural network algorithm to perform image quality detection, and obtain an image quality detection result output by the image quality detection model that includes a quality score value corresponding to each image feature point in the corrected image; The adjustment factor determination module specifically includes an adjustment factor determination unit, which is used to input the image quality detection result and the image correction parameter corresponding to the corrected image into a prediction model constructed based on a second convolutional neural network algorithm to determine an adjustment factor for the image correction parameter; After the image correction parameter is adjusted by using the adjustment factor, the method further comprises: A comprehensive quality determination unit, configured to determine a comprehensive quality value corresponding to the corrected image according to the image quality detection result; A regression analysis unit, used for performing regression analysis on the comprehensive quality values ​​corresponding to the multiple frames of the corrected images to obtain corresponding regression analysis results; A reinforcement learning unit is used to determine whether reinforcement learning is needed for the prediction model based on the regression analysis result, and to update the prediction model using the reinforcement model after reinforcement learning.

7. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, configured to execute the computer program to implement the steps of the image signal processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that: Used to store computer programs; wherein, when the computer program is executed by a processor, the steps of the image signal processing method according to any one of claims 1 to 5 are implemented.

Images