Image processing method and device, equipment, storage medium and product

Abstract

The invention belongs to the technical field of image vision, and discloses an image processing method and device, equipment, a storage medium and a product. According to the invention, image processing is carried out on the collected image signal according to the first image processing intensity to generate the first input image, and image processing is carried out on the image signal according to the second image processing intensity to generate the second input image; performing model reasoning on the first input image to generate a prediction image, and performing algorithm processing on the second input image to generate a processing image; and fusing the predicted image and the processed image to generate an output image. Since the acquired images are respectively processed by using the model and the traditional algorithm, and the image is output after the results of the two paths of processing are fused, the processing intensity of the two paths of output can be dynamically adjusted according to the actual demand, thereby ensuring that a better image effect and resource overhead can be obtained, and improving the image quality. Therefore, the processing effect of the convolutional neural network applied to the image processing field is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the image vision technical field, and particularly relates to an image processing method, device, equipment, storage medium and product. BACKGROUND

[0002] At present, the neural network has been gradually applied in the image processing field, can down-sample the input image into a smaller resolution image, and processes the down-sampled smaller image through the convolutional neural network (CNN), compared with processing the original resolution image, processing the down-sampled image can process low resolution low bit width data, and can save the calculation time consumption and memory resources.

[0003] However, the whole technology cannot output the full resolution image, and the image information will be lost; the saving degree of the calculation in the image processing process is insufficient only by relying on the down-sampling, and the multiple is generally fixed when the down-sampling, the processing is not flexible enough, and the optimization of the effect and resources under different conditions cannot be achieved, finally leading to the poor processing effect of the convolutional neural network applied in the image processing field. SUMMARY

[0004] The main purpose of the present application is to provide an image processing method, device, equipment, storage medium and product, and aims to solve the technical problem of poor processing effect of the convolutional neural network applied in the image processing field in the prior art.

[0005] To achieve the above purpose, the present application provides an image processing method, which comprises the following steps:

[0006] According to the first image processing intensity, the image signal collected is processed to generate a first input image, and according to the second image processing intensity, the image signal is processed to generate a second input image;

[0007] The first input image is subjected to model inference to generate a predicted image, and the second input image is subjected to algorithm processing to generate a processed image;

[0008] The predicted image and the processed image are fused to generate an output image.

[0009] In a possible implementation manner of the present application, the first input image is subjected to image prediction to generate a predicted image, which comprises:

[0010] The first input image is subjected to image preprocessing to generate a preprocessed image, and the image preprocessing comprises image resolution reduction and / or image data bit width compression;

[0011] The preprocessed image is subjected to model inference to generate a predicted image.

[0012] In one possible implementation of this application, at least one of the following is satisfied:

[0013] The first image processing intensity is less than or equal to the second image processing intensity;

[0014] The image processing functions include at least one of temporal domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, dehazing, deblurring, and sharpening;

[0015] The same processing function is used to generate the first input image and the second input image.

[0016] In one possible implementation of this application, the model inference is performed by a neural network module, and the algorithm processing is performed by a conventional algorithm module;

[0017] Meet at least one of the following:

[0018] The processing functions of the neural network module include at least one of temporal domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, dehazing, deblurring, and sharpening.

[0019] The processing functions of the traditional algorithm module include at least one of temporal domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, dehazing, deblurring, and sharpening;

[0020] The processing functions of the neural network module include those of the traditional algorithm module;

[0021] The processing effect of the model inference is adjusted by controlling the network parameters of the model used in the neural network module. The network parameters are determined based on the first image processing intensity and / or the second image processing intensity, or based on the first input image and / or the second input image.

[0022] In one possible implementation of this application, the model inference is performed by a neural network module, and the image processing is performed by an image processing module;

[0023] If the neural network module includes the processing function of the image processing module, then the first image processing intensity is less than the second image processing intensity.

[0024] If the neural network module does not include the processing function of the image processing module, then the first image processing intensity is less than or equal to the second image processing intensity.

[0025] In one possible implementation of this application, fusing the predicted image and the processed image to generate an output image includes:

[0026] The predicted image and the processed image are fused to generate a fused image;

[0027] The fused image is then subjected to post-image processing to generate an output image.

[0028] In one possible implementation of this application, the step of performing image processing on the acquired image signal according to a first image processing intensity to generate a first input image, and performing image processing on the image signal according to a second image processing intensity to generate a second input image, includes:

[0029] The acquired image signal is processed to generate a pre-image signal;

[0030] The preceding image signal is processed to generate a first input image and a second input image.

[0031] In one possible implementation of this application, fusing the predicted image and the processed image to generate an output image includes:

[0032] Obtain the weight coefficients corresponding to the predicted image and the processed image, respectively;

[0033] The predicted image and the processed image are weighted and fused based on the weight coefficients to obtain the output image.

[0034] In one possible implementation of this application, the weighting coefficient is a preset value, determined based on the image brightness corresponding to the predicted image and the processed image, determined based on the edge intensity corresponding to the predicted image and the processed image, or determined based on the image processing intensity corresponding to the first input image and the second input image.

[0035] In one possible implementation of this application, the step of weighted fusing of the predicted image and the processed image based on the weight coefficients to obtain an output image includes:

[0036] The predicted image is filtered, upsampled, and / or bit-width increased to generate a processed predicted image.

[0037] The processed predicted image and the processed image are weighted and fused based on the weight coefficients to obtain the output image.

[0038] In one possible implementation of this application, the algorithm processing includes pass-through processing, single-stage filtering processing, multi-stage series filtering processing, or multi-stage parallel filtering processing.

[0039] Furthermore, to achieve the above objectives, this application also proposes an image processing apparatus, the image processing apparatus comprising:

[0040] The image processing module is used to perform image processing on the acquired image signal according to a first image processing intensity to generate a first input image, and to perform image processing on the image signal according to a second image processing intensity to generate a second input image;

[0041] The image generation module is used to perform model inference on the first input image to generate a predicted image, and to perform algorithm processing on the second input image to generate a processed image;

[0042] An image synthesis module is used to fuse the predicted image and the processed image to generate an output image.

[0043] In addition, to achieve the above objectives, this application also proposes an image processing apparatus, which includes: a processor, a memory, and an image processing program stored in the memory and executable on the processor, wherein the image processing program, when executed by the processor, implements the steps of the image processing method described above.

[0044] In addition, to achieve the above objectives, this application also proposes a computer-readable storage medium storing an image processing program, which, when executed, implements the steps of the image processing method described above.

[0045] In addition, to achieve the above objectives, this application also proposes a computer program product, which includes an image processing program that, when executed, implements the steps of the image processing method described above.

[0046] This application generates a first input image by processing the acquired image signal according to a first image processing intensity, and generates a second input image by processing the image signal according to a second image processing intensity. The first input image is then used for model inference to generate a predicted image, and the second input image is processed using an algorithm to generate a processed image. Finally, the predicted image and the processed image are fused to generate an output image. Because the acquired image is processed separately using both a model and a traditional algorithm, and the results of both processing are fused before outputting the image, the processing intensity of the two outputs can be dynamically adjusted according to actual needs. This ensures better image quality and lower resource consumption, thereby improving the processing performance of convolutional neural networks in the field of image processing. Attached Figure Description

[0047] Figure 1 This is a schematic diagram of the structure of an electronic device in the hardware operating environment involved in the embodiments of this application;

[0048] Figure 2 This is a flowchart illustrating the first embodiment of the image processing method of this application;

[0049] Figure 3 This is a schematic diagram showing the functional modules of an image processing device according to an embodiment of this application;

[0050] Figure 4 This is a schematic diagram of the execution flow of the first image processing module according to an embodiment of this application;

[0051] Figure 5 This is a schematic diagram illustrating the execution of an extension of the first image processing module according to an embodiment of this application;

[0052] Figure 6 This is a schematic diagram of a multi-level filtering execution process according to an embodiment of this application;

[0053] Figure 7 This is a flowchart illustrating the second embodiment of the image processing method of this application;

[0054] Figure 8 This is a schematic diagram of image synthesis filtering according to an embodiment of this application;

[0055] Figure 9 This is a schematic diagram of the first image fusion processing flow according to an embodiment of this application;

[0056] Figure 10 This is a schematic diagram of the second image fusion processing flow according to an embodiment of this application;

[0057] Figure 11 This is a schematic diagram of the sensor arrangement and spectral response curve according to an embodiment of this application;

[0058] Figure 12 This is a structural block diagram of the first embodiment of the image processing apparatus of this application.

[0059] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0060] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of this application.

[0061] Reference Figure 1 , Figure 1 This is a schematic diagram of the image processing device structure in the hardware operating environment of the embodiment of this application.

[0062] like Figure 1As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be high-speed random access memory (RAM) or stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0063] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0064] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and an image processing program.

[0065] exist Figure 1 In the electronic device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and memory 1005 in the electronic device of this application can be set in the image processing device. The electronic device calls the image processing program stored in the memory 1005 through the processor 1001 and executes the image processing method provided in the embodiment of this application.

[0066] This application provides an image processing method, referring to... Figure 2 , Figure 2 This is a flowchart illustrating a first embodiment of an image processing method according to this application.

[0067] In this embodiment, the image processing method includes the following steps:

[0068] Step S10: Perform image processing on the acquired image signal according to the first image processing intensity to generate a first input image, and perform image processing on the image signal according to the second image processing intensity to generate a second input image.

[0069] It should be noted that the execution subject of this embodiment can be the image processing device, which can be a personal computer, server or other electronic device, or other device that can achieve the same or similar functions. This embodiment does not limit this. In this embodiment and the following embodiments, the image processing device is used as an example to describe the image processing method of this application.

[0070] It should be noted that the image processing functions can include at least one of temporal noise reduction, spatial noise reduction, tone mapping, enhancement, dehazing, deblurring, and sharpening. The image processing intensity can be the degree of processing when performing image processing on the acquired image signal. For example, if the image processing is temporal noise reduction, then the image processing intensity is the temporal noise reduction intensity; if the image processing is contrast enhancement, brightness enhancement, or other enhancement processing, then the image processing intensity is the magnitude of image enhancement.

[0071] The first image processing intensity is less than or equal to the second image processing intensity, and the processing functions used to generate the first and second input images are the same. For example, both the first and second input images can be generated using temporal noise reduction and sharpening. The acquired image signals have the same image format as the first and second input images, and the image resolution and data bit width of the first and second input images are the same.

[0072] Step S20: Perform model inference on the first input image to generate a predicted image, and perform algorithm processing on the second input image to generate a processed image.

[0073] In practical applications, the prediction image is generated by performing model inference on the first input image. This can be achieved by using a pre-trained neural network model (such as a convolutional neural network model or a similar model) to process the first input image and generate the prediction image.

[0074] When the neural network model processes the first input image, the processing method (i.e., processing function) may include at least one of temporal denoising, spatial denoising, tone mapping, enhancement, dehazing, deblurring, and sharpening. The image resolution of the predicted image is no greater than the image resolution of the acquired image signal, and the data bit width of the predicted image is no greater than the data bit width of the acquired image signal.

[0075] In practical applications, the second input image is processed using algorithms to generate a processed image. This can be achieved by using traditional image processing algorithms to process the second input image and generate the processed image.

[0076] Traditional image processing algorithms can perform at least one of the following: temporal denoising, spatial denoising, tone mapping, enhancement, dehazing, deblurring, and sharpening. The image resolution of the processed image is consistent with the image resolution of the acquired image signal, and the data bit width of the predicted image is consistent with the data bit width of the acquired image signal.

[0077] In practical applications, model inference can be performed by the neural network module, while algorithm processing can be performed by the traditional algorithm module. The processing functions of the neural network module can include at least one of temporal denoising, spatial denoising, tone mapping, enhancement, dehazing, deblurring, and sharpening; the processing functions of the traditional algorithm module can include at least one of temporal denoising, spatial denoising, tone mapping, enhancement, dehazing, deblurring, and sharpening.

[0078] The processing functions of the model inference can include all the processing functions of the traditional algorithm module, and can also include other processing functions. For example, the processing functions of the traditional algorithm module can include noise reduction and enhancement. In this case, the processing functions of the neural network module can include not only noise reduction and enhancement, but also other processing functions such as dehazing and sharpening.

[0079] The processing effect of model inference can be adjusted by controlling the network parameters of the model (i.e., the pre-trained neural network model mentioned above) used in the neural network module. The network parameters can be determined based on the first image processing intensity and / or the second image processing intensity. For example, the network parameters can be determined based on the numerical range of the first image processing intensity or the second image processing intensity, or based on the difference between the first image processing intensity and the second image processing intensity.

[0080] Of course, network parameters can also be determined based on the generated first input image and / or second input image, for example: based on the image brightness, edge intensity, etc. of the first input image or the second input image, or based on the image differences between the first input image and the second input image (such as differences in image brightness, edge intensity, etc.).

[0081] The specific control method for the network parameters in the control model (i.e., the pre-trained neural network model mentioned above) can be switching network parameters, correcting network processing hyperparameters, etc. If necessary, other control methods that can achieve the same effect can also be included. This embodiment does not limit this.

[0082] In practical applications, model inference can be performed by the neural network module, algorithm processing by the traditional algorithm module, and image processing by the image processing module. The processing capabilities of the neural network module may include those of the image processing module. In this case, the more powerful processing capabilities of the model can be utilized to reduce the processing load on the related image processing modules. This ensures that the design complexity of the related image processing modules is reduced while achieving the same processing effect, or that the processing effect is improved without increasing resources. In this situation, the first image processing intensity can be set lower than the second image processing intensity.

[0083] If the neural network module does not have the processing function of the image processing module, then it is necessary to ensure that the first image processing intensity is less than or equal to the second image processing intensity.

[0084] The reason for the variable strength is that the neural network module has stronger processing power, more flexible control over the processing strength, and a larger solution space for the algorithm, resulting in better processing performance.

[0085] For example, when the image processing module (i.e., the image processing related functional module) performs temporal denoising, the neural network module (i.e., the neural network model used for model inference, or the related functional module that performs model inference, referred to as AI) performs temporal denoising + spatial denoising, and the traditional algorithm module (i.e., the related functional module that performs algorithm processing, referred to as TD) performs spatial denoising, then because the neural network module supplements the processing of the temporal denoising function, and the processing effect of the neural network module is better than that of the image processing module in this function, a better processing effect can be obtained. Therefore, the first image processing intensity (temporal denoising intensity) can be weaker than the second image processing intensity.

[0086] However, if the image processing module performs temporal denoising and the neural network module and the traditional algorithm module perform spatial denoising, the intensity of the first image processing can be less than that of the second image processing.

[0087] In one possible implementation of this embodiment, since the higher the image resolution and the larger the data bit width of the image during image processing, the more neural network computing resources are consumed, in order to ensure that the neural network computing resources of the image processing device can be saved when necessary, the step of performing model inference on the first input image to generate a predicted image in this embodiment may include:

[0088] The first input image is preprocessed to generate a preprocessed image;

[0089] The preprocessed image is used for model inference to generate a predicted image.

[0090] It should be noted that the image preprocessing may include image resolution reduction and / or image data bit width compression. Image resolution reduction may involve downsampling the image so that the resolution of the processed image is less than or equal to the resolution of the original image. Image data bit width compression may involve compressing the data bit width of the image so that the data bit width of the processed image is less than or equal to the data bit width of the original image.

[0091] In practical use, if the image processing equipment does not have sufficient resources, it is necessary to reduce the neural network computing resources consumed during image processing. Therefore, it is necessary to reduce the image resolution of the image processed by the neural network or compress the data bit width of the image processed by the neural network. Thus, the first input image can be preprocessed to generate a preprocessed image, and then the neural network can be used to perform model inference on the preprocessed image to generate a predicted image, thereby saving the neural network computing resources of the image processing equipment.

[0092] In practical applications, when processing the second input image using algorithms, any one of the following methods can be used: pass-through processing, single-stage filtering processing, multi-stage serial filtering processing, or multi-stage parallel filtering processing.

[0093] Among these, pass-through processing can refer to using the second input image directly as the processed image without any processing; single-stage filtering processing can refer to using a single traditional algorithm as a filtering unit to directly process the second input image and generate a processed image; multi-stage cascaded filtering processing can refer to using a single traditional algorithm as a filtering unit, cascading multiple filtering units, and continuously processing the second input image through the cascaded multi-stage filtering units to generate a processed image; multi-stage parallel filtering processing can refer to using a single traditional algorithm as a filtering unit, processing the second input image in parallel through multiple different filtering units, and then summarizing and fusing the processing results to generate a processed image.

[0094] In practical applications, multi-stage cascade filtering and multi-stage parallel filtering can be combined, that is, both cascade and parallel methods can be used to process the second input image.

[0095] Step S30: Fuse the predicted image and the processed image to generate an output image.

[0096] In a specific implementation, the predicted image and the processed image are fused to generate the output image. This can be achieved by weighted fusion of the predicted image and the processed image. The weight coefficients used in the weighted fusion can be preset values ​​or calculated based on the image attributes of the images to be fused.

[0097] Of course, other methods can be used for fusion in actual practice, such as image overlay or similar methods. This embodiment does not limit this.

[0098] In practical use, the image resolution of the output image is equal to the image resolution of the acquired image signal, and the data bit width of the output image is less than or equal to the data bit width of the acquired image signal.

[0099] In one possible implementation of this embodiment, in order to ensure that the image signal during image processing is a high-quality image, step S10 of this embodiment may include:

[0100] The acquired image signal is processed to generate a pre-image signal;

[0101] The preceding image signal is processed according to a first image processing intensity to generate a first input image, and the preceding image signal is processed according to a second image processing intensity to generate a second input image.

[0102] It should be noted that due to differences in device performance, the acquired image signal may be of poor quality or have some defects. Direct image processing in this case may affect subsequent processing. To avoid this situation, after acquiring the image signal, image processing can be performed on the acquired image signal first, that is, pre-image processing, and then the pre-image signal can be further processed according to the first image processing intensity and the second image processing intensity.

[0103] The pre-processing of images may include, but is not limited to, one or more of the following: noise reduction, enhancement, dehazing, deblurring, and sharpening.

[0104] In one possible implementation of this embodiment, to ensure the quality of the output image, step S30 may include:

[0105] The predicted image and the processed image are fused to generate a fused image;

[0106] The fused image is then subjected to post-image processing to generate an output image.

[0107] It should be noted that since weighted fusion or similar fusion methods are used when performing image fusion, after fusing the predicted image and the processed image, the fused image can be further processed, i.e., post-processing can be performed to optimize the image, remove defects, and generate the output image.

[0108] The post-processing may include, but is not limited to, one or more of the following: noise reduction, enhancement, dehazing, deblurring, and sharpening.

[0109] To facilitate understanding, we will now combine... Figure 3 , 4 5 and 6 will be explained, but will not limit this scheme. Figure 3 This is a schematic diagram showing the functional modules of the image processing device in this embodiment. Figure 4 This is a schematic diagram of the execution flow of the first image processing module in this embodiment; Figure 5 This is a schematic diagram illustrating the execution of the first image processing module in this embodiment; Figure 6 This is a schematic diagram of the multi-level filtering execution process in this embodiment.

[0110] like Figure 3 As shown in Figure a, the image acquisition device of this embodiment includes an image acquisition unit and an image processing unit. The image acquisition unit is used to acquire light signals, convert them into image signals, and transmit the acquired image signals to the image processing unit. The image processing unit processes the acquired image signals to generate an output image.

[0111] The structure of the image processing unit is as follows: Figure 3 As shown in b, it includes a first image processing module, an image preprocessing module, a neural network module, a traditional processing module, and an image synthesis module;

[0112] The first image processing module can be implemented using a neural network or a traditional algorithm. This module processes the acquired image signals to generate a first input image and a second input image.

[0113] The image preprocessing module is used to preprocess the first input image, including reducing the image resolution and / or compressing the image data bit width, to generate a preprocessed image.

[0114] The neural network module contains a pre-trained neural network model, which is used to perform model inference on the pre-processed image and generate a predicted image.

[0115] The traditional processing module is equipped with traditional image processing algorithms to process the second input image and generate a processed image.

[0116] The image synthesis module is used to fuse the generated predicted image and the processed image to generate the output image.

[0117] In this embodiment, a pre-processing module can be set before the first image processing module. The pre-processing module can perform pre-image processing on the acquired image signal to generate a pre-image signal. Then, the first image processing module processes the pre-image signal to generate a first input image and a second input image.

[0118] A post-processing module can be set after the image synthesis module to perform post-image processing on the fused image generated by the image synthesis module and generate an output image.

[0119] In practical use, either the pre-processing module or the post-processing module can be set, or both can be set. The characteristics of the images generated by each module and the processing capabilities of each module can be found in the explanations of the steps described in the embodiment; they will not be repeated here.

[0120] The first image processing module can apply different image processing intensities to the acquired image signals during image processing, thereby generating a first input image and a second input image (e.g., ...). Figure 4 (as shown in a).

[0121] Due to the requirements of image processing, in addition to acquiring the collected image signal, a reference image and / or reference information can also be acquired during image processing. Then, based on the reference image and reference information, image processing is performed on the acquired image signal according to the image processing intensity (e.g., ...). Figure 4 (as shown in b).

[0122] The reference image can be the result of historical frame processing. The historical frame can be an image frame that has been previously processed. The result of historical frame processing can be the result of processing other image frames. The reference information can include preset curves (such as tone mapping curves), preset parameters (such as enhancement auxiliary parameters), historical frame statistical information (information obtained by statistically analyzing historically processed images, such as the average, maximum, and minimum values ​​of channel, color, and other data, or similar information), and if necessary, other information for reference can also be included. This example does not limit this.

[0123] When the first image processing module processes and generates the first and second input images, it can also generate additional calculation information and / or intermediate calculation results (such as...) to assist other subsequent modules in their processing and calculations. Figure 4 (as shown in c).

[0124] In practical applications, reference images and reference information can coexist with additional computational information and / or intermediate computational results. In this case, the input and output of the first image processing module can be as follows: Figure 4 As shown in d.

[0125] In one possible implementation of this embodiment, when the image processing performed by the first image processing module is temporal denoising, residual calculation and filtering can be performed on the input image information based on historical frame information (such as historical frame image information, historical frame motion information, or historical frame noise intensity information, etc.) to obtain motion information, and temporal denoising processing can be performed on the input image signal according to the motion information. The calculated output results, i.e., the temporal denoising intensities of the first input image and the second input image, are different.

[0126] At this point, in order to ensure that the temporal denoising intensities of the first and second input images are different, at least one of the following methods can be used:

[0127] (1) When performing residual calculation, use different thresholds to truncate and limit the residual, use different scaling factors to scale the residual, or use both thresholds and scaling factors to process the residual.

[0128] (2) When performing filtering, filters of different scales and different templates are used to filter the residuals at different scales and with different smoothing degrees; or the filtering intensity is changed by controlling the weighting coefficient of the multi-scale filtering result fusion.

[0129] (3) Before performing temporal denoising, the motion information is mapped in different ways to obtain different temporal weighting coefficients;

[0130] (4) After time-domain denoising, the denoised image is weighted and fused with the input image to obtain images with different denoising intensities.

[0131] Therefore, by controlling the temporal denoising intensity of the first and second input images respectively, the noise intensity and trailing of the images can be adjusted to adapt to the processing capabilities of the traditional processing module and the neural network processing module, thereby achieving better denoising results. In addition, the inputs of the traditional processing module and the neural network module are more flexible in processing, which is conducive to obtaining their respective optimal effects in different frequency bands.

[0132] Based on this, the input of the first image processing module can be expanded to include the acquired image signal and historical frame information, and the output of the first image processing module can be expanded to include at least the first input image and the second input image, and may also additionally include at least one of the following: residual result, motion decision result, and noise estimation result (e.g., ...). Figure 5 (as shown in a).

[0133] In one possible implementation of this embodiment, when the function of the first image processing module is spatial domain noise reduction, implemented using a neural network, the following implementation method can be adopted:

[0134] Taking spatial noise reduction using a neural network module (AI) as an example, the first processing module is implemented by using a neural network to process the input acquired image signal and output the first input image and the second input image.

[0135] The image processing intensity control of the first input image and the second input image can be achieved through at least one of the following methods, ensuring that the spatial domain noise reduction intensity of the first input image and the second input image is different:

[0136] (1) Switch the neural network parameters and process the input image twice to obtain the first input image and the second input image respectively. The neural network parameters are obtained by training the neural network using image pairs composed of noiseless images and noisy images. Different neural network parameters are obtained by one or more of the following methods: controlling the number of training iterations, controlling the regularization strength of the loss function, controlling the noise intensity of the noiseless image or the noisy image to obtain different training image pairs, etc.

[0137] (2) Figure 5 As shown in b, after denoising, the images before and after processing are fused with different intensities to calculate the output images with different denoising intensities.

[0138] In application, when controlling the intensity of image processing on the first and second input images, the intensity of image processing can be controlled according to external conditions.

[0139] Among these, external conditions can be the processing performance of the neural network, preset neural network processing capability control parameters, and control the processing intensity of the first processing module according to the control parameters, thereby adjusting the processing intensity of the input image and indirectly controlling the processing difficulty of the image to be processed to achieve a match with the neural network processing capability; external conditions can also be the performance of the traditional processing module, dynamically adjusting the processing intensity of the first image processing module according to the processing capability of the traditional processing module used; external conditions can also be the image acquisition environment, dynamically adjusting the processing intensity of the first image processing module according to different ambient lighting conditions; external conditions can also be processing parameters or statistical results of other image processing modules, such as dynamically adjusting the processing intensity of the first image processing module according to the exposure control module.

[0140] In one possible implementation of this embodiment, when the function of the first image processing module is tone mapping, the following implementation method can be adopted:

[0141] The first image processing module calculates the acquired image signals to obtain and output a first input image and a second input image; wherein the input of the first processing module, in addition to the acquired image signals, may also include a reference image and / or reference information (such as...). Figure 5 (as shown in c);

[0142] The first processing module calculates tone mapping curves and mapping function parameters based on the acquired image signals, as well as reference images and / or reference information. It then processes the acquired image signals based on the tone mapping curves and mapping function parameters to generate a first input image and a second input image, wherein the tone mapping intensities of the first input image and the second input image are different.

[0143] The difference in tone mapping intensity refers to the difference in tone between the first input image and the acquired image signal, and the difference in tone between the second input image and the acquired image signal, i.e., the difference in processing intensity. Either the first input image or the second input image (only one) can be exactly the same as the acquired image signal, i.e., the processing intensity is 0.

[0144] Specific implementation methods for controlling the different processing intensities of the first input image and the second input image include, but are not limited to, one or more of the following:

[0145] (1) When performing tone mapping curve calculation, the weighted fusion image of the acquired image signal and the reference image is used for calculation. By controlling the weighted fusion ratio of the acquired image signal and the reference image, different tone mapping calculation inputs are obtained, and then different tone mapping curves and mapping function parameters are obtained. The acquired image signal is tone mapped using different tone mapping curves and mapping function parameters to obtain the first input image and the second input image with different processing intensities.

[0146] (2) When performing tone mapping curve calculation, different filter kernels, calculation weights, and calculation function parameters are used to obtain different tone mapping curves and mapping function parameters. The different tone mapping curves and mapping function parameters are used to perform tone mapping on the acquired image signals to obtain first and second input images with different processing intensities.

[0147] (3) After calculating the tone mapping curve, the calculated tone mapping curve and the original curve provided by the reference information are weighted and calculated. Different tone mapping curves are obtained by controlling the weighted fusion weight. The different tone mapping curves are used to perform tone mapping on the acquired image signal to obtain the first input image and the second input image with different processing intensities.

[0148] (4) By using different mapping function parameters provided by the reference information to process the acquired image signals, a first input image and a second input image with different processing intensities are obtained;

[0149] (5) After processing the acquired image signal using tone mapping curve, a temporary processed image is obtained. The temporary processed image is then fused and weighted with the acquired image signal. By controlling the weighting fusion weights, a first input image and a second input image with different processing intensities are obtained.

[0150] (6) When performing tone mapping processing on the acquired image signal, a loop processing mode is adopted. By controlling the number of loop processing, a first input image and a second input image with different processing intensities are obtained.

[0151] Based on this, the output of the first image processing module can also include tone calculation information, which can be used as input for subsequent processing modules (such as...). Figure 5 (as shown in d).

[0152] The image preprocessing module can be implemented by a neural network model or by a traditional algorithm. It is used to preprocess the first input image and output a preprocessed image.

[0153] In this process, the image preprocessing module can be implemented by downsampling and bit-width compression of the first input image to generate a preprocessed image. The image resolution and bit-width of the preprocessed image are both less than or equal to the image resolution and bit-width of the first input image.

[0154] For example: Suppose the width and height of the first input image are w and h, respectively. If downsampling and bit-width compression are performed on the first input image simultaneously to generate a preprocessed image, then the downsampling can be:

[0155] The first input image is sampled horizontally and vertically with a stride of n (n is a sampling factor, an integer greater than or equal to 1) to obtain downsampled images with widths and heights of w / n and h / n, respectively. Then, n*n multiple mean downsampling (n is a sampling factor, an integer greater than or equal to 1) is used, meaning that every n*n pixels is averaged with a stride of n in both the horizontal and vertical directions to obtain downsampled images with widths and heights of w / n and h / n, respectively. Alternatively, an m*m filter kernel can be used to convolve the second image point-by-point, and then the convolved image is sampled horizontally and vertically with a stride of n (n is a sampling factor, an integer greater than or equal to 1) to obtain downsampled images with widths and heights of w / n and h / n, respectively. Of course, downsampling can also be achieved using other methods that reduce image resolution.

[0156] Bit width compression can be performed as follows:

[0157] The downsampled image is shifted right by n bits (n bits is the number of shifts, which is an integer greater than or equal to 0) to obtain a preprocessed image with reduced bit width. Alternatively, a curve mapping method can be used, where a curve is set to map from the bit width of the first input image to a lower bit width, and then the downsampled image is mapped point by point to obtain a preprocessed image with reduced bit width. Of course, bit width compression can also be achieved using other methods to reduce the image bit width.

[0158] The order of downsampling and bit-width compression can be reversed, with bit-width compression performed first and then downsampling, or they can be performed simultaneously, with bit-width compression occurring during downsampling.

[0159] In practical applications, bit-width enlargement processing can be the opposite of bit-width compression processing. Taking image prediction and image processing as an example, bit-width enlargement processing can be:

[0160] The upsampled image is shifted left by n bits, where the shift number can be the difference in bit width between the predicted image and the processed image. Alternatively, the bit width increase process can use a preset curve with the same input and output bit widths as the predicted image and the same mapping result bit width as the processed image. The upsampled image is then mapped using this preset curve to obtain a bit-width-increased image B. Another approach is to first shift the upsampled image, with the shift number equal to the difference in bit width between the predicted and processed images. A preset curve with the same input and output bit widths as the processed image is then used to map the upsampled image, and the mapped image is shifted, with the shift number equal to the difference in bit width between the predicted and processed images. Finally, the bit width increase process can also first shift the upsampled image, with the shift number equal to the difference in bit width between the predicted and processed images. A preset curve is then set, with the same input and output bit widths as the processed image. The shifted image is then mapped using this preset curve to obtain a bit-width-increased image.

[0161] In actual processing, the image preprocessing module can also adjust the image resolution and data bit width of the output preprocessed image according to preset conditions.

[0162] In practical applications, a second power consumption threshold can be preset. When the power consumption of the image processing unit is higher than the second power consumption threshold, the downsampling factor of the preprocessed image is increased, resulting in a lower resolution of the preprocessed image. This also reduces the data bit width of the preprocessed image, effectively reducing the amount of data transmitted to the neural network module for processing. Consequently, the computational load and power consumption of the neural network module are effectively reduced, thereby lowering the power consumption of the image processing unit. A fourth power consumption threshold can be preset (usually lower than the second power consumption threshold). When the power consumption of the image processing unit is lower than the fourth power consumption threshold, the downsampling factor of the preprocessed image is reduced, resulting in a higher resolution of the preprocessed image. This also increases the data bit width of the preprocessed image, increasing the amount of data processed by the neural network module. Consequently, the power consumption of the image processing unit is increased, improving the image processing effect.

[0163] Alternatively, the resolution and data bit width of the output preprocessed image can be adjusted based on exposure parameters. Exposure parameters can refer to gain. In this case, a second gain threshold can be preset. When the gain of the imaging device is greater than the second gain threshold (the gain of the imaging device refers to the gain that enables the image signal to reach a suitable brightness after gain amplification, usually including the analog gain of the sensor, the digital gain of the sensor, and the digital gain of the image processing unit), the downsampling factor of the preprocessed image is increased, resulting in a lower resolution of the preprocessed image. It can also reduce the data bit width of the preprocessed image, making the preprocessed image processed by the neural network lower in frequency and with better low-frequency performance. A fourth gain threshold can be preset (usually the fourth gain threshold is less than the second gain threshold). When the gain of the imaging device is less than the fourth gain threshold, the downsampling factor of the preprocessed image is reduced, resulting in a higher resolution of the preprocessed image. It can also increase the data bit width of the preprocessed image, resulting in a larger amount of data transmitted to the neural network module and better high-frequency processing performance.

[0164] Alternatively, the resolution and data bit width of the output preprocessed image can be adjusted based on the transmission bandwidth (the total amount of data transmitted between modules per unit time). A second bandwidth threshold is preset. When the transmission bandwidth is greater than the second bandwidth threshold, the downsampling factor of the preprocessed image is increased, resulting in a lower resolution and a reduced data bit width, thus reducing the amount of data transmitted and lowering the transmission bandwidth. A fourth bandwidth threshold is preset (usually lower than the second bandwidth threshold). When the transmission bandwidth is less than the fourth bandwidth threshold, the downsampling factor of the preprocessed image is reduced, resulting in a higher resolution and a higher data bit width, leading to better image processing results.

[0165] In practical implementation, the neural network module can be implemented by a deep neural network (DNN), a convolutional neural network (CNN), a recurrent neural network (RNN), or other neural networks. The neural network module can perform one or more image processing functions, including but not limited to noise reduction, enhancement, dehazing, deblurring, and sharpening.

[0166] Furthermore, the neural network module can also adjust the image resolution and data bit width according to external conditions. The resolution adjustment can be super-resolution or downsampling, and the data bit width adjustment can be bit width enhancement or bit width compression. The resolution and bit width of the processed image output by the neural network module are no greater than the resolution and bit width of the acquired image signal.

[0167] This external condition can be the transmission bandwidth (the total amount of data transmitted between modules per unit time). A fourth bandwidth threshold is preset. When the transmission bandwidth is greater than the fourth bandwidth threshold, the neural network module can downsample the image and / or compress the bit width to reduce the amount of output data, thereby reducing the transmission bandwidth. A fifth bandwidth threshold is preset (usually the fourth bandwidth threshold is less than the fourth bandwidth threshold). When the transmission bandwidth is less than the fifth bandwidth threshold, the neural network module can perform super-resolution and / or bit width enhancement on the image, thereby making the processing effect of the neural network module output better.

[0168] The external condition can also be power consumption. For example, a fourth power consumption threshold can be preset. When the power consumption of the image processing unit is greater than the fourth power consumption threshold, the neural network module can downsample and / or compress the image to reduce the resource consumption of the neural network module and thus reduce power consumption. Alternatively, a fifth power consumption threshold can be preset (usually the fifth power consumption threshold is less than the fourth power consumption threshold). When the power consumption of the image processing unit is less than the fifth power consumption threshold, the neural network module can perform super-resolution and / or bit-width enhancement on the image to improve the output image effect of the neural network module.

[0169] Furthermore, the training process of the neural network model in the neural network module can be adjusted based on external information to obtain better network parameters. This external information can be the processing intensity information of the first processing module. During the training of the neural network maze, the training process can be intervened based on the image processing intensity used by the first processing module when generating the first input image.

[0170] Tuning the training process can include, but is not limited to, adjusting the loss function, adjusting pre-training parameters, adjusting training process hyperparameters, and generating training sample pairs.

[0171] The inference process of the neural network model can also adjust the processing intensity (i.e., adjust the processing effect) based on external information. This external information can be the image processing intensity of the first image processing module. Methods for adjusting the processing intensity include, but are not limited to, switching network parameters and correcting network processing hyperparameters.

[0172] In practice, the traditional processing module is used to process the second input image and output the processed image.

[0173] The image resolution and data bit width of the processed image are consistent with those of the acquired image signal. Traditional processing modules can be implemented using traditional algorithms, including but not limited to one or more of the following: noise reduction, enhancement, dehazing, deblurring, and sharpening.

[0174] Specifically, the generated and processed image can be achieved in the following ways:

[0175] 1. Pass-through processing: The processed image is generated using a pass-through method, that is, the second input image is directly used as the processed image for output.

[0176] 2. Single filter module processing implementation:

[0177] A single filtering module can be used to filter the second input image to generate a processed image;

[0178] During the filtering process, the size of the filter template can be m*m (m∈{m|m=2n-1,n∈N+}), then the input image edge extension is (M-1) / 2, and the extension pixel calculation method includes, but is not limited to, zero-padding extension, edge copy extension, mirror extension, etc., and the filtering interval can be 1.

[0179] 3. Multiple filters connected in series or parallel form a traditional processing module.

[0180] like Figure 6 As shown in diagram a, the second input image can be processed by using multiple filtering sub-modules connected in series to generate a processed image. For example, the second input image is processed by the first filtering sub-module to obtain the first image filtering result; the first image filtering result is processed by the second filtering sub-module to obtain the second image filtering result; other filtering modules can be connected in series to finally output the processed image. The number of modules connected in series is not limited.

[0181] like Figure 6 As shown in b, in cascade filtering, some filtering modules can be composed of multiple sub-filtering modules connected in parallel. For example, the second input image is processed by the first filtering sub-module, and the third, fourth, and fifth sub-filtering modules output the third image filtering result, the fourth image filtering result, and the predicted image filtering result; the third image filtering result, the fourth image filtering result, and the predicted image filtering result are then processed by the second filtering module to generate the processed image.

[0182] The number of parallel sub-filter modules is unlimited, and the sub-filter modules can still be composed of sub-sub-modules connected in series or in parallel.

[0183] This embodiment generates a first input image by processing the acquired image signal according to a first image processing intensity, and generates a second input image by processing the image signal according to a second image processing intensity. Model inference is then performed on the first input image to generate a predicted image, and algorithmic processing is applied to the second input image to generate a processed image. The predicted image and the processed image are then fused to generate an output image. Because the acquired image is processed separately using both the model and traditional algorithms, and the results of both processing are fused before outputting the image, the processing intensity of the two outputs can be dynamically adjusted according to actual needs. This ensures better image quality and lower resource consumption, thereby improving the processing performance of convolutional neural networks in the field of image processing.

[0184] refer to Figure 7 , Figure 7 This is a flowchart illustrating a second embodiment of an image processing method according to this application.

[0185] Based on the first embodiment described above, step S30 of the image processing method in this embodiment includes:

[0186] Step S301: Obtain the weight coefficients corresponding to the predicted image and the processed image respectively;

[0187] Step S302: Based on the weight coefficients, perform weighted fusion on the predicted image and the processed image to obtain the output image.

[0188] It should be noted that the weight coefficients corresponding to the predicted image and the processed image can be preset values, which are pre-set by the image processing equipment administrator.

[0189] In practical applications, the weighting coefficients can be calculated based on the brightness of the predicted image and the processed image. For example, if the predicted image is used as the fusion input image A, or the processed image is used as the fusion input image B, the weighting coefficients can be set based on the local brightness of either the fusion input image A or the fusion input image B. Different weighting coefficients can be set for different brightness ranges. The fusion input images A and B are then weighted and fused based on the weighting coefficients.

[0190] The weighting coefficients can also be determined based on the edge strength of the predicted image and the processed image. For example, if the predicted image is used as the fusion input image A, or the processed image is used as the fusion input image B, then the edge strength of either the fusion input image A or the fusion input image B is extracted first. The edge strength extraction can be performed using the Sobel operator, the Canny operator, or other edge extraction operators. Then, different fusion weights are set according to the edge strength of different local areas. Finally, the input fusion input image A and the fusion input image B are weighted and fused according to the local fusion weights.

[0191] The weighting coefficients can also be used to determine the weights based on the difference between the predicted image and the processed image and the edge intensity. For example, if the predicted image is used as the fusion input image A, or the processed image is used as the fusion input image B, the bit width of the fusion input image B can be reduced first (the bit width compression can be implemented using any of the bit width compression methods in the aforementioned image preprocessing module), and then the bit width of the compressed image can be increased (the bit width increase can be implemented using one of the aforementioned bit width increase methods).

[0192] The difference between the fused input image B and the image with increased bit width is calculated to obtain the difference image. Then, edge intensity is extracted from the fused input image A to obtain the edge intensity map (the edge intensity extraction can be performed using the Sobel operator, Canny operator, or other edge extraction operators). The difference image and the edge intensity map are weighted according to preset weights to obtain the overall weight coefficient. The fused input image A and the fused input image B are then weighted and fused according to the overall weight coefficient.

[0193] The weighting coefficients can also be calculated and determined based on the image processing intensity of the first image processing module. For example, the fusion weights of the predicted image and the processed image can be directly obtained by calculating the processing intensity of the first input image generated by the first image processing module and the processing intensity of the second input image generated by the first image processing module.

[0194] Alternatively, the fusion weights of the predicted image and the processed image can be calculated by combining the processing strength of the first image processing module with that of the neural network module and the traditional processing module.

[0195] Of course, other methods can be used to determine the weighting coefficients if needed, and this embodiment does not limit this.

[0196] In a specific implementation, to ensure the quality of the output image, step S302 in this embodiment may include:

[0197] The predicted image is subjected to temporal filtering, upsampling, and / or bit width increase processing to generate a processed predicted image;

[0198] The processed predicted image and the processed image are weighted and fused based on the weight coefficients to obtain the output image.

[0199] It should be noted that since the predicted image is obtained by a series of processing based on the first input image, it may have undergone downsampling and / or bit width compression processing, which may have caused changes in image resolution and data bit width. In this case, it is necessary to perform upsampling and / or bit width increase processing to restore its image resolution and data bit width, so as to ensure that it is consistent with the image resolution and data bit width of the initially acquired image signal, thereby ensuring that it can be properly fused with the processed image.

[0200] Since the predicted image is generated by processing a pre-trained neural network model, it may have some defects, such as added noise or insufficient image smoothness. In order to eliminate these defects, temporal filtering can be performed on it.

[0201] To facilitate understanding, we will now combine... Figure 8 , 9 The explanations for 10 and 11 are provided, but this does not limit the scope of this scheme. Figure 8 This is a schematic diagram of image synthesis filtering in this embodiment. Figure 9 This is a schematic diagram of the first image fusion processing flow in this embodiment. Figure 10 This is a schematic diagram of the second image fusion processing flow in this embodiment. Figure 11 This is a schematic diagram of the sensor arrangement and spectral response curve in this embodiment.

[0202] In one possible implementation of this embodiment, such as Figure 8 As shown, before fusing the predicted image and the processed image, the predicted image can be subjected to temporal filtering to suppress temporal noise and obtain a filtered predicted image, which is then combined with the processed image.

[0203] One implementation of temporal filtering is to input the Nth frame prediction image and the (N-1)th frame filtered prediction image, weight them according to the external input weights, output the weighted Nth frame filtered prediction image, and store the filtered prediction image in memory for temporal filtering in the next frame (N+1th frame).

[0204] Another implementation method is to calculate the difference between the predicted image of the Nth frame and the filtered predicted image of the (N-1)th frame, and then take the absolute value of the difference to obtain the frame difference image A. A weighted weight is calculated based on the frame difference image A. The weighted weight can be calculated based on the size of the frame difference; for example, the larger the frame difference, the larger the weight of the predicted image of the Nth frame, and the smaller the weight of the filtered predicted image of the (N-1)th frame; conversely, the smaller the frame difference, the smaller the weight of the predicted image of the Nth frame, and the larger the weight of the filtered predicted image of the (N-1)th frame. A weighted temporal filter is then applied to the predicted image of the Nth frame and the filtered predicted image of the (N-1)th frame based on the weighted weight, and the weighted filtered predicted image of the Nth frame is output. Simultaneously, the filtered predicted image of the Nth frame is stored in memory for temporal filtering processing in the next frame (the (N+1)th frame).

[0205] In one possible implementation of this embodiment, such as Figure 9As shown, the prediction image is first upsampled to the same resolution as the processed image to obtain an upsampled image A. This upsampling can be bilinear upsampling or nearest-neighbor interpolation upsampling. Then, the upsampled image A is bit-width increased to match the bit width of the fourth image, resulting in a bit-width-increased image B. This bit-width increase can be achieved by directly shifting the upsampled image by a number of bits equal to the difference in bit width between the prediction and processed images; alternatively, a preset curve can be used, where the input bit width of the preset curve is the same as the prediction image bit width, and the mapping result bit width is the same as the processed image bit width. The upsampled image is then subjected to curve mapping using this preset curve to obtain the bit-width-increased image B. Alternatively, a preset curve can be used for the bit-width increase processing. The input and output bit widths of the preset curve are the same as the bit width of the predicted image. The preset curve is used to perform curve mapping processing on the upsampled image to obtain a mapped image. The mapped image is then shifted by a number equal to the bit width difference between the predicted and processed images. Alternatively, the bit width increase processing can first shift the upsampled image by the same number of shifts as the bit width difference between the predicted and processed images. A preset curve is then set, with both its input and output bit widths matching the bit width of the processed image. The shifted image is then subjected to curve mapping processing using the preset curve to obtain a bit-width-increased image B. Finally, a weighted fusion of the bit-width-increased image B and the fourth image is performed using preset weights to obtain the output image.

[0206] In one possible implementation of this embodiment, such as Figure 10 As shown, the image is first downsampled to obtain a downsampled image A with the same resolution as the predicted image. This downsampling can be mean downsampling, sampling downsampling, or any of the downsampling methods mentioned in the image preprocessing module. The downsampled image A is then upsampled to obtain an upsampled image B with the same resolution as the fourth image. This upsampling can be achieved using bilinear upsampling or nearest-neighbor upsampling. The processed image is subtracted from the upsampled image B to obtain a high-frequency image C. The predicted image is then bit-width increased to obtain a bit-width increased image D. This bit-width increase can be achieved using any of the bit-width increase methods mentioned in the previous section. The bit-width increased image D and the downsampled image A are then weighted and fused using preset weights to obtain a fused image E. The fused image E is then upsampled to obtain an upsampled image F. This upsampling can be achieved using bilinear upsampling or nearest-neighbor upsampling. Finally, the upsampled image F and the high-frequency image C are superimposed to obtain the output image. This superposition process can be achieved by directly adding the two images.

[0207] Among them, the above Figure 8 , 9The weighting coefficients for weighted fusion in the examples related to 10 can all be preset values ​​or determined by the real-time calculation method described above. This embodiment does not impose any restrictions on this.

[0208] In practical applications, the image acquisition unit can be an image sensor, which can be an RGBW sensor or a sensor arranged in a Bayer or RGB-IR manner. An example of the sensor arrangement is shown below. Figure 11 As shown in Figure a, the spectral response curve of the RGBW sensor is as follows: Figure 11 As shown in b, the image sensor can convert the acquired light signal into an image signal through exposure, and then transmit it to the image processing unit as the acquired image signal.

[0209] In addition to using fixed preset weighting coefficients for weighted fusion, this embodiment can also calculate weighting coefficients in real time based on the image attributes of the images to be fused, or calculate weighting coefficients in real time based on the image processing intensity during image processing, to ensure that the image fusion meets the real-time image processing requirements and achieves the best fusion effect.

[0210] Furthermore, embodiments of this application also propose a storage medium storing an image processing program, which, when executed by a processor, implements the steps of the image processing method described above.

[0211] Reference Figure 12 , Figure 12 This is a structural block diagram of the first embodiment of the image processing apparatus of this application.

[0212] like Figure 12 As shown, the image processing apparatus proposed in this application includes:

[0213] Image processing module 10 is used to perform image processing on the acquired image signal according to a first image processing intensity to generate a first input image, and to perform image processing on the image signal according to a second image processing intensity to generate a second input image;

[0214] The image generation module 20 is used to perform model inference on the first input image to generate a predicted image, and to perform algorithm processing on the second input image to generate a processed image;

[0215] The image synthesis module 30 is used to fuse the predicted image and the processed image to generate an output image.

[0216] This embodiment generates a first input image by processing the acquired image signal according to a first image processing intensity, and generates a second input image by processing the image signal according to a second image processing intensity. Model inference is then performed on the first input image to generate a predicted image, and algorithmic processing is applied to the second input image to generate a processed image. The predicted image and the processed image are then fused to generate an output image. Because the acquired image is processed separately using both the model and traditional algorithms, and the results of both processing are fused before outputting the image, the processing intensity of the two outputs can be dynamically adjusted according to actual needs. This ensures better image quality and lower resource consumption, thereby improving the processing performance of convolutional neural networks in the field of image processing.

[0217] In one possible implementation of this embodiment, the image generation module 20 is further configured to perform image preprocessing on the first input image to generate a preprocessed image, wherein the image preprocessing includes image resolution reduction and / or image data bit width compression; and perform image prediction on the preprocessed image to generate a predicted image.

[0218] In one possible implementation of this embodiment, at least one of the following is satisfied:

[0219] The first image processing intensity is less than or equal to the second image processing intensity;

[0220] The image processing functions include at least one of temporal domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, dehazing, deblurring, and sharpening;

[0221] The same processing function is used to generate the first input image and the second input image.

[0222] In one possible implementation of this embodiment, the model inference is performed by a neural network module, and the algorithm processing is performed by a traditional algorithm module.

[0223] Meet at least one of the following:

[0224] The processing functions of the neural network module include at least one of temporal domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, dehazing, deblurring, and sharpening.

[0225] The processing functions of the traditional algorithm module include at least one of temporal domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, dehazing, deblurring, and sharpening;

[0226] The processing functions of the neural network module include those of the traditional algorithm module;

[0227] The processing effect of the model inference is adjusted by controlling the network parameters of the model used in the neural network module. The network parameters are determined based on the first image processing intensity and / or the second image processing intensity, or based on the first input image and / or the second input image.

[0228] In one possible implementation of this embodiment, the model inference is performed by a neural network module, and the image processing is performed by an image processing module.

[0229] If the neural network module includes the processing function of the image processing module, then the first image processing intensity is less than the second image processing intensity.

[0230] If the neural network module does not include the processing function of the image processing module, then the first image processing intensity is less than or equal to the second image processing intensity.

[0231] In one possible implementation of this embodiment, the image synthesis module 30 is further configured to fuse the predicted image and the processed image to generate a fused image; and to perform post-image processing on the fused image to generate an output image.

[0232] In one possible implementation of this embodiment, the image processing module 10 is further configured to perform pre-image processing on the acquired image signal to generate a pre-image signal; perform image processing on the pre-image signal according to a first image processing intensity to generate a first input image; and perform image processing on the pre-image signal according to a second image processing intensity to generate a second input image.

[0233] In one possible implementation of this embodiment, the image synthesis module 30 is further configured to obtain the weight coefficients corresponding to the predicted image and the processed image respectively; and to perform weighted fusion of the predicted image and the processed image based on the weight coefficients to obtain an output image.

[0234] In one possible implementation of this embodiment, the weighting coefficient is a preset value, determined based on the image brightness corresponding to the predicted image and the processed image, determined based on the edge intensity corresponding to the predicted image and the processed image, or determined based on the image processing intensity corresponding to the first input image and the second input image.

[0235] In one possible implementation of this embodiment, the image synthesis module 30 is further configured to perform temporal filtering, upsampling, and / or bit-width increase processing on the predicted image to generate a processed predicted image; and to perform weighted fusion of the processed predicted image and the processed image based on the weight coefficients to obtain an output image.

[0236] In one possible implementation of this embodiment, the algorithm processing includes pass-through processing, single-stage filtering processing, multi-stage series filtering processing, or multi-stage parallel filtering processing.

[0237] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solution of this application. In specific applications, those skilled in the art can make settings as needed, and this application does not impose any restrictions on this.

[0238] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this application. In practical applications, those skilled in the art can select some or all of it to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0239] In addition, for technical details not described in detail in this embodiment, please refer to the image processing method provided in any embodiment of this application, which will not be repeated here.

[0240] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0241] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0242] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk), and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0243] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An image processing method, characterized by, The image processing method comprises: performing image processing on the collected image signal according to a first image processing intensity to generate a first input image and performing image processing on the image signal according to a second image processing intensity to generate a second input image; performing model inference on the first input image to generate a predicted image and performing algorithm processing on the second input image to generate a processed image; fusing the predicted image and the processed image to generate an output image.

2. The image processing method of claim 1, wherein, The image prediction on the first input image to generate a predicted image comprises: performing image preprocessing on the first input image to generate a preprocessed image, wherein the image preprocessing comprises image resolution reduction and / or image data bit width compression; performing model inference on the preprocessed image to generate a predicted image.

3. The image processing method of claim 1, wherein, At least one of the following is satisfied: The first image processing intensity is less than or equal to the second image processing intensity; The processing function of the image processing comprises at least one of time domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, defogging, deblurring and sharpening; The processing function adopted to generate the first input image and the second input image is the same.

4. The image processing method of claim 3, wherein, The model inference is performed by a neural network module, and the algorithm processing is performed by a traditional algorithm module; At least one of the following is satisfied: The processing function of the neural network module comprises at least one of time domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, defogging, deblurring and sharpening; The processing function of the traditional algorithm module comprises at least one of time domain noise reduction, spatial domain noise reduction, tone mapping, enhancement, defogging, deblurring and sharpening; The processing function of the neural network module comprises the processing function of the traditional algorithm module; The processing effect of the model inference is adjusted by controlling network parameters of a model used in the neural network module, wherein the network parameters are determined according to the first image processing intensity and / or the second image processing intensity, or are determined according to the first input image and / or the second input image.

5. The image processing method of claim 3, wherein, The model inference is performed by a neural network module, and the image processing is performed by an image processing module; If the neural network module comprises the processing function of the image processing module, the first image processing intensity is less than the second image processing intensity; If the neural network module does not comprise the processing function of the image processing module, the first image processing intensity is less than or equal to the second image processing intensity.

6. The image processing method of claim 1, wherein, The fusion of the predicted image and the processed image to generate an output image comprises: fusing the predicted image and the processed image to generate a fused image; performing post-image processing on the fused image to generate an output image.

7. The image processing method of claim 1, wherein, The image processing on the collected image signal according to a first image processing intensity to generate a first input image and the image processing on the image signal according to a second image processing intensity to generate a second input image comprise: performing pre-image processing on the collected image signal to generate a pre-image signal; performing image processing on the pre-image signal to generate a first input image and a second input image.

8. The image processing method of claim 1, wherein, The fusing the predicted image and the processed image generates an output image, including: obtaining a weight coefficient corresponding to the predicted image and the processed image respectively; based on the weight coefficient, the predicted image and the processed image are weighted and fused to obtain an output image.

9. The image processing method of claim 8, wherein, The weight coefficient is a preset value, determined according to the image brightness corresponding to the predicted image and the processed image, determined according to the edge intensity corresponding to the predicted image and the processed image, or determined according to the image processing intensity corresponding to the first input image and the second input image.

10. The image processing method of claim 8, wherein, The weight coefficient is a preset value, determined according to the image brightness corresponding to the predicted image and the processed image, determined according to the edge intensity corresponding to the predicted image and the processed image, or determined according to the image processing intensity corresponding to the first input image and the second input image. The weight coefficient is a preset value, determined according to the image brightness corresponding to the predicted image and the processed image, determined according to the edge intensity corresponding to the predicted image and the processed image, or determined according to the image processing intensity corresponding to the first input image and the second input image. The algorithm processing includes straight-through processing, single-stage filtering processing, multi-stage series filtering processing or multi-stage filtering parallel processing.

11. The image processing method of any one of claims 1-10, wherein, The image processing device includes:

12. An image processing apparatus characterized by comprising: an image processing module, configured to perform image processing on the collected image signal according to a first image processing intensity to generate a first input image, and perform image processing on the image signal according to a second image processing intensity to generate a second input image; an image generation module, configured to perform model inference on the first input image to generate a predicted image, and perform algorithm processing on the second input image to generate a processed image; an image synthesis module, configured to fuse the predicted image and the processed image to generate an output image. The image processing device includes a processor, a memory, and an image processing program stored on the memory and executable on the processor, and the image processing program is executed by the processor to implement the steps of the image processing method according to any one of claims 1-9.

13. An image processing apparatus characterized by comprising: The computer readable storage medium stores an image processing program, and the image processing program is executed to implement the steps of the image processing method according to any one of claims 1-11.

14. A computer-readable storage medium, characterized in that, The computer program product includes an image processing program, and the image processing program is executed to implement the steps of the image processing method according to any one of claims 1-11.

15. A computer program product, characterised in that, ​

Images