Image processing method, device, chip, electronic device and storage medium
By acquiring the characteristics of the image sequence and its reverse sequence for fusion processing, the problem of poor image detail enhancement and denoising effects in the prior art is solved, and high-quality image processing is achieved.
Patent Information
- Application Number
- CN202210323341.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-29
- Publication Date
- 2024-12-17
- Estimated Expiration
- 2042-03-29
AI Technical Summary
In the prior art, image detail enhancement and denoising processing effects are poor, and the image quality of the processing results is difficult to achieve user satisfaction.
By acquiring the sequence of images to be processed and generating its reverse image sequence, the first deep features and the second deep features are extracted respectively, and feature alignment processing is performed, and finally they are fused to generate the target image. This method utilizes a neural network model, including a feature extraction module, a feature alignment module and a feature fusion module, making full use of the front and reverse direction feature information of the image sequence.
It significantly improves the image enhancement and denoising effect, improves the quality of the target image, and meets user satisfaction.
Smart Images
Figure CN114648478B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of image processing technology, and in particular to an image processing method, device, chip, electronic device and storage medium. Background Art
[0002] In recent years, with the development of artificial intelligence technology, various automated processing can be performed on images and videos, such as image detail enhancement and denoising. Image detail enhancement technology is a technology that uses image processing methods to increase or restore details on the basis of the original image. The processed image will have finer texture and visual performance than the original image. In related technologies, images can be processed by multi-frame fusion, that is, by denoising and enhancing details by combining several continuously shot images, but the processing effect achieved needs to be improved, and the image quality of the processing result is difficult to satisfy users. Summary of the invention
[0003] In order to overcome the problems existing in the related art, the embodiments of the present disclosure provide an image processing method, device, chip, electronic device and storage medium to solve the defects in the related art.
[0004] According to a first aspect of an embodiment of the present disclosure, there is provided an image processing method, including:
[0005] Acquire a sequence of images to be processed, and generate a reverse image sequence of the sequence of images to be processed, wherein the sequence of images to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the images to be processed;
[0006] Respectively extracting first deep features of the image sequence to be processed and second deep features of the reverse image sequence of the image sequence to be processed;
[0007] Performing feature alignment processing on the first deep features, and performing feature alignment processing on the second deep features;
[0008] The first deep feature and the second deep feature are fused, and a target image is generated according to a fusion result.
[0009] In one embodiment, obtaining a sequence of images to be processed includes:
[0010] Acquire multiple frames of images to be processed that are arranged in sequence, and generate a noise map of each frame of the image to be processed according to the parameters of an acquisition device that acquires the images to be processed;
[0011] A sequence of images to be processed is generated by sequentially arranging a plurality of frames of images to be processed and a noise map of each frame of the image to be processed.
[0012] In one embodiment, performing feature alignment processing on the first deep features and performing feature alignment processing on the second deep features includes:
[0013] Extracting shallow features of the image sequence to be processed for each color channel and noise map of the image to be processed in the image sequence to be processed;
[0014] The shallow features are used as guidance information for feature alignment processing, and feature alignment processing is performed on the first deep features and the second deep features respectively.
[0015] In one embodiment, the image processing method is performed using a neural network model, and the neural network model includes a feature extraction module, a feature alignment module, and a feature fusion module;
[0016] Among them, the feature extraction module is used to extract the first deep features of the image sequence to be processed and the second deep features of the reverse image sequence of the image sequence to be processed respectively; the feature alignment module is used to perform feature alignment processing on the first deep features and perform feature alignment processing on the second deep features; the feature fusion module is used to fuse the first deep features and the second deep features, and generate a target image according to the result of the fusion processing.
[0017] In one embodiment, the neural network model also includes a preprocessing module for acquiring an image sequence to be processed and generating a reverse image sequence of the image sequence to be processed, wherein the image sequence to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the image to be processed.
[0018] In one embodiment, it further includes:
[0019] Acquire a training image sequence in a training set, and add noise to each frame of the training image in the training image sequence to obtain a noise map of each frame of the training image;
[0020] Inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, and outputting the predicted image;
[0021] A network loss value is determined according to the predicted image and the label image of the training image sequence, and network parameters of the neural network model are adjusted according to the network loss value until the neural network model converges.
[0022] In one embodiment, before inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, the method further includes:
[0023] For the training image sequence, the position of the training image of the middle frame is kept unchanged, and the training images of other frames are randomly arranged.
[0024] In one embodiment, before inputting the training image sequence, the reverse image sequence of the training image sequence, the noise map of each frame of the training image, and the noise map of each frame of the training image into the neural network model, the method further includes:
[0025] Each frame of training image is enhanced separately.
[0026] In one embodiment, it further includes:
[0027] Acquire multiple original image sequences, wherein each original image sequence includes a preset number of original images;
[0028] Perform blur processing on each original image in each original image sequence to obtain a training image corresponding to each original image;
[0029] According to the arrangement order of each original image in each original image sequence, the training images corresponding to each original image are arranged to obtain a training image sequence corresponding to each original image sequence, and a training set is determined according to all the training image sequences.
[0030] In one embodiment, it further includes:
[0031] The middle frame original image of the original image sequence is determined as the label image of the corresponding training image sequence.
[0032] According to a second aspect of an embodiment of the present disclosure, there is provided an image processing apparatus, including:
[0033] A preprocessing module, used for acquiring an image sequence to be processed and generating a reverse image sequence of the image sequence to be processed, wherein the image sequence to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the image to be processed;
[0034] A feature extraction module, used to extract first deep features of the image sequence to be processed and second deep features of the reverse image sequence of the image sequence to be processed respectively;
[0035] A feature alignment module, used for performing feature alignment processing on the first deep features and performing feature alignment processing on the second deep features;
[0036] The feature fusion module is used to fuse the first deep feature and the second deep feature, and generate a target image according to the result of the fusion processing.
[0037] In one embodiment, when the acquisition module is used to acquire the image sequence to be processed, it is specifically used to:
[0038] Acquire multiple frames of images to be processed that are arranged in sequence, and generate a noise map of each frame of the image to be processed according to the parameters of an acquisition device that acquires the images to be processed;
[0039] A sequence of images to be processed is generated by sequentially arranging a plurality of frames of images to be processed and a noise map of each frame of the image to be processed.
[0040] In one embodiment, the feature alignment module is specifically used to:
[0041] Extracting shallow features of the image sequence to be processed for each color channel and noise map of the image to be processed in the image sequence to be processed;
[0042] The shallow features are used as guidance information for feature alignment processing, and feature alignment processing is performed on the first deep features and the second deep features respectively.
[0043] In one embodiment, the image processing device includes a neural network model, and the neural network model includes the feature extraction module, the feature alignment module and the feature fusion module.
[0044] In one embodiment, the neural network model also includes the preprocessing module.
[0045] In one embodiment, the image processing apparatus further comprises a training module, which is used to:
[0046] Acquire a training image sequence in a training set, and add noise to each frame of the training image in the training image sequence to obtain a noise map of each frame of the training image;
[0047] Inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, and outputting the predicted image;
[0048] A network loss value is determined according to the predicted image and the label image of the training image sequence, and network parameters of the neural network model are adjusted according to the network loss value until the neural network model converges.
[0049] In one embodiment, the training module is further used to:
[0050] Before inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, for the training image sequence, the position of the training image of the middle frame is kept unchanged, and the training images of other frames are randomly arranged.
[0051] In one embodiment, the training module is further used to:
[0052] Before inputting the training image sequence, the reverse image sequence of the training image sequence, the noise map of each frame of training image and the noise map of each frame of training image into the neural network model, each frame of training image is enhanced.
[0053] In one embodiment, a training set generation module is further included, which is used to:
[0054] Acquire multiple original image sequences, wherein each original image sequence includes a preset number of original images;
[0055] Perform blur processing on each original image in each original image sequence to obtain a training image corresponding to each original image;
[0056] According to the arrangement order of each original image in each original image sequence, the training images corresponding to each original image are arranged to obtain a training image sequence corresponding to each original image sequence, and a training set is determined according to all the training image sequences.
[0057] In one embodiment, the training set generation module is further used to:
[0058] The middle frame original image of the original image sequence is determined as the label image of the corresponding training image sequence.
[0059] According to a third aspect of an embodiment of the present disclosure, a chip is provided, the chip comprising a processor, the processor being used to execute the image processing method described in the first aspect.
[0060] According to a fourth aspect of an embodiment of the present disclosure, an electronic device is provided, comprising a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to execute the computer instructions based on the image processing method described in the first aspect.
[0061] According to a fifth aspect of an embodiment of the present disclosure, there is provided a computer-readable storage medium on which a computer program is stored, and when the program is executed by a processor, the method described in the first aspect is implemented.
[0062] The technical solution provided by the embodiments of the present disclosure may have the following beneficial effects:
[0063] The present disclosure obtains a sequence of images to be processed and generates a reverse image sequence of the sequence of images to be processed, and can respectively extract the first deep features of the sequence of images to be processed and the second deep features of the reverse image sequence of the sequence of images to be processed, and can further perform feature alignment processing on the first deep features and feature alignment processing on the second deep features, and finally can perform fusion processing on the first deep features and the second deep features, and generate a target image according to the result of the fusion processing. Since the deep features extracted and aligned in both the positive and negative directions of the sequence of images to be processed are fused, the feature information of the image to be processed in the sequence of images to be processed is fully utilized, the image enhancement and denoising effects are more obvious, the target image obtained is of high quality, and the user satisfaction is improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0064] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the invention and, together with the description, serve to explain the principles of the invention.
[0065] Figure 1 is a flowchart of an image processing method shown in an exemplary embodiment of the present disclosure;
[0066] Figure 2 is a structural schematic diagram of a feature extraction module shown in an exemplary embodiment of the present disclosure;
[0067] Figure 3 is a flowchart of a training method for a neural network model shown in an exemplary embodiment of the present disclosure;
[0068] Figure 4 is a structural schematic diagram of an image processing device shown in an exemplary embodiment of the present disclosure;
[0069] Figure 5 It is a structural block diagram of an electronic device shown in an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION
[0070] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.
[0071] The terms used in this disclosure are for the purpose of describing specific embodiments only and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items.
[0072] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".
[0073] In a first aspect, at least one embodiment of the present disclosure provides an image processing method. Figure 1 , which shows the process of the method, including step S101 and step S104.
[0074] The method can be used to perform detail enhancement and denoising on an image. The method can perform detail enhancement and denoising on an image by using a multi-frame fusion method. For example, in a possible scenario, the method is applied to a camera program, where a camera can continuously capture multiple frames of original images, and then process the multiple frames of original images to obtain a final output image.
[0075] The method is applied to terminal devices or servers such as user equipment (UE), mobile devices, user terminals, terminals, cellular phones, cordless phones, personal digital assistant (PDA) handheld devices, computing devices, vehicle-mounted devices, wearable devices, etc.
[0076] In step S101, a sequence of images to be processed is obtained, and a reverse image sequence of the sequence of images to be processed is generated, wherein the sequence of images to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the image to be processed.
[0077] The image sequence to be processed may be a plurality of frames of images continuously captured by a camera, or a plurality of continuous image frames in a video captured by a camera. The image to be processed may be an RGB three-channel image.
[0078] In a possible embodiment, the sequence of images to be processed can be obtained in the following manner: first, multiple frames of images to be processed arranged in sequence are obtained, and a noise map of each frame of the image to be processed is generated according to the parameters of the acquisition device that acquires the images to be processed, wherein the acquisition device can be a camera, a webcam, etc. Exemplarily, according to the preset number requirement, the preset number of frames of images to be processed that are newly acquired by the acquisition device can be acquired in real time. For example, if the preset number is 5, a time window with a frame number of 5 can be set, and the time window ends with the latest acquired frame of the image to be processed. Each time a frame of the image to be processed is acquired, 5 frames of the image to be processed within the time window are acquired. The parameters of the acquisition device can affect the noise of the image it acquires, so a noise map of each frame of the image to be processed can be generated according to the parameters. The noise map can reflect the noise position and noise intensity, and the size and number of channels of the noise map are consistent with the image to be processed. Next, the sequence of images to be processed is generated by using the multiple frames of images to be processed arranged in sequence and the noise map of each frame of the image to be processed.
[0079] Among them, the arrangement order of the multiple frames of images to be processed in the reverse image sequence of the image sequence to be processed is opposite to the arrangement order of the multiple frames of images to be processed in the image sequence to be processed. In a possible embodiment, the reverse image sequence of the image sequence to be processed can be generated in the following manner: first, the arrangement order of the multiple frames of images to be processed in the image sequence to be processed is adjusted to a completely opposite order, for example, there are 5 frames of images in total, the original 1st frame is adjusted to the 5th frame, the original 2nd frame is adjusted to the 4th frame, the original 3rd frame remains unchanged, the original 4th frame is adjusted to the 2nd frame, and the original 5th frame is adjusted to the 1st frame; then, the reverse image sequence of the image sequence to be processed is generated using the multiple frames of images to be processed arranged in reverse and the noise map of each frame of the image to be processed.
[0080] It is understandable that this step can be performed using a neural network model, that is, after the image sequence to be processed is input into the neural network model, the neural network model performs this step. Exemplarily, the neural network model has a preprocessing module, and the preprocessing module is used to perform this step.
[0081] In step S102, first deep features of the image sequence to be processed and second deep features of the reverse image sequence of the image sequence to be processed are extracted respectively.
[0082] Among them, the arrangement order of the images to be processed in the image sequence to be processed and its reverse image sequence is opposite, so the information flow is opposite. Therefore, the extracted deep features represent different contents. The deep features in two directions can more comprehensively represent the image content in the image sequence to be processed.
[0083] It is understandable that this step can be performed using a neural network model. Exemplarily, the neural network model has a feature extraction module for performing this step. If step S101 is performed by a preprocessing module of the neural network model, the preprocessing module can input the image sequence to be processed and the reverse image sequence of the image sequence to be processed into the feature extraction module; then the feature extraction module performs consistent extraction operations on the image sequence to be processed and its reverse image sequence, respectively, that is, extracting the first deep features of the image sequence to be processed and the second deep features of the reverse image sequence of the image sequence to be processed, respectively; if step S101 is not performed by the neural network model, the module for performing step S101 can input the image sequence to be processed and the reverse image sequence of the image sequence to be processed into the feature extraction module of the neural network model, so that it completes feature extraction in the above manner.
[0084] The feature extraction module may include a multi-layer convolution unit group stacked in a residual connection manner, each layer of the convolution unit group having at least two convolution units, and also includes a cascade layer, an attention mechanism layer, etc. For example, it may be a residual feature distillation block.
[0085] In one example, the feature extraction module can be attached Figure 2In the structure shown, after the data to be feature extracted (input) is input into the module, the channels of the data are divided into two parts, for example, 16 channels are divided into 8+8 forms, and then a part of the channels (for example, 8 channels) are convolved by the 3*3 convolution unit (Conv3) of the first layer, and the other part of the channels (for example, 8 channels) are convolved by the 1*1 convolution unit (Conv1) of the first layer; then the channels of the convolution processing results of the 3*3 convolution unit of the first layer are divided into two parts, for example, 8 channels are divided into 4+4 forms, and then a part of the channels (for example, 4 channels) are convolved by the 3*3 convolution unit of the second layer, and the other part of the channels (for example, 4 channels) are convolved by the 1*1 convolution unit of the second layer; then the channels of the convolution processing results of the 3*3 convolution unit of the second layer are divided into two parts, for example, 4 channels are divided into 2+2 form, then part of the channels (for example, 2 channels) are convolved by the 3*3 convolution unit of the third layer, and the other part of the channels (for example, 2 channels) are convolved by the 1*1 convolution unit of the third layer; then the convolution processing result of the 3*3 convolution unit of the third layer is convolved by the 3*3 convolution unit of the fourth layer; then the cascade layer (Concat) cascades the convolution results of the 1*1 convolution unit of the first layer, the convolution results of the 1*1 convolution unit of the second layer, the convolution results of the 1*1 convolution unit of the third layer, and the convolution processing results of the 3*3 convolution unit of the fourth layer; then the cascade result is convolved by the 1*1 convolution unit of the fourth layer; the convolution result of the 1*1 convolution unit of the fourth layer is input to the attention mechanism layer (attention) for processing, and the processing result is combined with the input data to generate the features extracted by the module (output). In this example, the split channel and residual learning method are used to improve the network effect without significantly increasing the amount of calculation.
[0086] In step S103, feature alignment processing is performed on the first deep features, and feature alignment processing is performed on the second deep features.
[0087] In this step, the shallow features of the image sequence to be processed can be first extracted for each color channel and noise map of the image to be processed in the image sequence to be processed; then, the shallow features are used as the guiding information for feature alignment processing, and the first deep features and the second deep features are respectively subjected to feature alignment processing. Since each color channel of the image to be processed contains important and effective information, the shallow features extracted for all channels can improve the effect of feature alignment. Compared with the guiding effect of a single color channel, the guiding effect of the present application is greatly improved.
[0088] It is understandable that this step can be performed using a neural network model. Exemplarily, the neural network model has a feature alignment module for performing this step, and the feature extraction module performing step S102 inputs the extracted first deep features and second deep features into the feature alignment module, and then the feature alignment module can align the first deep features and the second deep features respectively in the above manner.
[0089] The feature alignment module can be designed based on the pyramid ConvLSTM structure, and the internal convolution is in the form of 3x3 convolution. In the existing algorithm, the convolution in the pyramid ConvLSTM uses deformable convolution, which can be changed to traditional 3x3 convolution through structural adjustment to meet more hardware requirements.
[0090] In step S104, the first deep feature and the second deep feature are fused, and a target image is generated according to the result of the fusion process.
[0091] It is understandable that this step can be performed using a neural network model. Exemplarily, the neural network model has a feature fusion module for performing this step, and the feature alignment module of step S103 inputs the aligned first deep feature and the second deep feature into the feature fusion module, and then the feature fusion module can fuse the two deep features to generate a target image, that is, an image after detail enhancement and noise reduction.
[0092] The feature fusion module adopts the UNet structure, and its internal convolution kernel can also be used as follows Figure 2 The feature extraction module shown is replaced by a channel-wise residual structure, thereby improving the fusion effect.
[0093] The present disclosure obtains a sequence of images to be processed and generates a reverse image sequence of the sequence of images to be processed, and can respectively extract the first deep features of the sequence of images to be processed and the second deep features of the reverse image sequence of the sequence of images to be processed, and can further perform feature alignment processing on the first deep features and feature alignment processing on the second deep features, and finally can perform fusion processing on the first deep features and the second deep features, and generate a target image according to the result of the fusion processing. Since the deep features extracted and aligned in both the positive and negative directions of the sequence of images to be processed are fused, the feature information of the image to be processed in the sequence of images to be processed is fully utilized, the image enhancement and denoising effects are more obvious, the target image obtained is of high quality, and the user satisfaction is improved.
[0094] In some embodiments of the present disclosure, a neural network model is used to perform the image processing method of the present disclosure, and the neural network model includes a preprocessing module, a feature extraction module, a feature alignment module and a feature fusion module; wherein the preprocessing module is used to obtain a sequence of images to be processed and generate a reverse image sequence of the sequence of images to be processed, wherein the sequence of images to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the image to be processed (i.e., performing the preprocessing module). Figure 1 The feature extraction module is used to extract the first deep features of the image sequence to be processed and the second deep features of the reverse image sequence of the image sequence to be processed respectively (i.e., perform the attached Figure 1 The feature alignment module is used to perform feature alignment processing on the first deep features and perform feature alignment processing on the second deep features (i.e., perform the attached Figure 1 The feature fusion module is used to fuse the first deep feature and the second deep feature, and generate a target image according to the result of the fusion process (i.e., perform the attached Figure 1 ) as shown in step S104.
[0095] Based on the neural network model in the above embodiments, some embodiments of the present disclosure adopt Figure 3 The model training is performed in the manner shown, including steps S301 to S303.
[0096] In step S301, a training image sequence in a training set is obtained, and noise is added to each frame of the training image in the training image sequence to obtain a noise map of each frame of the training image.
[0097] You can select a dataset to generate a training set. In one case, the dataset is a REDS dataset, etc. The data (image sequence or video) in the dataset exists in pairs, and the clarity of each pair of data is different. In this case, the dataset can be directly used as a training set, that is, the data with poor clarity in each pair of data is used as a training image sequence, and a frame image in the data with higher clarity in each pair of data is used as the corresponding label image.
[0098] In another case, the data set is a Vimeo90K data set, etc., and the data (image sequence or video) in the data set does not exist in pairs. In this case, the training set can be generated in the following manner: first, multiple original image sequences in the data set are obtained, wherein each original image sequence includes a preset number (e.g., 5 frames) of original images; then, each original image in each original image sequence is blurred to obtain a training image corresponding to each original image; finally, the training images corresponding to each original image are arranged according to the arrangement order of each original image in each original image sequence to obtain a training image sequence corresponding to each original image sequence, and the training set is determined based on all the training image sequences. At the same time, the original image of the intermediate frame of the original image sequence can also be determined as the label image of the corresponding training image sequence.
[0099] Shot noise and read noise can be randomly added to each training image, and then a noise map corresponding to the noise intensity can be calculated.
[0100] In step S302, the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image are input into the neural network model, and the predicted image is output.
[0101] Optionally, before inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, the training image sequence can also be subjected to at least one of the following processing: for the training image sequence, the position of the training image of the middle frame is kept unchanged, and the training images of other frames are randomly arranged; each frame of the training image is enhanced separately, for example, at least one of the following algorithms can be used to enhance the image: cutblur algorithm (cut blur algorithm), random rgb algorithm (random color channel algorithm), copyblend algorithm (copy blend algorithm), invertcolor algorithm (invert color algorithm), hightlight algorithm (highlight algorithm).
[0102] After the order of the training image sequence is adjusted or the image is enhanced, the order and image of the corresponding reverse image sequence are also updated accordingly, that is, the order of the training image sequence and its reverse image sequence is always kept opposite, and at the same time, the state of the same image in the training image sequence and its reverse image sequence is guaranteed to be consistent.
[0103] By changing the order and image quality of the training image sequence, the amount of training data can be expanded, that is, the same training image sequence can be expanded into more training image sequences through the above two processing aspects, thereby improving the training effect.
[0104] Among them, the neural network model can extract deep features from the input training image sequence and reverse image sequence respectively, and use shallow features to align the two deep features, and finally fuse the two deep features to obtain the predicted image.
[0105] In step S303, a network loss value is determined according to the predicted image and the label image of the training image sequence, and the network parameters of the neural network model are adjusted according to the network loss value until the neural network model converges.
[0106] Loss functions such as L1 loss and sRGB loss can be used. To avoid gradient explosion when the learning rate is large, the initial learning rate can be 10e -5 , decays by 10e every 20 epochs -1 , decays three times.
[0107] In this embodiment, the neural network model is trained by the training image sequence in the training set, so that the neural network model can perform image enhancement on multiple frames of continuously shot images, thereby generating high-quality target images to meet the quality requirements of users.
[0108] According to a second aspect of the present disclosure, an image processing device is provided. Figure 4 , the device comprises:
[0109] A preprocessing module 401 is used to obtain a sequence of images to be processed and generate a reverse image sequence of the sequence of images to be processed, wherein the sequence of images to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the images to be processed;
[0110] A feature extraction module 402 is used to extract first deep features of the image sequence to be processed and second deep features of the reverse image sequence of the image sequence to be processed respectively;
[0111] A feature alignment module 403, configured to perform feature alignment processing on the first deep features and perform feature alignment processing on the second deep features;
[0112] The feature fusion module 404 is used to fuse the first deep feature and the second deep feature, and generate a target image according to the result of the fusion process.
[0113] In some embodiments of the present disclosure, when the acquisition module is used to acquire the image sequence to be processed, it is specifically used to:
[0114] Acquire multiple frames of images to be processed that are arranged in sequence, and generate a noise map of each frame of the image to be processed according to the parameters of an acquisition device that acquires the images to be processed;
[0115] A sequence of images to be processed is generated by sequentially arranging a plurality of frames of images to be processed and a noise map of each frame of the image to be processed.
[0116] In some embodiments of the present disclosure, the feature alignment module is specifically used to:
[0117] Extracting shallow features of the image sequence to be processed for each color channel and noise map of the image to be processed in the image sequence to be processed;
[0118] The shallow features are used as guidance information for feature alignment processing, and feature alignment processing is performed on the first deep features and the second deep features respectively.
[0119] In some embodiments of the present disclosure, the image processing device includes a neural network model, and the neural network model includes the feature extraction module, the feature alignment module and the feature fusion module.
[0120] In some embodiments of the present disclosure, the neural network model also includes the preprocessing module.
[0121] In some embodiments of the present disclosure, the image processing apparatus further includes a training module for:
[0122] Acquire a training image sequence in a training set, and add noise to each frame of the training image in the training image sequence to obtain a noise map of each frame of the training image;
[0123] Inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, and outputting the predicted image;
[0124] A network loss value is determined according to the predicted image and the label image of the training image sequence, and network parameters of the neural network model are adjusted according to the network loss value until the neural network model converges.
[0125] In some embodiments of the present disclosure, the training module is further used to:
[0126] Before inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, for the training image sequence, the position of the training image of the middle frame is kept unchanged, and the training images of other frames are randomly arranged.
[0127] In some embodiments of the present disclosure, the training module is further used to:
[0128] Before inputting the training image sequence, the reverse image sequence of the training image sequence, the noise map of each frame of training image and the noise map of each frame of training image into the neural network model, each frame of training image is enhanced.
[0129] In some embodiments of the present disclosure, a training set generation module is further included, which is used to:
[0130] Acquire multiple original image sequences, wherein each original image sequence includes a preset number of original images;
[0131] Perform blur processing on each original image in each original image sequence to obtain a training image corresponding to each original image;
[0132] According to the arrangement order of each original image in each original image sequence, the training images corresponding to each original image are arranged to obtain a training image sequence corresponding to each original image sequence, and a training set is determined according to all the training image sequences.
[0133] In some embodiments of the present disclosure, the training set generation module is further used to:
[0134] The middle frame original image of the original image sequence is determined as the label image of the corresponding training image sequence.
[0135] Regarding the device in the above embodiment, the specific manner in which each module performs the operation has been described in detail in the embodiment of the first aspect related to the method, and will not be elaborated here.
[0136] According to a third aspect of an embodiment of the present disclosure, a chip is provided, the chip comprising a processor, the processor being used to execute the image processing method described in the first aspect. The chip may be a conventional CPU (central processing unit) chip, a GPU (graphics processing unit) chip, etc., or an acceleration chip dedicated to artificial intelligence technology, such as an AI (Artificial Intelligence) accelerator, etc.
[0137] According to the fourth aspect of the embodiment of the present disclosure, please refer to the attached Figure 5 , which exemplarily shows a block diagram of an electronic device. For example, the device 500 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.
[0138] Reference Figure 5, the device 500 may include one or more of the following components: a processing component 502 , a memory 504 , a power component 506 , a multimedia component 508 , an audio component 510 , an input / output (I / O) interface 512 , a sensor component 514 , and a communication component 516 .
[0139] The processing component 502 generally controls the overall operation of the device 500, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 502 may include one or more processors 520 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 502 may include one or more modules to facilitate the interaction between the processing component 502 and other components. For example, the processing component 502 may include a multimedia module to facilitate the interaction between the multimedia component 508 and the processing component 502.
[0140] The memory 504 is configured to store various types of data to support operations on the device 500. Examples of such data include instructions for any application or method operating on the device 500, contact data, phone book data, messages, pictures, videos, etc. The memory 504 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.
[0141] Power component 506 provides power to various components of device 500. Power component 506 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for device 500.
[0142] The multimedia component 508 includes a screen that provides an output interface between the device 500 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 508 includes a front camera and / or a rear camera. When the device 500 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and the rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.
[0143] The audio component 510 is configured to output and / or input audio signals. For example, the audio component 510 includes a microphone (MIC), and when the device 500 is in an operating mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 504 or sent via the communication component 516. In some embodiments, the audio component 510 also includes a speaker for outputting audio signals.
[0144] I / O interface 512 provides an interface between processing component 502 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.
[0145] The sensor assembly 514 includes one or more sensors for providing various aspects of status assessment for the device 500. For example, the sensor assembly 514 can detect the open / closed state of the device 500, the relative positioning of components, such as the display and keypad of the device 500, the sensor assembly 514 can also detect the position change of the device 500 or a component of the device 500, the presence or absence of user contact with the device 500, the orientation or acceleration / deceleration of the device 500 and the temperature change of the device 500. The sensor assembly 514 can also include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 514 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 514 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.
[0146] The communication component 516 is configured to facilitate wired or wireless communication between the device 500 and other devices. The device 500 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, 4G or 5G or a combination thereof. In an exemplary embodiment, the communication component 516 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 516 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.
[0147] In an exemplary embodiment, the apparatus 500 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to execute the power supply method for the above-mentioned electronic device.
[0148] In a fifth aspect, the present disclosure, in an exemplary embodiment, further provides a non-transitory computer-readable storage medium including instructions, such as a memory 504 including instructions, and the instructions can be executed by a processor 520 of the device 500 to complete the power supply method of the electronic device. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.
[0149] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the disclosure disclosed herein. This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art that are not disclosed in the present disclosure. The description and examples are intended to be exemplary only, and the true scope and spirit of the present disclosure are indicated by the following claims.
[0150] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.
Claims
1. An image processing method, characterized in that: include: Acquire a sequence of images to be processed, and generate a reverse image sequence of the sequence of images to be processed, wherein the sequence of images to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the images to be processed; Respectively extracting first deep features of the image sequence to be processed and second deep features of the reverse image sequence of the image sequence to be processed; Performing feature alignment processing on the first deep features, and performing feature alignment processing on the second deep features; Fusing the first deep feature and the second deep feature, and generating a target image according to a result of the fusion process; The step of obtaining a sequence of images to be processed comprises: Acquire multiple frames of images to be processed that are arranged in sequence, and generate a noise map of each frame of the image to be processed according to the parameters of an acquisition device that acquires the images to be processed; Generate a sequence of images to be processed by sequentially arranging a plurality of frames of images to be processed and a noise map of each frame of the images to be processed; The performing feature alignment processing on the first deep features and performing feature alignment processing on the second deep features includes: Extracting shallow features of the image sequence to be processed for each color channel and noise map of the image to be processed in the image sequence to be processed; The shallow features are used as guidance information for feature alignment processing, and feature alignment processing is performed on the first deep features and the second deep features respectively.
2. The image processing method according to claim 1, characterized in that: The image processing method is performed using a neural network model, which includes a feature extraction module, a feature alignment module and a feature fusion module; Among them, the feature extraction module is used to extract the first deep features of the image sequence to be processed and the second deep features of the reverse image sequence of the image sequence to be processed respectively; the feature alignment module is used to perform feature alignment processing on the first deep features and perform feature alignment processing on the second deep features; the feature fusion module is used to fuse the first deep features and the second deep features, and generate a target image according to the result of the fusion processing.
3. The image processing method according to claim 2, characterized in that: The neural network model also includes a preprocessing module for acquiring an image sequence to be processed and generating a reverse image sequence of the image sequence to be processed, wherein the image sequence to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the image to be processed.
4. The image processing method according to claim 2, characterized in that: Also includes: Acquire a training image sequence in a training set, and add noise to each frame of the training image in the training image sequence to obtain a noise map of each frame of the training image; Inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, and outputting a predicted image; A network loss value is determined according to the predicted image and the label image of the training image sequence, and network parameters of the neural network model are adjusted according to the network loss value until the neural network model converges.
5. The image processing method according to claim 4, characterized in that: Before inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, the method further includes: For the training image sequence, the position of the training image of the middle frame is kept unchanged, and the training images of other frames are randomly arranged.
6. The image processing method according to claim 4, characterized in that: Before inputting the training image sequence, the reverse image sequence of the training image sequence, the noise map of each frame of the training image, and the noise map of each frame of the training image into the neural network model, the method further includes: Each frame of training image is enhanced separately.
7. The image processing method according to claim 4, characterized in that: Also includes: Acquire multiple original image sequences, wherein each original image sequence includes a preset number of original images; Perform blur processing on each original image in each original image sequence to obtain a training image corresponding to each original image; According to the arrangement order of each original image in each original image sequence, the training images corresponding to each original image are arranged to obtain a training image sequence corresponding to each original image sequence, and a training set is determined according to all the training image sequences.
8. The image processing method according to claim 7, characterized in that: Also includes: The middle frame original image of the original image sequence is determined as the label image of the corresponding training image sequence.
9. An image processing device, characterized in that: include: A preprocessing module, used for acquiring an image sequence to be processed and generating a reverse image sequence of the image sequence to be processed, wherein the image sequence to be processed includes a plurality of frames of images to be processed arranged in sequence and a noise map of each frame of the image to be processed; A feature extraction module, used to extract first deep features of the image sequence to be processed and second deep features of the reverse image sequence of the image sequence to be processed respectively; A feature alignment module, used for performing feature alignment processing on the first deep features and performing feature alignment processing on the second deep features; A feature fusion module, used for fusing the first deep feature and the second deep feature, and generating a target image according to the result of the fusion process; When the preprocessing module is used to obtain the image sequence to be processed, it is specifically used to: Acquire multiple frames of images to be processed that are arranged in sequence, and generate a noise map of each frame of the image to be processed according to the parameters of an acquisition device that acquires the images to be processed; Generate a sequence of images to be processed by sequentially arranging a plurality of frames of images to be processed and a noise map of each frame of the images to be processed; The feature alignment module is specifically used for: Extracting shallow features of the image sequence to be processed for each color channel and noise map of the image to be processed in the image sequence to be processed; The shallow features are used as guidance information for feature alignment processing, and feature alignment processing is performed on the first deep features and the second deep features respectively.
10. The image processing device according to claim 9, characterized in that: The image processing device includes a neural network model, and the neural network model includes the feature extraction module, the feature alignment module and the feature fusion module.
11. The image processing device according to claim 10, characterized in that: The neural network model also includes the preprocessing module.
12. The image processing device according to claim 10, characterized in that: The image processing device further comprises a training module, which is used for: Acquire a training image sequence in a training set, and add noise to each frame of the training image in the training image sequence to obtain a noise map of each frame of the training image; Inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, and outputting a predicted image; A network loss value is determined according to the predicted image and the label image of the training image sequence, and network parameters of the neural network model are adjusted according to the network loss value until the neural network model converges.
13. The image processing device according to claim 12, characterized in that: The training module is also used to: Before inputting the training image sequence and the noise map of each frame of the training image, and the reverse image sequence of the training image sequence and the noise map of each frame of the training image into the neural network model, for the training image sequence, the position of the training image of the middle frame is kept unchanged, and the training images of other frames are randomly arranged.
14. The image processing device according to claim 12, characterized in that: The training module is also used to: Before inputting the training image sequence, the reverse image sequence of the training image sequence, the noise map of each frame of training image and the noise map of each frame of training image into the neural network model, each frame of training image is enhanced.
15. The image processing device according to claim 12, characterized in that: It also includes a training set generation module for: Acquire multiple original image sequences, wherein each original image sequence includes a preset number of original images; Perform blur processing on each original image in each original image sequence to obtain a training image corresponding to each original image; According to the arrangement order of each original image in each original image sequence, the training images corresponding to each original image are arranged to obtain a training image sequence corresponding to each original image sequence, and a training set is determined according to all the training image sequences.
16. The image processing device according to claim 15, characterized in that: The training set generation module is also used for: The middle frame original image of the original image sequence is determined as the label image of the corresponding training image sequence.
17. A chip, characterized in that: The chip comprises a processor, and the processor is used to execute the image processing method according to any one of claims 1 to 8.
18. An electronic device, characterized in that: The electronic device comprises a memory and a processor, wherein the memory is used to store computer instructions executable on the processor, and the processor is used to execute the computer instructions based on the image processing method according to any one of claims 1 to 8.
19. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.